Malcolm-Soft

UTF-16 Anywhere?

2016-01-03T15:40:53+03:00

Another question about Unicode popped up not so long ago into the “attention magnet” list of questions on the StackExchange network. What’s interesting, even though the question itself is about the total characters possible using UTF-8, the discussion quickly jumped to the limitations of UTF-16 and how this standard needs to be phased out.

It probably all began from the StackOverflow question, Should UTF-16 be Considered Harmful, which has since been migrated to Programmers.SE. The author of the top voted post didn’t settle with his answer though, and now there’s a whole website promoting UTF-8: utf8everywhere.org. And I’ve been really curious about the effort he put into that.

The facts on the website look correct overall, the only strange fact is that they say that

UTF-16 is often misused as a fixed-width encoding.

The problem is, there is no such miracle as fixed-width UTF-16. If we always have two bytes per character, that’s UCS-2, period. It’s not UTF-16. That’s definitely a problem, but we’ll come back to it later.

The conclusion section is the most interesting one. Here we can take the arguments one by one.

UTF-16 is too wide.

Being too wide is not really much of a problem since earlier he makes the argument himself that compressed text size doesn’t differ between UTF-8 and UTF-16. And for efficient communication it will always be better to use compression, so this “disadvantage” doesn’t really hold any water.

Where it would matter is in the in-memory representations where compressing the strings back and forth would have a certain tax on performance. But it doesn’t necessarily mean that we should simply use UTF-8. For example, JEP 254 on compacting strings in Java uses the approach of representing strings in ASCII where possible and in UTF-16 in all other cases. This will save the memory even better than UTF-8 because it will not only be at least as efficient as UTF-8, but it will also use two bytes where UTF-8 would use three.

It exists only for historical reasons and creates a lot of confusion.

Historical reasons - that doesn’t say all, it exists for compatibility reasons (with UCS-2). And so does UTF-8, to be compatible with ASCII. As for creating lots of confusion, we’ll take a look at that later.

Adding wchar_t to the C++ standard was a mistake, and so are the Unicode additions to C++11.

I believe that they shouldn’t have had wchar_t in the standard as well, but the thing is the standard doesn’t really say anything about the relation between wchar_t and Unicode. Isn’t this the fault of the vague standard though?

This is even more supported by the website having a whole section on how to work with the strings in C++ on Windows. Not just any language or any platform, but this combination in particular. It makes sense, of course, because the author (or at least one of them, since there are three names) even mentions primarily being a Windows developer, but it just looks strange to pay so much attention to a specific case on a site dealing with the encodings in general. Doesn’t so much attention to this special case hint that the problem lies in there and not with UTF-16?

Safety is an important feature of every design, and encodings are no exception.

Here’s a fun fact: when you use UTF-8, it doesn’t automatically mean that you will get proper non-BMP code point support. I had my share of surprise when I found some time ago that MySQL apparently has a so called 4-byte UTF-8 Unicode encoding. As it turns out, if you specify regular UTF-8, it will not handle non-BMP characters with the same effect as if they declared the use of UTF-16 and used UCS-2 instead. And this is a relatively recent addition, this wonderful encoding appeared in MySQL 5.5.3 in 2010, while (crippled) UTF-8 has been there long before that, successfully confusing people.

So as we see UTF-8 can have problems too. Of course, there are much more cases of that with the UTF-16. But there’s nothing particularly wrong with UTF-16, it is as valid compatibility tool as UTF-8. What people are really complaining about is that there is a lot of software which works with UCS-2 but says that it supports UTF-16. This results from UCS-2 being a thing for a noticeable time period and a considerable amount of software embracing it during that time. This is where all the confusion really comes from. Of course, when a transition needs to happen later, it never goes smoothly because legacy software and backward compatibility is always a thing. As a result we have to put up with the consequences, and there are serious problems with that indeed. But that’s not UTF-16’s fault, and this couldn’t have happened any other way. If not for UTF-16, we would be stuck with UCS-2 with no hope to support non-BMP characters at all since rewriting existing software completely is rarely an option.

It surely would be advantageous to use one encoding everywhere. If one day every piece of software started supporting one encoding and nothing else, it would be awesome, be it UTF-8 or UTF-16. However, the chances of this happening in the foreseeable future are nil. Let’s take a look at the programming language rankings by RedMonk which I like so much. What is the string object representation in the top ten languages?

JavaScript - UTF-16;
Java - UTF-16;
PHP - no specific encoding;
Python - UTF-8;
C# - UTF-16;
C++ - no specific encoding;
Ruby - UTF-8;
CSS - N/A;
C - no specific encoding;
Objective-C - UTF-16.

Four out of nine languages (CSS doesn’t count as a programming language) use UTF-16 and only two use UTF-8. One can argue that something like C++ could give the developer a choice of encoding, but that’s not entirely the case, because you will have to take the platform and existing libraries into account, which is why there’s a long section about Windows on utf8everywhere.org.

But even that’s not all, the languages which use UTF-16 represent underlying platforms: Java represents the JVM, browser scripts will have to compile to JavaScript (and not only browser scripts if you remember node.js), and C# represents the .NET platform. We could say Objective-C is tied to a platform too, even though it is limited to Apple software. What this means is that anything that uses these platforms will have to deal with UTF-16.

I’m sorry to break it to the zealous UTF-8 supporters, but UTF-16 isn’t going anywhere anytime soon. Understand why it’s here, understand how it works, understand that there are legacy functions and methods which deal with UTF-16 code units and not code points, and embrace this encoding. And please fix the legacy software if you are maintaining it.

Automatic Android resource generation with Photoshop scripts

2014-01-28T23:55:33+04:00

A little foreword would be proper, I guess. As it is known, when developing for Android, you have to keep in mind that you should create resources for all the pixel densities. Originally there were only three of them: ldpi, mdpi, and hdpi. However, the progress waits for no one: pixel densities have been growing to crazy numbers, and Google has been slyly adding the letter “x” and got to xxxhdpi by now, which means that now there are six major screen configurations. But even that’s not all, because some resources have several different states. Buttons on the action panel have two states, and that would be decent enough, but common buttons have much more of them.

There are several ways out of this situation: nag your artist, let multiple density support slide and hope that the system is going to somehow handle resource scaling on its own, or you could also do what programmers like to do the most: automation. There are various tools which can help with this. The most advanced one is probably Android Asset Studio. But the icons are, of course, drawn only for the common cases, and if you need to draw something custom, with unique styles, it is a poor helper. And that’s where we have something to rescue us: scripting support in a fairly well-known instrument, Adobe Photoshop. In order to simplify the whole process, your humble servant has written several scripts for myself just for the cases like this, and now I’m sharing them with the readers. This article is where I’m going to describe how they work and how they are used. All the sources can be grabbed on BitBucket, and here I will show major points of interest and also I will shed some light on various tricks of working with Photoshop scripts which could be not obvious for beginners.

Using the scripts and how they work

For those who are completely unfamiliar with the scripting for Adobe Photoshop, I will briefly describe what it involves. The standard tool for this is ExtendScript Toolkit which comes bundled with Photoshop. Some guys on the Internet mentioned that the tool is rather silly, and I have to sorely agree with them that it is completely true, but that’s what we have. It also has documentation on the function built into Photoshop, on the F1 key, which is as unwieldy as the editor itself, but at least it does its main function. The scripts themselves can be written in many languages, and personally I used JavaScript.

Creating icons for all densities

Returning to the matter at hand, all the scripts for working with resources can be divided into two categories: some of them directly launch the necessary actions (all of them start with “Make”), and the others serve as libraries with functions. The most important and and ubiquitous instrument is MakeForAllDensities which does exactly what says its name: it creates resources for all densities. The document which we use as the source has some requirements:

It should be created for the mdpi densitiy. It is condidered the baseline, and then it is scaled to the required size.
The document should already be saved somewhere so that the script reads the filename correctly (and also determines whether this is a nine-patch or not by the postfix “.9”). It is best to save in a subfolder of the main Android project folder, this way the script will find the resources folder by itself.
An additional requirement: if this is a nine-patch, the lines should be drawn on a separate layer, the lowermost one.
And the image should certainly be vector and not raster, because otherwise there’s little point in scaling it with Photoshop instead of relying on Android. Just in case I should point out that nine-patch lines are an exception, and they can be raster.

If all the requirements are met, the rest is easy: open the document in Photoshop and launch the script by double clicking. After it launches, it will ask to show where the folder with resources is (“res”). If the document is saved in a subfolder of the main project folder, it will find out where to save, and will finish up the rest by itself.

The script itself looks very simple:

//@include ResizingAndSaving.jsx

#target photoshop

var outputFolder = detectFolder();
if (outputFolder) saveForAllDensities(outputFolder, 0, "");

Typical JavaScript, except for the first two weird lines. The first weird line looks like a common comment which is actually not a comment at all, but an import of a different file with the functions we need. This is the trick number one, because in the standard JavaScript you can’t get by with things like this. The second weird line, as you can guess, hints that this is a script for Photoshop. As for what the rest of the lines do, we’re going to find that out by looking at the ResizingAndSaving file.

I’m not going to give detectFolder here as it has nothing too special in it: the function check if there is a “res” folder in the parent folder of the document’s folder, and returns it if it is found. If not, then the user will be asked. But then something more interesting begins:

function saveForAllDensities(outputFolder, version, postfix, ninePatchLines) {
    if (!ninePatchLines) ninePatchLines = computeNinePatchLines();
    
    var versionStr = version ?  "-v" + version : "";
    saveInFolder(outputFolder, "drawable-mdpi" + versionStr, 100, postfix, ninePatchLines);
    saveInFolder(outputFolder, "drawable-hdpi" + versionStr, 150, postfix, ninePatchLines);
    saveInFolder(outputFolder, "drawable-xhdpi" + versionStr, 200, postfix, ninePatchLines);
    saveInFolder(outputFolder, "drawable-xxhdpi" + versionStr, 300, postfix, ninePatchLines);
    saveInFolder(outputFolder, "drawable-xxxhdpi" + versionStr, 400, postfix, ninePatchLines);
};

I will answer the question in advance if someone has it: there’s no ldpi density here because Google doesn’t recommend creating resources for it explicitly. As it has been mentioned, the file could happen to be a nine-patch, and in terms of editing the file that means that it has a separate layer with the black lines on the sides. And we can’t just take these lines and scale them: we have to paint the pixels fully black or not paint them at all, and also we can’t touch the adjacent pixels. Besides, we have to take into account that the lines could not be continuous. That’s where the computeNinePatchLines function comes in.

function computeNinePatchLines() {
    var docName = getDocName(false);
    if (!isNinePatch(docName)) return null;
    
    var ninePatchLines = null;
    
    var doc = app.activeDocument;
    var areaCheckingFunctions = [
        function(pos) {return areaIsEmpty(doc, pos, 0);},
        function(pos) {return areaIsEmpty(doc, 0, pos);},
        function(pos) {return areaIsEmpty(doc, pos, doc.height - 1);},
        function(pos) {return areaIsEmpty(doc, doc.width - 1, pos);}
    ];
    maxPositions = [doc.width, doc.height, doc.width, doc.height];
    ninePatchLines = new Array();
    for (var pos = 0; pos < areaCheckingFunctions.length; pos++) {
        ninePatchLines.push(findLines(maxPositions[pos], areaCheckingFunctions[pos]));
    }

    return ninePatchLines;
}

It is somewhat interesting to look into the function called areaIsEmpty():

function areaIsEmpty(doc, x, y) {
    var state = getState();
    
    if (doc.colorSamplers.length == 0) {
        var colorSampler = doc.colorSamplers.add([x,y]);
    } else {
        var colorSampler = doc.colorSamplers[0];
        colorSampler.move([x, y]);
    }

    var areaEmpty;
    try {
        areaEmpty = colorSampler.color.rgb.hexValue !== "000000";
    } catch (e) {
        areaEmpty = true;
    }

    restoreState(state);
    
    return areaEmpty;
}

This function’s purpose is to check if the pixel is filled with black or not. But the thing is, Photoshop, as it turns out, is unable to tell us if the pixel is empty or not. Because of that we have to put a color sampler on it and check if an exception is going to be thrown. If yes, then the pixel indeed is empty. If not, then we can take a look at its color. That’s the trick number two. The findLines function, which is not shown here, simply applies areaIsEmpty for all the pixels along each of the four edges of the screen and records their positions.

After this we can scale the resources and save them into a folder.

function saveInFolder(outputFolder, subFolder, scaling, postfix, ninePatchLines) {
    var opts = new ExportOptionsSaveForWeb(); 
    opts.format = SaveDocumentType.PNG; 
    opts.PNG8 = false; 
    opts.transparency = true; 
    opts.quality = 100;
    
    var state = getState();
    
    if (ninePatchLines) {
        var doc = app.activeDocument;	
        doc.resizeCanvas(doc.width - 2, doc.height - 2);
        resize(scaling, true);
        doc.resizeCanvas(doc.width + 2, doc.height + 2);
        drawLines(doc, scaling / 100, ninePatchLines);
    } else {
        resize(scaling, true);
    }
    activeDocument.exportDocument(createFile(outputFolder, subFolder, postfix, ".png", false), ExportType.SAVEFORWEB, opts);
    restoreState(state);
}

Everything is overall pretty clear here, but the way the picture is resized deserves a separate explanation. You’d think that there is a function Document.resize, and you could simply call it, right? But nothing will come out of this: the layer styles are not scaled this way. We could record an action and then play it programmatically. This works, but this solution has a problem that we will have to also keep a library of actions in addition to the script itself which must be imported into Photoshop before launching the script, which is kind of inconvenient.

Another solution is to use an interesting tool called ScriptListener.8li. It allows us to record all the actions done in Photoshop as a script regardless of whether these actions are in the standard API. The scripts that come out are rather inarticulate, but do their job perfectly. With a certain effort it is possible to find out specific functions of specific parameters and to create a working function out of the recorded actions. That’s how this incomprehensible, but working function came to be:

function resize(size, relative) {
    var idImgS = charIDToTypeID( "ImgS" );
        var desc89 = new ActionDescriptor();
        var idWdth = charIDToTypeID( "Wdth" );
        var idPxl = charIDToTypeID( relative ? "#Prc" : "#Pxl" );
        desc89.putUnitDouble( idWdth, idPxl, size );
        var idscaleStyles = stringIDToTypeID( "scaleStyles" );
        desc89.putBoolean( idscaleStyles, true );
        var idCnsP = charIDToTypeID( "CnsP" );
        desc89.putBoolean( idCnsP, true );
        var idIntr = charIDToTypeID( "Intr" );
        var idIntp = charIDToTypeID( "Intp" );
        var idbicubicSharper = stringIDToTypeID( "bicubicAutomatic" );
        desc89.putEnumerated( idIntr, idIntp, idbicubicSharper );
    executeAction( idImgS, desc89, DialogModes.NO );
}

That sums up the trick number three. After we have the necessary size of the image, we can draw the nine-patch lines (if we need to), and the freshly made resource goes in the corresponding folder.

Creating icons for the action bar

Besides MakeForAllDensities there are four scripts MakeActionBarIcons, which make the icons for the action bar: for the black and white theme, with and without the disabled state. They are used in the same manner as MakeForAllDensities with the exception that the document is now supposed to contain just one layer. The most important part for this layer is to maintain the shape of the icon, and the styles will be applied automatically.

Now the difficulty is that Google has certain requirements for the button style depending on their state. If the button has just one state, then everything is simple and easy, but if it can be disabled, he have to devise a way to modify the layer appearance programmatically. In the case of action bar icons we have to know how we can change the layer opacity and color. There are no problems with the former, but with the latter the standard API shows its weak points again. That means we have to turn to our trusty ScriptListener.8li again. As a result of its use we got the function setLayerColor in the Styles file which is going to help us with changing the color of the vector layer. I think the abracadabra in the function body is best to be omitted.

Considering the action bar icons, nothing besides already mentioned is required to create them per se. But those who already looked into the Styles file, have already found out that there are lots of functions which came to be with the help of ScriptListener.8li, and which purpose is to manipulate layer effects. They were written for my own icons, for which I didn’t add the scripts into the repository. Therefore someone may find them to be not enough and will have to create more. Again, it is possible to record actions or to make some styles and apply them programmatically. But this is not really convenient, so it’s best to stick to the function again, especially now that we mastered ScriptListener.8li. And that’s when we’re going to have another quirk to deal with: if we create a script to add a certain effect to a layer and make a function out of it, we’ll find out that when we apply it, previous effects are gone. This is where the trick number four comes useful. If we look closer at the beginning of the gibberish produced by ScriptListener.8li when we apply a layer effect, we’ll keep seeing the following lines:

var idsetd = charIDToTypeID( "setd" );
    var desc = new ActionDescriptor();
    var idnull = charIDToTypeID( "null" );
        var ref = new ActionReference();
        var idPrpr = charIDToTypeID( "Prpr" );
        var idLefx = charIDToTypeID( "Lefx" );
        ref.putProperty( idPrpr, idLefx );
        var idLyr = charIDToTypeID( "Lyr " );
        var idOrdn = charIDToTypeID( "Ordn" );
        var idTrgt = charIDToTypeID( "Trgt" );
        ref.putEnumerated( idLyr, idOrdn, idTrgt );
    desc.putReference( idnull, ref );
    var idT = charIDToTypeID( "T   " );
        var desc2 = new ActionDescriptor();
        var idScl = charIDToTypeID( "Scl " );
        var idPrc = charIDToTypeID( "#Prc" );
        desc2.putUnitDouble( idScl, idPrc, 100.000000 );

And the script is ended with the following chord:

var idLefx = charIDToTypeID( "Lefx" );
desc.putObject( idT, idLefx, desc2 );
executeAction( idsetd, desc, DialogModes.NO );

The lines responsible for the creation of the style itself are the last four lines from the first portion of the code above, the ones which create desc2 and set the scale for the style. The rest is exactly what applies the style. Now that we know which lines do this, we can separate veins from meat and cut the boilerplate code from the code which applies the effect itself. The repeated lines have been placed into special functions in Styles which are used in the following way:

var style = newStyle(); //Create new style
addColorOverlay(style, 0xFF, 0xFF, 0xFF, 100); //Apply the effect
addStroke(style, 0x00, 0x00, 0x00, 3); //Apply one more effect
applyStyle(style); //And, finally, apply the whole style

Now we’ve got the instruments in our hands, it’s about time to use them. I have to remind that all these miraculous things were contrived to apply different styles to the resource depending on its state. The makeIcons function in MenuIcons does exactly this: it applies different styles to the icon for the action bar and saves the results. I give the most important part here:

if (makeStateful) {
    var selectorData = [
        {
            state_enabled: false,
            postfix: "disabled"
        },
        {
            postfix: "normal"
        }
    ];
    makeSelectorXml(selectorData, outputFolder, "drawable");
}

var styleFunctions = [function(style) {applyActionBarItemStyle(whiteTheme, false)}];
var postfixes = ["normal"];
if (makeStateful) {
    styleFunctions.unshift(function(style) {applyActionBarItemStyle(whiteTheme, true)});
    postfixes.unshift("disabled");
}
saveStyledDrawables(outputFolder, styleFunctions, postfixes);

The first part of the function create a selector for our resource. You can read more about selectors in the Android documentation. In our case we have two states for the action bar: when the button is disabled and when it is in the normal condition. Correspondingly, an array describing the states is passed into the function makeSelectorXml. The objects have a field called postfix and, if required, one or more fields beginning with “state_”. After that, makeSelectorXml makes an XML-file of the selector out of this monster which goes into the drawables folder.

The second part of the function creates two arrays: one of them creates function which apply the styles for the states, and the second one contains postfixes corresponding to the states. Each style applying function has the style argument at its disposal. It is exactly that object which comes out of the newStyle, which we were struggling with no so long ago. There’s no need to call applyStyle, the saveStyledDrawavles will take care of everything. The functions makeSelectorXml and saveStyledDrawables shouldn’t be put here, I think, because there’s nothing in them except for plain and boring JavaScript.

Conclusion

So this is how it is possible to not draw a bunch of icons, and use a ready solution instead to cook up everything from a single image. You could use Android Asset Studio, of course, but the scripting way has its advantages. First of all, you can create scripts for your own icons which are simply absent in a third party tool (which is exactly why I started writing all this stuff). Secondly, having to upload a file to the website, dancing around the settings, and then downloading the images and shoving them into the necessary folders is not exactly as easy as just double clicking a script so it does all properly by itself. Besides that, Android Asset Studio doesn’t wish to work with PSDs directly (at least at the moment of writing this article), doesn’t see the difference between a common button and a nine patch, and batch file processing is a can’t do.

I hope the article will be useful for those who develop applications for Android, and also for the ones who want to learn how to write scripts for Adobe Photoshop.

Some useful links

Project page on BitBucket - that’s what this whole article is about.
Iconography of Android - everyone has read this, I think, but if not, this should be fixed as soon as possible.
Nine patches in Android - some information about what a nine-patch is.
Selectors for drawables with state - official documentation on the stateful drawables.

About the blog, the site, and myself

2014-01-23T21:43:04+04:00

So, here’s my website! Here you will find information about myself, my projects, and also a good share of my thoughts and different stuff I want to share (if you read the posts, that is). Most of my time is spent programming for mobile devices. Today this means that I write mostly for Android. It is certainly possible that it will change in some future because you never know what new developments in IT can surprise you with, but for now Android is my primary area of interest.

That certainly doesn’t mean I don’t do anything else. There is a lot of interesting stuff around. We always love to try out this and that, right? So that’s what I do: experiment with different stuff, such as programming languages, frameworks, tools and so on. And some of these experiments find their way here (if they are worthy enough, of course, and I’m in the right mood to write).

And this is also my blog

… which means that I write about the stuff. Something I experiment with, as I mentioned, or something interesting I find - my personal take on the things. I have to admit that I criticize things a lot. As a wise man once wrote, everything we create sucks. At least, to some extent. That includes our software, tools we use, and programming languages (and they are tools too). But that doesn’t matter as long as we know the weak spots and work on them. Don’t take it personally if I mentioned your tool unfavorably too!

My nicknames

If you read the page about me, you should’ve noticed that I mentioned two of my nicknames: Malcolm and Dr. Metallius. The first one is a nickname which also gave the name for this website. The name is that uncommon, and quite far from my real name. Only those who played Unreal Tournament in the days of its glory will know it’s that Malcolm. By the way, I really hope the days of UT come back to us, but I digress.

Unfortunately, it’s also usually taken on the websites (did I mention that this name is not uncommon?). So when it is, Dr. Metallius is at your disposal. I happen to like thrash metal music, and I also play an electric guitar, so now you know where this nick comes from. The name also starts with an “M”, so I don’t even have to change my avatar! You can see it on the left.

How do you contact me?

Plenty of contact links are available at the page about me. The most obvious one is the email, I’ll put it here too: alexander [dot] gazarov [at] gmail [dot] com.