Got Unicode?

I'm updating my FileMaker Unicode database database to reflect the changes in the recent versions of Unicode. As part of the database, I like to have the decimal version of the code point handy as well as the actual hexadecimal version (it's good for debugging purposes).

Now the default version does not appear to have to hex to decimal conversion built in (not even in FileMaker 10), so here's my (updated) solution.

1. In the main table corresponding to the list of code points, I created a field for the Hexadecimal Unicode code point value. I'll call this HexValue for now. It must be a Text field. You can create a Decimal field (Calculated), but you won't be able to fill in the formula yet.
2. Then I created a second table to store the correspondence between a hex digit (0-F) and its decimal value (0-15). The HexValuefield is Text, but the DecValue field is a Number. See the sample table below (some values skipped).

   HexValue (Text)	DecValue (Number)
   0	0
   1	1
   2	2
   3	3
   4...9 (1 row each)	4...9
   A	10
   B	11
   C...E (1 row each)	12...14
   F	15

3. To do all the conversions, you need to extract the text value of each position in the code point. So, I created fields corresponding to the value for each place in the hex code point as shown in the list below. I'll explain the formulas below.

   Note: In case you're wondering, the name of the places are semi-inspired by Roman numerals and algebra.

   • Rightmost digit Units (n) : nhex = Right(HexValue;1)
   • Penultimate digit (t) : thex = Left(Right(UnicodeHex;2);1)
   • Antepenultimate digit (c) : chex = Left(Right(UnicodeHex;3);1)
   • 4th from right (m): mhex =Left(Right(UnicodeHex;4);1)
   • 5th from right (d): dhex =If(Length (UnicodeHex)>4;Left(Right(UnicodeHex;5);1);"0")
   • 6th from right (x): xhex = If(Length (UnicodeHex)>5;Left(Right(UnicodeHex;6);1);"0")

   The challenge for modern Unicode is that code points now come in variable lengths (4-6), so if you count from the left you can't always know you are the appropriate digit. That means you have to count from the right, but there's no simple formula for picking the 2nd digit from the right. My solution is to take a rightmost chunk then count in from the left. So to get the 3rd hex digit from the left, I take the right most 3 digits, then find the leftmost digit in that chunk (hence the embedded left(right) formulas).

   I also have to check to see if the length is greater than 4. When the length is 4, some digits are filled in with the value 0, otherwise you do a string extraction. Hence the formulas for dhex and xhex use conditional logic. Hopefully though, if Unicode adds more digits, these formulas will continue to work (unlike my original attempt which only assumed 4 digits in the code point.

4. To convert each extracted digit to its decimal version. I need to set up some Relationships between tables so that each extracted digit can look up the decimal equivalent. For each of the intermediate digit fields above, I created a link to an instance of the Hexadecimal Lookup table (there are 4 instances total). It's important to make sure each instance has a name you can remember later; mine mention which digit I am working on. See the Relationships diagram below.
   
5. Now we can finally get that decimal value! If you haven't already, create a DecimalValue field and make it Calculated.
6. Here's my calculation. I'll explain what the parts mean below
   HexLookup N::DecValue + 16*HexLookup T::DecValue + 16^2* HexLookup C::DecValue + 16^3*HexLookup M::DecValue + 16^4*HexLookup D::DecValue+16^5*HexLookup X::DecValue
   • "HexLookupN::DecValue" means give me the equivalent decimal value column based on the hex value in the "HexLookupN" (units digit) table instance.
   • "HexLookup T::DecValue" does a look up for the tens unit. I multiply the value by 16 an add it to the ones value. Remember the hex #FF (F=15) means 15*16+15
   • I look up the hundreds place decimal value and multiply it by 16^2 (256), then the thousands place decimal and multiply it by 16^3 (4096).
   • I add up the results of each converted decimal digits times its appropriate power of 16.The calculation is complete.

Prelude about the Problem

In terms of handling non-English characters, apps come in two types (at least on the Mac). There are apps which switch fonts behind the scenes without telling you, and those which don't...but then you have to guess which font to use.

To take a concrete example, if I switch from the English keyboard to Japanese input in FileMaker, the font will automatically switch to one of the Japanese fonts. In theory, once I switch back to English, I should return to the original font (except when I don't...we'll get to that). The same principle applies in most apps including TextEdit, FileMaker and so forth. In contrast, if I switch to Japanese input in Adobe Photoshop, I also have to change fonts.

In theory, the automatic font switching sounds nice except when 1) when the font doesn't change back after typing the exotic character (this happens a lot in phonetic transcription and elsewhere) or 2) you're trying to figure if font X actually has that glyph (or whether it's the illusion of font switching in action. With the Adobe products, the manual font switching means you know exactly which font you are using at all times, which is important in desktop publishing.

FileMaker

For instance...I uploaded a version of the UCD Unicode files into FileMaker so I would have a searchable reference locally. An additional function is that I can display glyphs in different fonts for comparison. I have most of the mega fonts selected, but few fonts have everything, so I know there are gaps.

However, because FileMaker switches fonts behind the scenes, I can't always be sure if font X actually has that glyph. If I see a bunch of boxes with identical glyphs, I can suspect an unannounced font switch...but to what?

Solution

The best solution now is to copy and paste the text into TextEdit then open up the font formatting palette (Command+T), and see what it says. Kind of dorky, but still more information than I had.

For the record, I understand why FileMaker is set up this way. For most purposes, you don't want your data entry operators to fidget with fonts. However, you can get inconsistent results if you are not careful. For instance, once I do switch to Japanese, I get the Japanese font, but if I return to English...I still get the Japanese font. I know Japanese fonts contain Latin characters, but the formatting is almost always NOT the one I intended.

It would be nice if FileMaker and the other apps (including Microsoft Office) could return you to your original English font formatting after your exotic sidetrip to the higher code points of Unicode.

What these are

The superscript a/o (sometimes underlined) are abbreviations for ordinal numbers used in Spanish, Italian and Portuguese similar to English -th (as in "4th, 5th, 6th.."). The use of "o" vs "a" depends on the gender of the noun. For instance, the "1st American woman" would be 1ª americana in Spanish and the "1st American man" would be 1º americano. The 5th Amercan woman and man would be 5ª americana/5º americano.

The Codes

I got a request for putting codes for these on the Penn State Web Computing with Accents Web site in various locations, so I thought I would summarize the codes here.

But Wait There's More

But in the land of Unicode, there's always more to know...such as that in Spanish 1º primero '1st.masc' or '1º' may be shortened to primer which can be abbreviated as '1^er'...or that you may write octavo 'eight.masc' as 8º or 8.º or possibly 8^vo...although Google tends to have more instances of 8º.

What's important though is that only º and ª have their own code points in Unicode. For English ^{-th, -nd, -rd} or Spanish ^-vo,-er you have to rely on the old fashioned SUP (superscript) tag or its equivalent in CSS.

The latest Unicode Standard, Version 5.2, was released at the beginning of October, 2009. A lot is added each standard, but I confess that the most noteworthy for me was that an Egyptian Heiroglyphic block (U+13000 to U+1342E) was added. It was certainly the largest block added at 1071 code points.

Additional code points added included blocks for Avestan, Old South Arabic, Samaratian, Imperial Aramaic, Inscriptional Parthian, Old Turkic. In addition, supporting characters were added for the Coptic, Devanagari (esp Vedic support), Hangul (Old Korean), Phonecian and other ancient script blocks.

In South and Southeast Asia, support was added for Javanese, Tai Tham, Lisu, Kaithi, Meitei Mayak, Myanmar (new points), New Tai Lue (new points) and others. In other regions, a new Caniadian Aboriginal Syllabics Extended block was created with 80 additional code points. Some African scripts were also encoded including the Banum script and Rumi numerals. Additions were also made to various math and symbol blocks.

For a complete list of changes, see the information on the DerivedAge.txt file (scroll to end) and Revised Unicode 5.2 charts. In terms of support, there may be freeware (or commercial) fonts available, but time will be needed to develop the input utilities and then for these glyphs to be incorporated into major operating systems.

Until then...there's always Unicode 6.0.

Defining Emoji

There were lots of interesting sessions at last week's Unicode conference, but the one that I think non-experts can relate to the most was the one about Emoji or those little tiny icons popular in Japanese e-mail messages.

A rough translation of emoji might be emoticon, but the range of images goes way beyond smiley faces to include weather symbols, hearts, beer steins, sports icons, high heels,fast food, astrological signs, warnings, hand gestures and bikinis.

Why Unicode?

It's good to catalog and standardize any symbol set, but in this case economic necessity is driving this campaign. Specifically, Google and Apple (and its iPhone) who want to expand more into the Japanese market.

According to our presenters, the three major Japanese cell phone carriers all support emoji, and these images are popular with most adults (even the ones over 30). It's an important enough feature that iPhone (and iChat), Gmail and even Twitter support emoji.

But really it would be good to support one encoded set of emoji, not a hack of three emoji encodings from the Japanese cell phone carriers...hence the need for a unified encoding which combines those items already encoded (e.g. zodiac symbols) with symbols not currently in Unicode.

Remaining Issues

Because no Unicode script block is free of quirks, I document the issues overheard at the conference and at the Web. Namely:

1. Color - Real emoji have colors (really bright ones), but the spec is in black and white. This makes sense because the rest of Unicode is also in black and white. Plus you will have more options to add the colors you want!

2. 5-Digit Code Points - Or more technically, the new glyphs will be assigned a number above U+FFFF (i.e. not in the BMP or Plane 0). Not surprisingly, many mobile devices are limited to U+FFFF and below. The committee's comment was that they expected that moble developers would learn to overcome this restriction...because they really are running out of room in the U+0000-FFFF range. That may be good news for anyone wanting to transmit the ancient scripts over cell phones. You never know when you need to access a Mycenaean Greek text away from the office or when the next Linear B revival may happen.

3. There's a Jailbreak App for that - When researching this article I encountered articles about tricks for enabling emoji on non-Japanese iPhones, not all of which were legit. For a while, Apple was discouraging use of emoji outside of Japan so it was hiding the emoji. Fortunately, there is a legal way to enable emoji now (both a trick and an app).

So there you have it - thanks to the great folks at Google and Apple, we will all be able to standardize the addition of cute icons in our online communication...or at least we will have a documented explanation of what they were for future generations. Trust me, in about 500 years, we will need it.

• MacKeyboardTutorialPyatt.2.pdf (Wednesday)
• KeyboardLayoutFiles2.zip (Wednesday)
• Unicode33PyattLogic.ppt (Friday)

New operating systems often mean new i18n toys to play with and even through the upgrade from Apple 10.5 (Leopard) to 10.5 (Snow Leopard) is not supposed to be full of new features, there are, in fact, new features scheduled for the upgrade.

According to the Apple Snow Leopard Enhancement page, 10.6 will include:

• Redesign of Pinyin Chinese input with faster speed and enhanced dictionary
• Improvements to handwritten Chinese input
• New Asian fonts - Heiti SC, Heiti TC, Hiragino Sans B.
• New generic monospace font Menlo to be used in applications such as Terminal
• Enhanced RTL support including split cursor option to show text direction in documents with bidirectional text
• General Text substitution (e.g. (c) to ©) across applications. Could be handy for a lot of situations when you need to enter an unusual symbol. This already exists in Microsoft Office (Mac/PC).

But I almost missed the big one - the International pane in the System Preferences has been redesigned and will now be the Language and Text pane, presumably with more features. There may be other enhancements in the works that are too minor to be announced (or at least too minor for most people), but there may be more things to find out.

How will they work? Alas, no details from Apple yet. I guess we won't know until we know....

A useful utility on the Mac is the Unicode Hex keyboard which allows you to press Option plus any four digit Unicode code to get that character.

For instance, if you need to enter the rarely seen archaic Roman numeral symbol for 5,000 (ↁ), you could look up its Unicode character number (U+2181), then activate this keyboard then type Option+2181 and generate the code (assuming the correct font is loaded).

But a lot of ancient scripts are in Plane 1, meaning they have Unicode values with five digits (i.e. U+10000 or higher). In Unicode world, adding the fifth digit means that some processes go slightly awry, and the Unicode Hex keyboard is one of them. Suppose I want to input Phonecian character Alf (Aleph) (𐤀 or an A on its side), which is U+10900. If I enter Option+10900 on the Unicode Hex keyboard, I will not get Alf, but ႐ instead.

Note: Characters U+0000 to U+FFFF are in Plane 0 or the BMP (Basic Multilingual Plane). A lot of systems are set up to deal with BMP only, but need special support for codes beyond U+FFFF. The four-digit restriction corresponds to 16-bytes which a constraint in older systems. If you're not a programmer, let's just say it's a long story and leave it at that.

It turns out that the Unicode Hex keyboard has a four-digit limit. To get around it, you can break U+10900 into two 16-byte (i.e. 4-digit) sequences, also known as as a UTF-16 Surrogate Pair. For U+10900, the surrogate pair is D802+DD0C. So in the Unicode Hex utility, you can now do this.

1. Hold down the Option key.
2. Type D802+DD0C, where the + means type the Plus sign.
3. Release the Option key.

I bet you're asking - how did she get from U+10900 to D802+DD0C? There is an algorithm, but in this case I got it by opening the Character Palette, finding the character I wanted and mousing over it. When you do that, the Unicode code point appears along with its surrogate pair in parentheses.

Of course, you could also directly Insert the character with the palette, but actually there are times when the Insert doesn't quite work (at some points in the careers of my laptops, I have corrupted my Character Palette so badly, it refused to play with me anymore).

Although this utility seems a little limited at the moment, if there's one thing I have learned is that Unicode no trick has ever gone to waste.

The biggest sensation in Unicode land these days is that the Korean script Hangul (or Hangeul/Han'gŭl depending on your transliteration preferences) has been adopted by the speakers of Cia-Cia in the nation of Indonesia. This will be the first time any language other than Korean has adopted Hangul as it's writing system, so it is a cultural triumph for them.

What's interesting is how this decision happened. The standard press releases are not giving much information and even the linguistic community is a little perplexed. It's actually more interesting if the Wikipedia report that Cia-Cia was formerly written in the Arabic script (specifically the Jawi variant in Indonesia) is accurate. According to Ethnologue, the population is still mostly Islamic, so there shouldn't be a religious reason to switch.

So what about it? First, let's discuss the switch from Arabic. Actually a lot of Muslim communities including speakers of Hausa, Swahili, Malay and Turkish have switched from Arabic to the Latin alphabet. Malaysia and Indonesian are two countries following this trend, although the Jawi/Arabic script is still used in some religious and cultural contexts. There may be a variety of reasons for this including European colonial policy or the perception that the Latin alphabet is easier to learn and enhances literacy (Turkish). A move to the Latin alphabet may also represent a move towards a secular government (as in the case of Turkey).

It should also be mentioned that the Arabic script must be modified heavily when it is used for non-Semitic languages if all the sounds are to be represented. If you look at the Omniglot Jawi chart for example, you will see that many consonants have the same shape but with with different patterns of dots to indicate the differences. This also happens in the Latin alphabet (e.g. n vs. ñ in Spanish), but if Jawi also includes the different letter shapes depending on word position as Arabic, then the script becomes more complex.

Cia Cia is unique though in switching to something other than the Latin alphabet. One reader commented that this may be due to the fact that in South and Southeast Asia, a language gains social status by having its own script. In Indonesia, Balinese, Javanese and Sundanese have their own historic scripts. Although these scripts may not be used on an everyday basis, they do show that there is a cultural tradition having nothing to do with the West.

In theory, Cia Cia could adopt one of these scripts or one from India (e.g. Devanagari) would would probably be a good fit, but none would probably be perceived as being unique in Indonesia. On the other hand...no one else in Indonesia is using Hangul. It is very unique. Fortunately, Hangul is probably a good fit. Although the forms are somewhat angular like Chinese writing, the underlying principles are actually very similar those used in India and Southeast Asia (with some differences of course).

There's another benefit to Hangul over scripts like Javanese and Balinese and that's enhanced Unicode support. Korea has been fortunate enough to have the economic and political influence for developers to develop functional encoding schemes, fonts and input utilities for Hangul. Many Southeast Asian scripts are still catching up Unicode wise.

Whether this is the reason Cia Cia switched to Hangul or not, I wish them the best of luck. I think there are lots of people now invested in the success of this project.

Here at Penn State my duties include being an accessibility guru as well as being a Unicode guru, and not too surprisingly, Unicode can enhance accessibility in some situations. And not just in the abstract "standards enhance accessibility" but more concretely as in:

It's An Encoded Character, Not a Font Trick

We all know that relying on fonts to display characters (e.g. the use of the Symbol font for Greek characters) is a Bad, Bad Idea, but it's even worse for a screen reader. Consider the expression θ = 2π. In the old Symbol font days, this might have been coded as:

<p> <font face="Symbol">q = 2p</font></p>

And guess what the screen reader would read - Q equals 2 P. Since the screen reader is essentially "font blind", the underlying text is what is read. Hence the Unicode correct code below is preferred:

<p> θ = 2π</p>

-OR-

<p> &theta; = 2&pi;</p>

If you think about it, the screen reader is a good tool for conceptualizing how characters (and their variants) may function semantically in different contexts.

I should mention that screen readers can get confused with a Unicode character if it can't recognize it, but that's more of a dictionary problem than a Unicode problem. For Jaws, it is possible to install .sbl pronunciation files to increase the character repertoire, especially for math and science.

It's Text, Not An Image

Perhaps the biggest advantage for Unicode though is that it allows characters that used to be embedded in images to be just plain text. For instance you could embed the following equation for the volume of a sphere:

Text

V = 4/3πr³

Image

Consider what happens though if a low-vision reader (or a middle aged reader with decrepit eye sight) needs to zoom in on the text. As you will see in the screen capture below, the image will pixelate while the text remains crisp.

Zoomed Text vs Zoomed Image

When you combine Unicode with creative CSS, you can see the possibilities for replacing images, including buttons with text. Not only is this more accessible, but it also results in smaller file sizes and is easier to edit.

Hearing Impaired Users

Unicode is actually important for these users because they need to read text captions or transcripts for video and audio. Once you get beyond basic English (e.g. Spanish subtitles)...well you know Unicode will be important.

Motion Impaired Users

For these users, the issue probably isn't so much reading text as being able to input it - which is the job of developers of operating systems and software. For motion impaired users, a good generalization is that keyboard access is better than using the mouse which requires a little more hand control. In the past I've commented on usability of various inputting devices, but since most do rely on key strokes, there are really no major complaints here.

One audience I didn't touch was color deficient vision, but except possibly for the Aztec script (which isn't even in Unicode yet)...it's not too much of an issue.