Now the default version does not appear to have to hex to decimal conversion built in (not even in FileMaker 10), so here's my (updated) solution.

1. In the main table corresponding to the list of code points, I created a field for the Hexadecimal Unicode code point value. I'll call this HexValue for now. It must be a Text field. You can create a Decimal field (Calculated), but you won't be able to fill in the formula yet.
2. Then I created a second table to store the correspondence between a hex digit (0-F) and its decimal value (0-15). The HexValuefield is Text, but the DecValue field is a Number. See the sample table below (some values skipped).

   HexValue (Text)	DecValue (Number)
   0	0
   1	1
   2	2
   3	3
   4...9 (1 row each)	4...9
   A	10
   B	11
   C...E (1 row each)	12...14
   F	15

3. To do all the conversions, you need to extract the text value of each position in the code point. So, I created fields corresponding to the value for each place in the hex code point as shown in the list below. I'll explain the formulas below.

   Note: In case you're wondering, the name of the places are semi-inspired by Roman numerals and algebra.

   • Rightmost digit Units (n) : nhex = Right(HexValue;1)
   • Penultimate digit (t) : thex = Left(Right(UnicodeHex;2);1)
   • Antepenultimate digit (c) : chex = Left(Right(UnicodeHex;3);1)
   • 4th from right (m): mhex =Left(Right(UnicodeHex;4);1)
   • 5th from right (d): dhex =If(Length (UnicodeHex)>4;Left(Right(UnicodeHex;5);1);"0")
   • 6th from right (x): xhex = If(Length (UnicodeHex)>5;Left(Right(UnicodeHex;6);1);"0")

   The challenge for modern Unicode is that code points now come in variable lengths (4-6), so if you count from the left you can't always know you are the appropriate digit. That means you have to count from the right, but there's no simple formula for picking the 2nd digit from the right. My solution is to take a rightmost chunk then count in from the left. So to get the 3rd hex digit from the left, I take the right most 3 digits, then find the leftmost digit in that chunk (hence the embedded left(right) formulas).

   I also have to check to see if the length is greater than 4. When the length is 4, some digits are filled in with the value 0, otherwise you do a string extraction. Hence the formulas for dhex and xhex use conditional logic. Hopefully though, if Unicode adds more digits, these formulas will continue to work (unlike my original attempt which only assumed 4 digits in the code point.

4. To convert each extracted digit to its decimal version. I need to set up some Relationships between tables so that each extracted digit can look up the decimal equivalent. For each of the intermediate digit fields above, I created a link to an instance of the Hexadecimal Lookup table (there are 4 instances total). It's important to make sure each instance has a name you can remember later; mine mention which digit I am working on. See the Relationships diagram below.
   
5. Now we can finally get that decimal value! If you haven't already, create a DecimalValue field and make it Calculated.
6. Here's my calculation. I'll explain what the parts mean below
   HexLookup N::DecValue + 16*HexLookup T::DecValue + 16^2* HexLookup C::DecValue + 16^3*HexLookup M::DecValue + 16^4*HexLookup D::DecValue+16^5*HexLookup X::DecValue
   • "HexLookupN::DecValue" means give me the equivalent decimal value column based on the hex value in the "HexLookupN" (units digit) table instance.
   • "HexLookup T::DecValue" does a look up for the tens unit. I multiply the value by 16 an add it to the ones value. Remember the hex #FF (F=15) means 15*16+15
   • I look up the hundreds place decimal value and multiply it by 16^2 (256), then the thousands place decimal and multiply it by 16^3 (4096).
   • I add up the results of each converted decimal digits times its appropriate power of 16.The calculation is complete.

In terms of handling non-English characters, apps come in two types (at least on the Mac). There are apps which switch fonts behind the scenes without telling you, and those which don't...but then you have to guess which font to use.

To take a concrete example, if I switch from the English keyboard to Japanese input in FileMaker, the font will automatically switch to one of the Japanese fonts. In theory, once I switch back to English, I should return to the original font (except when I don't...we'll get to that). The same principle applies in most apps including TextEdit, FileMaker and so forth. In contrast, if I switch to Japanese input in Adobe Photoshop, I also have to change fonts.

In theory, the automatic font switching sounds nice except when 1) when the font doesn't change back after typing the exotic character (this happens a lot in phonetic transcription and elsewhere) or 2) you're trying to figure if font X actually has that glyph (or whether it's the illusion of font switching in action. With the Adobe products, the manual font switching means you know exactly which font you are using at all times, which is important in desktop publishing.

FileMaker

For instance...I uploaded a version of the UCD Unicode files into FileMaker so I would have a searchable reference locally. An additional function is that I can display glyphs in different fonts for comparison. I have most of the mega fonts selected, but few fonts have everything, so I know there are gaps.

However, because FileMaker switches fonts behind the scenes, I can't always be sure if font X actually has that glyph. If I see a bunch of boxes with identical glyphs, I can suspect an unannounced font switch...but to what?

Solution

The best solution now is to copy and paste the text into TextEdit then open up the font formatting palette (Command+T), and see what it says. Kind of dorky, but still more information than I had.

For the record, I understand why FileMaker is set up this way. For most purposes, you don't want your data entry operators to fidget with fonts. However, you can get inconsistent results if you are not careful. For instance, once I do switch to Japanese, I get the Japanese font, but if I return to English...I still get the Japanese font. I know Japanese fonts contain Latin characters, but the formatting is almost always NOT the one I intended.

It would be nice if FileMaker and the other apps (including Microsoft Office) could return you to your original English font formatting after your exotic sidetrip to the higher code points of Unicode.

Additional code points added included blocks for Avestan, Old South Arabic, Samaratian, Imperial Aramaic, Inscriptional Parthian, Old Turkic. In addition, supporting characters were added for the Coptic, Devanagari (esp Vedic support), Hangul (Old Korean), Phonecian and other ancient script blocks.

In South and Southeast Asia, support was added for Javanese, Tai Tham, Lisu, Kaithi, Meitei Mayak, Myanmar (new points), New Tai Lue (new points) and others. In other regions, a new Caniadian Aboriginal Syllabics Extended block was created with 80 additional code points. Some African scripts were also encoded including the Banum script and Rumi numerals. Additions were also made to various math and symbol blocks.

For a complete list of changes, see the information on the DerivedAge.txt file (scroll to end) and Revised Unicode 5.2 charts. In terms of support, there may be freeware (or commercial) fonts available, but time will be needed to develop the input utilities and then for these glyphs to be incorporated into major operating systems.

Until then...there's always Unicode 6.0.

There were lots of interesting sessions at last week's Unicode conference, but the one that I think non-experts can relate to the most was the one about Emoji or those little tiny icons popular in Japanese e-mail messages.

A rough translation of emoji might be emoticon, but the range of images goes way beyond smiley faces to include weather symbols, hearts, beer steins, sports icons, high heels,fast food, astrological signs, warnings, hand gestures and bikinis.

Why Unicode?

It's good to catalog and standardize any symbol set, but in this case economic necessity is driving this campaign. Specifically, Google and Apple (and its iPhone) who want to expand more into the Japanese market.

According to our presenters, the three major Japanese cell phone carriers all support emoji, and these images are popular with most adults (even the ones over 30). It's an important enough feature that iPhone (and iChat), Gmail and even Twitter support emoji.

But really it would be good to support one encoded set of emoji, not a hack of three emoji encodings from the Japanese cell phone carriers...hence the need for a unified encoding which combines those items already encoded (e.g. zodiac symbols) with symbols not currently in Unicode.

Remaining Issues

Because no Unicode script block is free of quirks, I document the issues overheard at the conference and at the Web. Namely:

1. Color - Real emoji have colors (really bright ones), but the spec is in black and white. This makes sense because the rest of Unicode is also in black and white. Plus you will have more options to add the colors you want!

2. 5-Digit Code Points - Or more technically, the new glyphs will be assigned a number above U+FFFF (i.e. not in the BMP or Plane 0). Not surprisingly, many mobile devices are limited to U+FFFF and below. The committee's comment was that they expected that moble developers would learn to overcome this restriction...because they really are running out of room in the U+0000-FFFF range. That may be good news for anyone wanting to transmit the ancient scripts over cell phones. You never know when you need to access a Mycenaean Greek text away from the office or when the next Linear B revival may happen.

3. There's a Jailbreak App for that - When researching this article I encountered articles about tricks for enabling emoji on non-Japanese iPhones, not all of which were legit. For a while, Apple was discouraging use of emoji outside of Japan so it was hiding the emoji. Fortunately, there is a legal way to enable emoji now (both a trick and an app).

So there you have it - thanks to the great folks at Google and Apple, we will all be able to standardize the addition of cute icons in our online communication...or at least we will have a documented explanation of what they were for future generations. Trust me, in about 500 years, we will need it.

What's interesting is how this decision happened. The standard press releases are not giving much information and even the linguistic community is a little perplexed. It's actually more interesting if the Wikipedia report that Cia-Cia was formerly written in the Arabic script (specifically the Jawi variant in Indonesia) is accurate. According to Ethnologue, the population is still mostly Islamic, so there shouldn't be a religious reason to switch.

So what about it? First, let's discuss the switch from Arabic. Actually a lot of Muslim communities including speakers of Hausa, Swahili, Malay and Turkish have switched from Arabic to the Latin alphabet. Malaysia and Indonesian are two countries following this trend, although the Jawi/Arabic script is still used in some religious and cultural contexts. There may be a variety of reasons for this including European colonial policy or the perception that the Latin alphabet is easier to learn and enhances literacy (Turkish). A move to the Latin alphabet may also represent a move towards a secular government (as in the case of Turkey).

It should also be mentioned that the Arabic script must be modified heavily when it is used for non-Semitic languages if all the sounds are to be represented. If you look at the Omniglot Jawi chart for example, you will see that many consonants have the same shape but with with different patterns of dots to indicate the differences. This also happens in the Latin alphabet (e.g. n vs. ñ in Spanish), but if Jawi also includes the different letter shapes depending on word position as Arabic, then the script becomes more complex.

Cia Cia is unique though in switching to something other than the Latin alphabet. One reader commented that this may be due to the fact that in South and Southeast Asia, a language gains social status by having its own script. In Indonesia, Balinese, Javanese and Sundanese have their own historic scripts. Although these scripts may not be used on an everyday basis, they do show that there is a cultural tradition having nothing to do with the West.

In theory, Cia Cia could adopt one of these scripts or one from India (e.g. Devanagari) would would probably be a good fit, but none would probably be perceived as being unique in Indonesia. On the other hand...no one else in Indonesia is using Hangul. It is very unique. Fortunately, Hangul is probably a good fit. Although the forms are somewhat angular like Chinese writing, the underlying principles are actually very similar those used in India and Southeast Asia (with some differences of course).

There's another benefit to Hangul over scripts like Javanese and Balinese and that's enhanced Unicode support. Korea has been fortunate enough to have the economic and political influence for developers to develop functional encoding schemes, fonts and input utilities for Hangul. Many Southeast Asian scripts are still catching up Unicode wise.

Whether this is the reason Cia Cia switched to Hangul or not, I wish them the best of luck. I think there are lots of people now invested in the success of this project.

Sorin Sbarnea

I was interested in adding a comment to your article from http://www.personal.psu.edu/ejp10/blogs/gotunicode/2009/06/iphone-30-unicode-support-stil.html but I was not able to find any way of adding a comment on the page.

IPhone 3.0 does have full support for all Romanian characters, previous versions did not. In order to be able to enter special characters you need to use a Romanian keyboard. Due to ergonomics they decided to not include all character on one keyboard because it would be too hard to use. Also I want to say that the drag model is genuine and very practical - after few days you'll see that is much smarter and more precise than the old click model.

I remember that I submitted 3 bugs on Apple for Romanian support, they solved one in 2.1 and the rest in 3.0 - so all you have to do is to complain to them ;)

Just to give you an example: in the first implementation of the Romanian keyboard the Ă character was the last on the right of the list of accented A characters after characters that are not used in Romanian - I complained to them explaining the reasons and now they solved it. I don't know if I was the cause or the only one complaining but they solved it anyway. Also - I wasn't able to find *any* error in Romanian translation of the iPhone - this is something very good - let's say I wasn't expecting this level of quality for Romanian translation.

Comments

The lesson here is that Apple does listen...if you know where to send input. Still not convinced about dragging on the iPhone in general (except for solitaire), but I can be stubborn. I'm glad Sorin is a satisfied Apple customer.

As to the question about submitting comments - I disabled mine because about 99% of them were offers for land in Florida or pharmaceuticals I cannot use. For now, please feel free to contact me at ejp10@psu.edu. If I hear an outcry for commenting, I may change my mind.

Heady stuff for technologies we normally associate with most insipid of Internet messaging ("OMG - The Orioles lost again?!?"). I'm glad Facebook and Twitter were there, but I suspect that some of the most important messages were in Persian (Farsi) and were made possible by another less glamorous technology - Unicode. Both Facebook and Twitter have had underlying Unicode support in the beginning, so assuming your system had the right fonts, you could communicate in any language from Persian to Igbo and then some.

Although I am normally a symbol geek in my love for Unicode (goes well with my lifelong obsession with fonts, foreign language and exotic characters), at times like these I realize that Unicode is an important tool to the dream of the Internet enabling anyone, anywhere to speak out and be heard. If you are not a symbol geek, but wonder why Unicode is important...I think bloggers and Tweeters in Iran, China and everywhere can show you the answer. Unicode makes it possible for everyone to be heard...even if you haven't had the chance to learn English.

Postscript: English Digital Divide

In some countries there is a real digital divide based on language - that is those who have learned a major language such as English, French or Spanish or Chinese and Arabic are able to use the Internet while others who only know a relatively under supported language do not have little to zero access.

For instance, I asked a scholar at a Sri Lankan university how they computed in the Sinhala script, and his answer was that all computing was assumed to be in English (partly because Sri Lanka used to be the British colony Ceylon). I was a little startled, but it makes sense. Until recently, I suspect that only a few people or institutions in the upper economic tiers could have afforded computers and they were likely already educated in English. Since English support is built in, it might seem a waste to work in support for a "local" script. Still, I think a lot of people and organizations understand the importance of Unicode in increasing access (and preserving local languages) and are working to provide low-cost utilities for these communities.

As a warning, I have to admit that I'm a little behind the times in mobile computing, so bear with me if I repeat something you already know. Still, I'm not seeing this information all in one place it it may be a good over (at least for me).

The good news is that there does appear to be more character support, but the feature is still too well-hidden (I really had to work hard to find Welsh support). The iPhone also fails my test for general Unicode readiness because I am not able to yet enter phonetic characters like /ŋ,ɛ,ʃ/ (if nothing else that would kill the iPhone as a remote data entry device). However I doubt the iPhone is really not alone in that area.

So if you are wondering what I am talking about, let me discuss in context:

Baseline Support

Unicode data and display for major languages is generally supported. If Safari can display your Unicode Webpage, it will appear correctly on your iPhone...assuming that the built in fonts support the character. Further, if you have entered/purchased an exotic title in iTunes, it will appear correctly in your synched iTunes list on the iPhone.

Entering Accents

The next challenge is entering some exotic characters into e-mail or a notes application. If you are dealing with Roman characters, iPhone does have some support, but not as much as I would like. The easiest non-English characters to find are foreign currency symbols like £ (pound), ¥ (yen) and € (euro). You typically access these by clicking the the symbol set (often right after the numerals).

While I was able to figure that out, I admit to being stumped as to how to enter accented letters such as Spanish ñ or French è. Fortunately a quick Google search turned up some help sites including this blog entry from Pixelcoma. As you can see, the trick is to hold down a base key such as N or E to see the options for accented characters.

The trick though is that you have to drag your finger across to the right character. You can't hold and double tap as I tried to do. Oops

As stated earlier, there are more options in the palette than in previous earlier versions. For instance, the Pixelcoma A options show A,À,Á,Ä,Æ,Ã,Å,Ą which already covers lots of Western and Central European languages, but Version 3 does add Ā (macron) which is good for Japanese Romaji, Hawaiian, Maori and Latin with long marks (I know there are Latin users out there). I assume that there are other important additions at the other base letters.

However, there are still apparent gaps such as Welsh accented W and Icelandic þ,ð/Ð as well as Romanian Ă, Turkish Ğ,Ş and İ, Latvian Ņ and other really exotic accented letters. It turns out that many are actually in keyboard options installed on the iPhone with additional characters. It still can feel like these languages are "second" class in comparison to Spanish, French and German (at least Polish, Czech and Hungarian have been "mainstreamed" which is a plus).

Before I leave this section though, I do have a comment for future devlopers:

Future developers - if you want to wow your audience with global accent support, you may want to start here at the Wikipedia Latin palette.

That way we can avoid the agonizing incremental addition of accented letters as individual user communities step forward. Why not be comprehensive at the start - like the Apple U.S. Extended keyboard (which is major reasons I still love Apple).

As much as I kvetch though, I don't think the iPhone is worse than any other U.S. mobile device. A forum post for Blackberry mentions holding down a vowel and moving a trackball. ¡Qué divertido!

Other Keyboards

As mentioned previously, if your character is not available in the accent palette, you may need to activate the keyboards (just like in the laptop/desktop). On the iPhone, you access these by clicking the Settings app, then going to General Settings then International. A number of keyboards for languages like Chinese, Japanese, Russian, Hebrew, Arabic as well as Icelandic, Turkish, Latvian are available (still no Welsh, unless it's hiding under the U.K. keyboard (yes it is !)).

This adds a globe icon (like the one below) to the usual iPhone keyboard and allows you to switch between keyboard modes. I just switched to the U.K. keyboard and behold, I found the ŵ under the W key (but now the ¥ key is missing).

Icon for International Keyboards on iPhone

What I Really Want...

Actually it's not necessarily more accented letters as I hold down a key. My thumb is shuddering at how the potential pain of dragging or trackballing additional accents on top of the other precision maneuvers required for English texting. I actually want several things

First, slightly better keyboard designs. The iPhone Google keyboard has the right idea when it makes the @ sign and .com extension basic keys. We already have options for switching on canned keyboards, but what if we had options for customizable keyboards. Maybe one with a "symbol" dock into which you drag the characters or phrases you need from a master slot (this way Americans learning Welsh CAN have their accented W's). Maybe you can reshuffle as well (like killing the \ key if you only synch with a Mac).

But I have to confess that I really want to be able to plug my iPhone into a keyboard. IThe touch interface is fine short small tasks on the run (like looking up movie times or weather by zip code), but still not so great for longer data entry or note taking tasks. I know it's Palm Pilot, but I am at a stage where I would like to ditch the laptop for short meetings and only carry a mobile device and take notes. I note that there are there are hacks out there already...despite the useful shortcuts provided. That should be a sign for Apple and other makers of mobile devices that the need is out there (bummer dudes).

It goes without saying that if true Mac keyboard integration comes, it should come with support for the U.S. Extended and other keyboard variations Apple and the user community have concocted (Windows users can use the U.S. International keyboard for the Mac).

A final wish though is better documentation. The Unicode support for iPhone is decent, but it's quite a chore tracking it all down through numerous user blogs and guessing. I know Apple relies somewhat on it's "intuitive" interface to help users through, but, for whatever reason, Unicode support is rarely intuitive. You just have to know where things are. I'm glad there's a user community out there but from the lack of documentation (especially in comparison to Microsoft) it seems like Apple doesn't care about these issues (when I think they really do).

Microsoft has various Globalization sites (in English), so why can't Apple (or at least one I can find)? Is it because we're in the U.S? To me, It's a little condescending to me to assume that just because I live in the U.S. I will rarely need to enter non-English text. In fact, I type something "non-English" nearly every day.