Got Unicode?

AATSEEL (American Association of Teachers of Slavic and East European Languages) has just posted a set of links to "Medieval Slavic Fonts" for Old Church Slavonic, Glagolitic and Blackletter.

See http://www.aatseel.org/medieval_slavic_font for more information

List includes Unicode fonts and older non-Unicode fonts

As I've been reporting in recent entries, I've been working with a symbolic logic course which has been using various exotic symbols including double struck P (ℙ). Since every Unicode point seems to have its own story, I thought I would report some of the ineresting challenges for this character.

Finding It

When you are discussing a topic with lots of different symbols, you soon realize that in terms of Unicode, they will come from multiple blocks. For instance double struck P is from the Letter Like Symbols block (starts at U+2100), while other math symbols may be in Arrows block, the Number Forms block, the Mathematical Operators Block or possibly the Dingbats Block. You can see from the Unicode Org Symbols and Punctuation Chart just how many blocks are involved.

Although a user doesn't normally have to know the Unicode point value, because many insertion tools such as the Windows Character Map, Mac Character Pallete or others are organized primarily by block, you do have to sort of have an idea of how blocks work.

Rarity

Fonts with a robust set of math symbols are still pretty rare, and sometimes the letter like symbols are even rarer. At one point I had ℙ (P) pulling from one font and ℚ (Q) from another...interesting. Below are some fonts I know have doublestruck letters like ℙ,ℚ.

• Windows/Mac Leopard - Arial Unicode MS
• Macintosh OS X - Apple Symbol, Hiragino Mincho Pro W3 (Japanese), Hiragino Mincho Pro W6 (Japanese ), Lucida Sans
• Unicode Symbols
• Hindsight Unicode
• Chrysanthi (Chryʃsanþi)

Formatting Issues

Normally I try to avoid font and size specifications, but double struck P is an interesting counterexample. One challenge is that because the legs are hollowed out, it has a much lighter visual appearance than say normal P. My base text is 12 px on the Web, but for the double struck P, I decided to bump up the size to about 16 px (in a standards-compliant way of 1.3 em).

The other issue was selecting font faces. I wanted one with thick double legs - if you look at the font chart below from my Mac, you'll see that some fonts had some very skinny legs.

I also prefer the serif fonts in this case since I personally believe serifs help inexperieced users in reading unfamiliar scripts (in this case undergraduate college students). For this course, I'll probably point students to some freeware fonts I like

Math symbols can stretch the boundaries of Unicode display technology, but not as much as some other related blocks like Letterlike Symbols the home of such symbols as ℙ (double struck P, see image below), ℚ (double struck Q), and even the pharmacy prescription symbol (℞).

Double struck letters in particular are used in different branches of mathematics to respresent, for instance, the set of all real numbers (double struck R) or in symbolic logic to symbolize any atomic proposition. See the table below for different double struck letters and their Unicode values. See the Penn State Math Symbol chart for other common letter like symbols of math.

I was able to upgrade to Leopard recently on my Mac which means I'm able to manipulate a working version of Arial Unicode MS for the Mac...yeah.

Web Display

My blog actually switched to Arial Unicode because of the way I had coded the CSS. It was very legible, but the x-height seemed smaller in comparison to the Apple Lucida Grande - so I reordered the priority. I will have to see if I can download Lucida Grande onto Windows via the Windows Safari download.

Back to the Logic Symbols in Word

Most of my recent Unicode adventures have been about inserting logic symbols like (∨,∧,⊃) into Word (and later Excel). My main struggle has been that if I insert them from the Character Palette, the font switches to Symbol... which is OK until I start typing English. At that point I will stop outputting the English alphabet and σταρτ ουτπυτιν τηε γρεεκ αλπηαβετ. Greek is great...unless you're typing English text. I was using the left arrow key quite a bit.

Now that Microsoft has developed a working version of Arial Unicode MS, I can input the symbols without switching over to Greek. The only gotcha is that I have to shif old logic symbols out of their pre Arial Unicode fonts (thank goodness for keyboard shortcuts). What I'm hoping is that I can bypass the big font switch in Windows word too.

So I'm happy to say that we're adding another small step towards Unicode compatibility. Finally I can have logic symbols in a non-Greek, non-Japanese, non-Chinese font!

A while ago, I was writing about the difficulty of defining some language tags like Cantonese because even though it's called a dialect, it's really a separate language.

The SIL group is using a new term I think should become more common - the macrolanguage. A macrolanguage is basically a set of related languages that share a common "identity" even though speakers can't normally understand each other.

Macrolanguages happen when language spreads to different regions and changes, but the cultural or political unity remains. Other macrolanguages include Arabic, Cree, Hmong, Quechua (as spoken in the Incan Empire), and Norweigian. I suspect that you could thrown in some other candidates like German and Italian - (we'd have more if the Roman Empire had made it to the 21st century.)

In any case, The ISO-639-3 language tag standard has a set of macrolanguage mappings which show how different related languages can map to each other so that either Mandarin Chinese (cmn) or Cantonese (yue) can also be called Chinese (zh or zho)

I really hope this term takes hold...because I really think it will simplify other discussions about language tags. After all, it was just this year that a language technology guru claimed that English had no "true dialects." I think he meant to say that English hasn't reached macrolanguage status yet.

Unicode version 5.1 was recently released, and includes some new code blocks as well as new specifications. As with all new versions of Unicode there will be a time lag until the new items can be incorporated into fonts and utilities, but here is a partial list of new items

If you're interested in the new characters, the best place to view them is at http://www.unicode.org/charts/

New Plane 0 Scripts

• Cham (Cambodia/Vietnam)
• Kayah Li (Thailand/Myanmar)
• Lepcha (India)
• Ol Chiki/Santali (India)
• Rejang (indonesia)
• Saurashtra (India)
• Sundanese (Indonesia)
• Vai (Liberia)

Script Extensions

These blocks add characters to previously encoded scripts.

• Cyrillic Extended-A
• Cyrillic Extended-B
• Arabic - characters for math, 4 Qu'ranic and multiple characters for different languages
• Indic - Malayalam, Tamil character sequences, Devanagari chandra a,
  Sanskrit sounds in Gurmukhi, Oriya, Telegu
• Latin - characters for minority languages and capital German sharp S (rare)
• Math Symbols
• Medievalist Punctuation - for research
• Myanmar Additions

New Plane 1 Ancient Scripts and Miscellaneous Symbols

• Carian (Anatolia/Turkey)
• Lycian (Anatolia/Turkey)
• Lydian (Anatolia/Turkey)
• Phaistos Disk (Crete)
• Domino Tile Symbols
• Mahjong Tile Symbols

The Insert Symbol Tool in Word

As I said last entry, I'm working on a symbolic logic course and am learning new quirks for dealing with with Unicode logic symbols...and one of them apparently is the Microsoft Word Insert Symbol tool (this is found by going to Insert » Symbol in most versions of Word.

Like the Windows Character Map and Mac Character Palette, the Insert Symbol tool lets you insert single characters into a document so you can change "P implies Q" to the logical formulation P ⊃ Q or P → Q depending on your symbolism (and you can also switch between "P and Q," P & Q or P ∧ Q).

But...unlike the Windows Character Map/Mac Character Palette, the Insert Symbol tool can take you on a little detour out of standard Unicode and into the Microsoft Private Use Area block - or the block where vendors can define their own characters. For instance, when I tried to insert the character ∩ (union) into a document, I noticed that the Insert Symbol palette gave a code point of U+F0C7 instead of the expected U+2229, and yes the U+F0 code is a sign that you are in the Private Use Area.

First I should say that there is a rationale for this. You'll notice that the font in the graphic is set to "Symbol" which is an older pre-Unicode font which was used to insert lots of special mathematical symbols. The Private Use set-up undoubtedly prevents a lot older documents from breaking.

So What?

If all you're doing is using with Word, the Insert Symbol tool may still be working for you. But these days, more and more documents are actually destined for the Web or some other format...and not all tools recognize the Microsoft Private Use codes.

The way I first noticed that the logic symbols weren't standard Unicode was that some logic symbols did not "convert" well to HTML in Course Genie but mysteriously became things like "(". The ones I had inserted properly converted, but not the ones inserted with the Word Symbol tool. Ugh.

The use of proper Unicode versus an older format does have a real world impact.

Summary

To avoid the Private Use function in new Word documents just always use the WWindows Character Map and Mac Character Palette. On Windows, you may need to switch the font to Arial Unicode.

Or if you're especially insane, you can develop your own logic symbol keyboard utility.

Since my day job is online course developer, I get to work with a lot of academic tools, including my newest tool the Course Genie (or Wimba Create) Word plugin.

This is a tool which takes a Word file "injected" with the right styles and converts a long Word manuscript into a set of well-formed HTML documents complete with table of contents page and page navigation. Even if you don't insert any self-test quizzes, this is a major time saver. But...can it do Unicode?

For once, this is a real issue since the course I'm working on is symbolic logic and uses plenty of specialized symbols like ∪,∩,∃x,∀x and so forth. So far I've been pleasantly surprised to discover that the CourseGenie planners did think ahead and implemented decent Unicode strategies.

The good news is that if your instructor (aka subject matter expert) hands you a Word file including these symbols, you may not have to do much other than make sure that the symbols are inserted from the Character Map and not from an old custom font. Course Genie by default will either convert these to numeric codes...or if you select a special UTF 8 theme, even include the UTF-8 meta tag.

For most modern browsers this is sufficient. The only gotcha is that it sets everything to Verdana text (even the symbols) and IE 5/6 acts a little strange when fonts for special characters are pre-specified for Arial Unicode.

The other complaint is that that most theme settings insert the ISO-8859-1 Latin-1 encoding meta tag instead of UTF-8...EVEN THOUGH the base XML file is UTF-8. Unless you know to select a UTF-8 theme, you won't get meta tag. Not only does this make me nervous on principle, but it means that you have to be extra careful if you ever edit the files in another program like Dreamweaver.

How does Facebook handle accents? Pretty well actually - but you can't use the numeric code. Instead you have to directly insert the character either by typing it in an Igbo Keyboard or via the Windows Character Map or Mac Character Palette.

For Web 1.0, the safest way to display accented letters was with numeric entity codes. For instance, if wanted to display Ụwa, I might write Ụwa within the HTML document. The codes were safer because they would work even if a developer forgot to include the UTF-8 meta tag.

In a Web based form, the rules may differ depending on how the developer configured the service. In some forms, you MUST enter the numeric code (often because the UTF-8 tag is missing). In other cases you CANNOT use the numeric code - this is true when you are entering data into a text field which will not go through any HTML formatting schemes. As long as the output has the UTF-8 meta tag (and Facebook does), you can avoid a numeric code (i.e. enter a "raw" accented letter) and still be OK.

How can you tell? Unfortunately, you have to test each application one by one. As I've commented before, applications which truly expect to support a global audience are generally UTF-8 ready and you can skip the numeric code. This includes Facebook, MovableType, iTunes, GoogleMaps, Twitter and so forth.

Being able to skip the numeric code is a positive sign (why memorize numbers when you can type?), but as with all change, there will be some old habits to break.

The standard super Unicode font from Apple is Lucida Grande, but now a version will be available to Windows users if they download the Windows version of Safari 3.
See http://www.alanwood.net/unicode/fonts.html#lucidagrande

Like the release of Microsoft's release Arial Unicode MS for Mac OS 10.5 (Leopard) I think this is really good for the  Web community. A lot of Mac-oriented designers have gotten hooked on Lucida Grande, but they don't always realize that it's not available on Windows (or they think that Arial is a good substitute, when Arial Unicode may be more appropriate.

With both Arial Unicode and Lucida Grande available for both platforms, the headaches of developing cross-OS friendly sites should be reduced in the future. We may be able to (gasp) pick a Unicode font we like and assume it will be on almost all machines. Wow!!!

FYI - If your site needs to display a lot of quirky characters (like this one does), I would still recommend your CSS file allow for both Arial Unicode MS and Lucida Grande...at least for the next few years.