The first was my standard phonetic character test...from a Mac. As far, as I'm concerned you have to pass this to be a serious global Unicode contender in my book. I also through in a long vowel (ā) and the one Hebrew word I can type (שבלת) or "shibboleth" to confirm right to left support.

What impressed me though, oddly was the the support for entity codes like é (é) and &&x0909; (उ);. Twitter can accept either raw é or it can take é and convert it to é. This differs from other modern tools like Facebook or XML which can only accept raw Unicode input (entity codes break).

Accepting either format is probably a pain to program, but very nice for the user. Having to remember when to enter entity codes and when to enter raw Unicode is confusing, but still an all-too-common reality. I appreciate Twitter for making the transition a little easier... even it's only for 140 characters.

For other resources, you can check the Penn State Hebrew Computing Information Page (which, by pure coincedence, I edit).

The tool of choice of for many in the math/science industry is the equation editor which allows you to insert text and symbols into different "layouts" (e.g. an integral, fraction, matrix, etc). See the image at the bottom. It's a lot quicker than Illustrator. And an equation editor can usually export the output in different graphics formats and some can export LaTeX and MathML (Ooooh!) I chose Math Magic primarily because it works on a Mac as well as Windows, but it's similar to other tools I have seen including the one bundled with Microsoft Office.

The one quirk that I previously developed methods to insert Unicode symbols via the Character Pallette or custom math symbol keyboard. Another time you might need to use non-Math Magic character insertion if you are using an especially exotic character (this happened to me once).

However, when I tried the Character Palette on MathMagic, the result was the square box of death meaning the character did not "exist." Fortunately...I realized that it was a font issue. As soon as I switched to a dedicated Math Unicode font like Unicode Symbols, all was well. But now I wonder about the default font.

The quirky fonts are not a problem if you're exporting an image or working with text, but if it's MathML it could be problematic (but maybe I'm being paranoid). In any case, I sense a future MathML test coming.

Typical Equation Editor Interface

Postscript: The MathML Test

The good news was that I was able to export a Math ML file and get the result to work in another HTML page. I should note that the <?xml...?> does not specify UTF-8 encoding. In theory, this shouldn't be a problem, but I might add the "encoding=UTF-8" part to make sure nothing weird is happening. The file also includes a custom <annotation encoding="MathMagic"> tag which is filled with vendor-generated code. I'm not sure what this does, but I will probably leave it in...just in case

About Sharp S (ß)

Many of you may already know about lowercase Sharp S (ß) which is used in German spelling as a replacement for "ss". For instance, the German word gross 'large' could also be spelled as groß and Strasse 'street' can be spelled as Straße. The form itself is an old manuscript convention that was incorporated into modern typograhpy.

So far so good, but what it means from a computing perspective is that any program working with German text has to know that gross and groß are essentially the same word, just with sligthly different spellings. If you're looking in a library database for instance, you would want to see both sets of results. On an interesting side note, I entered in groß and pulled up the English Wikipedia page on the "gross" unit of measure as the first result - correct, but weird..

But not capital ß

In official German spelling convention, there is NO CAPITAL SHARP S. First, no German word starts with "SS", so no word could ever begin with ß anyway. But even if a word is in all-caps or small caps, the convention should be to convert all ß to SS - thus groß should be GROSS in all caps.

Makes sense...except that people in German DO use capital Sharp S in some signs, gravestones and business names (similar to "Nite-Quil" instead of "Night-Quill"). The 2004 Proposal on Encoding Capital S Sharp (PDF) contains a variety of photographs of Capital S Sharp in use. You can see one of these on Wikipedia (Capital ß page). In other words, Unicode ultimately has to bow to social usage.

So finally we have the official Unicode announcement...

Official Unicode 5.1 Announcement

In particular, capital sharp s is intended for typographical representations of signage and uppercase titles, and other environments where users require the sharp s to be preserved in uppercase. Overall, such usage is rare. In contrast, standard German orthography uses the string "SS" as uppercase mapping for small sharp s. Thus, with the default Unicode casing operations, capital sharp s will lowercase to small sharp s, but not the reverse: small sharp s uppercases to "SS". In those instances where the reverse casing operation is needed, a tailored operation would be required.

http://unicode.org/versions/Unicode5.1.0/

There's a very nice write up of the issue at http://std.dkuug.dk/jtc1/sc2/wg2/docs/n2888.pdf (PDF)

Now What?

First the fonts will have to be developed to include a capital ß variant. This may or be in your system yet. Here's a quick test. It wasn't looking good, even though I am on on Leopard Mac.

Next comes the "casing" question. Casing is the set of eqiuvalences which match capital and lowercase letters as "the same" even though they are really two Unicode code points. For instance capital A is U+0041 (ASCII 65) encoded as while lowercase A is U+0061 (ASCII 97). When you search Google and most databases, both A and a are treated the same (yet are kept distinct enough so that you can switch between A and a in your word processor). Note that English casing also conflates Á,Å,À,Ä as just A.

Update from 8 Aug: Technically this probably isn't "casing", but the principle is the same - you conflate certain variants as "one" character.

As stated before, official German spelling does not recognize capital ß, but not surprisingly, there was a discussion in the Unicode list just this week on whether this too will change over time. I'll be staying tuned.

A Linguistic Closing Thought

Normally linguists talk about seeing a sound change or a grammar change in progress, but this appears to be a spelling change in progress. Wikipedia Capital ß page claims that legal documents often use capital sharp S in all cap names in order to avoid ambiguiity (e.g the defendant Hans Straßer or HANS STRAßER). And apparently the most notorious use of capital ß is the title page of Der Große Duden (The Great Duden dictionary) which was rendered as DER GROßE DUDEN. Clearly the capital sharp S was destined for permanent encoding.

Finding It

When you are discussing a topic with lots of different symbols, you soon realize that in terms of Unicode, they will come from multiple blocks. For instance double struck P is from the Letter Like Symbols block (starts at U+2100), while other math symbols may be in Arrows block, the Number Forms block, the Mathematical Operators Block or possibly the Dingbats Block. You can see from the Unicode Org Symbols and Punctuation Chart just how many blocks are involved.

Although a user doesn't normally have to know the Unicode point value, because many insertion tools such as the Windows Character Map, Mac Character Pallete or others are organized primarily by block, you do have to sort of have an idea of how blocks work.

Rarity

Fonts with a robust set of math symbols are still pretty rare, and sometimes the letter like symbols are even rarer. At one point I had ℙ (P) pulling from one font and ℚ (Q) from another...interesting. Below are some fonts I know have doublestruck letters like ℙ,ℚ.

• Windows/Mac Leopard - Arial Unicode MS
• Macintosh OS X - Apple Symbol, Hiragino Mincho Pro W3 (Japanese), Hiragino Mincho Pro W6 (Japanese ), Lucida Sans
• Unicode Symbols
• Hindsight Unicode
• Chrysanthi (Chryʃsanþi)

Formatting Issues

Normally I try to avoid font and size specifications, but double struck P is an interesting counterexample. One challenge is that because the legs are hollowed out, it has a much lighter visual appearance than say normal P. My base text is 12 px on the Web, but for the double struck P, I decided to bump up the size to about 16 px (in a standards-compliant way of 1.3 em).

The other issue was selecting font faces. I wanted one with thick double legs - if you look at the font chart below from my Mac, you'll see that some fonts had some very skinny legs.

I also prefer the serif fonts in this case since I personally believe serifs help inexperieced users in reading unfamiliar scripts (in this case undergraduate college students). For this course, I'll probably point students to some freeware fonts I like

Double struck letters in particular are used in different branches of mathematics to respresent, for instance, the set of all real numbers (double struck R) or in symbolic logic to symbolize any atomic proposition. See the table below for different double struck letters and their Unicode values. See the Penn State Math Symbol chart for other common letter like symbols of math.

Character Name	Character	Entity	Num Entity	Hex Entity
DOUBLE-STRUCK REAL NUMBER (Double R)	ℝ	--	ℝ	ℝ
COMPLEX NUMBERS (Double C)	ℂ	--	ℂ	ℂ
NATURAL NUMBERS (Double N)	ℕ	--	ℕ	ℕ
PRIME NUMBERS (Double P)	ℙ	--	ℙ	ℙ
RATIONAL NUMBERS (Double Q)	ℚ	--	ℚ	ℚ
INTEGERS (Double Z)	ℤ	--	ℤ	ℤ

]]> tag:www.personal.psu.edu,2008:/ejp10/blogs/gotunicode//516.9511 2008-05-01T21:02:04Z 2008-05-01T21:21:45Z

The SIL group is using a new term I think should become more common - the macrolanguage. A macrolanguage is basically a set of related languages that share a common "identity" even though speakers can't normally understand each other.

Macrolanguages happen when language spreads to different regions and changes, but the cultural or political unity remains. Other macrolanguages include Arabic, Cree, Hmong, Quechua (as spoken in the Incan Empire), and Norweigian. I suspect that you could thrown in some other candidates like German and Italian - (we'd have more if the Roman Empire had made it to the 21st century.)

In any case, The ISO-639-3 language tag standard has a set of macrolanguage mappings which show how different related languages can map to each other so that either Mandarin Chinese (cmn) or Cantonese (yue) can also be called Chinese (zh or zho)

I really hope this term takes hold...because I really think it will simplify other discussions about language tags. After all, it was just this year that a language technology guru claimed that English had no "true dialects." I think he meant to say that English hasn't reached macrolanguage status yet.

This is a tool which takes a Word file "injected" with the right styles and converts a long Word manuscript into a set of well-formed HTML documents complete with table of contents page and page navigation. Even if you don't insert any self-test quizzes, this is a major time saver. But...can it do Unicode?

For once, this is a real issue since the course I'm working on is symbolic logic and uses plenty of specialized symbols like ∪,∩,∃x,∀x and so forth. So far I've been pleasantly surprised to discover that the CourseGenie planners did think ahead and implemented decent Unicode strategies.

The good news is that if your instructor (aka subject matter expert) hands you a Word file including these symbols, you may not have to do much other than make sure that the symbols are inserted from the Character Map and not from an old custom font. Course Genie by default will either convert these to numeric codes...or if you select a special UTF 8 theme, even include the UTF-8 meta tag.

For most modern browsers this is sufficient. The only gotcha is that it sets everything to Verdana text (even the symbols) and IE 5/6 acts a little strange when fonts for special characters are pre-specified for Arial Unicode.

The other complaint is that that most theme settings insert the ISO-8859-1 Latin-1 encoding meta tag instead of UTF-8...EVEN THOUGH the base XML file is UTF-8. Unless you know to select a UTF-8 theme, you won't get meta tag. Not only does this make me nervous on principle, but it means that you have to be extra careful if you ever edit the files in another program like Dreamweaver.