Just for kicks, I decided to run Twitter through some Unicode tests, and I give it an A. For the record, I pretty much knew from Twittervision it supported a lot of encodings, but I threw a few more exotic tests...just to see.
The first was my standard phonetic character test...from a Mac. As far, as I'm concerned you have to pass this to be a serious global Unicode contender in my book. I also through in a long vowel (ā) and the one Hebrew word I can type (שבלת) or "shibboleth" to confirm right to left support.
What impressed me though, oddly was the the support for entity codes like é (é) and &&x0909; (उ);. Twitter can accept either raw é or it can take é and convert it to é. This differs from other modern tools like Facebook or XML which can only accept raw Unicode input (entity codes break).
Accepting either format is probably a pain to program, but very nice for the user. Having to remember when to enter entity codes and when to enter raw Unicode is confusing, but still an all-too-common reality. I appreciate Twitter for making the transition a little easier... even it's only for 140 characters.
If you are working with Hebrew, a helpful list may be the Hebrew Computing User Group on Yahoo. You have to join the list to see the messages, but they do cover a wide range of topics.
For other resources, you can check the Penn State Hebrew Computing Information Page (which, by pure coincedence, I edit).
One of the more interesting "color" pieces on the U.S. Olympics coverage on NBC was a piece on how the icons for the different sports were inspired by early Chinese pictograms, which were the precursors of the modern Chinese characters.
You can read a bit more about the design process in this article from the People's Daily Online.
The use of ancient art for modern Olympic pictograms is not new (see the entries from Athens, Syndney, Lillehammer and Salt Lake City) but I think this was the first time it made it to television.
Despite some of my previous entries, it's a fact that I really know very little about Chinese writing (I think I can recognize the characters 1,2,3). But if I really had to figure out what was going on the first place I would probably go to is Yale Chinese Mac which started back in the Mac Classic days.
Ironically though, the site is no longer just Chinese on a Mac, but includes information on Chinese on Windows, Chinese on Palm Pilot, encodings, free fonts and more. Many mysteries can be resolved here. If only I could find one of these for every script!
One challenge for math is laying out the actual equations like this integral below.
The tool of choice of for many in the math/science industry is the equation editor which allows you to insert text and symbols into different "layouts" (e.g. an integral, fraction, matrix, etc). See the image at the bottom. It's a lot quicker than Illustrator. And an equation editor can usually export the output in different graphics formats and some can export LaTeX and MathML (Ooooh!) I chose Math Magic primarily because it works on a Mac as well as Windows, but it's similar to other tools I have seen including the one bundled with Microsoft Office.
The one quirk that I previously developed methods to insert Unicode symbols via the Character Pallette or custom math symbol keyboard. Another time you might need to use non-Math Magic character insertion if you are using an especially exotic character (this happened to me once).
However, when I tried the Character Palette on MathMagic, the result was the square box of death meaning the character did not "exist." Fortunately...I realized that it was a font issue. As soon as I switched to a dedicated Math Unicode font like Unicode Symbols, all was well. But now I wonder about the default font.
The quirky fonts are not a problem if you're exporting an image or working with text, but if it's MathML it could be problematic (but maybe I'm being paranoid). In any case, I sense a future MathML test coming.
Typical Equation Editor Interface
Postscript: The MathML Test
The good news was that I was able to export a Math ML file and get the result to work in another HTML page. I should note that the <?xml...?> does not specify UTF-8 encoding. In theory, this shouldn't be a problem, but I might add the "encoding=UTF-8" part to make sure nothing weird is happening. The file also includes a custom <annotation encoding="MathMagic"> tag which is filled with vendor-generated code. I'm not sure what this does, but I will probably leave it in...just in case
A relatively "hot" new addition to Unicode 5.1 is LATIN CAPITAL LETTER DOUBLE S (aka Sharp S or ß) for German. I'd thought I'd write about this because it covers both policy and an important Unicode concept of casing.
About Sharp S (ß)
Many of you may already know about lowercase Sharp S (ß) which is used in German spelling as a replacement for "ss". For instance, the German word gross 'large' could also be spelled as groß and Strasse 'street' can be spelled as Straße. The form itself is an old manuscript convention that was incorporated into modern typograhpy.
So far so good, but what it means from a computing perspective is that any program working with German text has to know that gross and groß are essentially the same word, just with sligthly different spellings. If you're looking in a library database for instance, you would want to see both sets of results. On an interesting side note, I entered in groß and pulled up the English Wikipedia page on the "gross" unit of measure as the first result - correct, but weird..
But not capital ß
In official German spelling convention, there is NO CAPITAL SHARP S. First, no German word starts with "SS", so no word could ever begin with ß anyway. But even if a word is in all-caps or small caps, the convention should be to convert all ß to SS - thus groß should be GROSS in all caps.
Makes sense...except that people in German DO use capital Sharp S in some signs, gravestones and business names (similar to "Nite-Quil" instead of "Night-Quill"). The 2004 Proposal on Encoding Capital S Sharp (PDF) contains a variety of photographs of Capital S Sharp in use. You can see one of these on Wikipedia (Capital ß page). In other words, Unicode ultimately has to bow to social usage.
So finally we have the official Unicode announcement...
Official Unicode 5.1 Announcement
U+1E9E LATIN CAPITAL LETTER SHARP SIn particular, capital sharp s is intended for typographical representations of signage and uppercase titles, and other environments where users require the sharp s to be preserved in uppercase. Overall, such usage is rare. In contrast, standard German orthography uses the string "SS" as uppercase mapping for small sharp s. Thus, with the default Unicode casing operations, capital sharp s will lowercase to small sharp s, but not the reverse: small sharp s uppercases to "SS". In those instances where the reverse casing operation is needed, a tailored operation would be required.
http://unicode.org/versions/Unicode5.1.0/
There's a very nice write up of the issue at http://std.dkuug.dk/jtc1/sc2/wg2/docs/n2888.pdf (PDF)
Now What?
First the fonts will have to be developed to include a capital ß variant. This may or be in your system yet. Here's a quick test. It wasn't looking good, even though I am on on Leopard Mac.
| Character Name | Unicode Number | Character |
|---|---|---|
| LATIN SMALL LETTER SHARP S | U+00DF | ß |
| LATIN CAPITAL LETTER SHARP S | U+1E9E | ẞ |
Next comes the "casing" question. Casing is the set of eqiuvalences which match capital and lowercase letters as "the same" even though they are really two Unicode code points. For instance capital A is U+0041 (ASCII 65) encoded as while lowercase A is U+0061 (ASCII 97). When you search Google and most databases, both A and a are treated the same (yet are kept distinct enough so that you can switch between A and a in your word processor). Note that English casing also conflates Á,Å,À,Ä as just A.
Update from 8 Aug: Technically this probably isn't "casing", but the principle is the same - you conflate certain variants as "one" character.
As stated before, official German spelling does not recognize capital ß, but not surprisingly, there was a discussion in the Unicode list just this week on whether this too will change over time. I'll be staying tuned.
A Linguistic Closing Thought
Normally linguists talk about seeing a sound change or a grammar change in progress, but this appears to be a spelling change in progress. Wikipedia Capital ß page claims that legal documents often use capital sharp S in all cap names in order to avoid ambiguiity (e.g the defendant Hans Straßer or HANS STRAßER). And apparently the most notorious use of capital ß is the title page of Der Große Duden (The Great Duden dictionary) which was rendered as DER GROßE DUDEN. Clearly the capital sharp S was destined for permanent encoding.
AATSEEL (American Association of Teachers of Slavic and East European Languages) has just posted a set of links to "Medieval Slavic Fonts" for Old Church Slavonic, Glagolitic and Blackletter.
See http://www.aatseel.org/medieval_slavic_font for more information
List includes Unicode fonts and older non-Unicode fonts
As I've been reporting in recent entries, I've been working with a symbolic logic course which has been using various exotic symbols including double struck P (ℙ). Since every Unicode point seems to have its own story, I thought I would report some of the ineresting challenges for this character.
Finding It
When you are discussing a topic with lots of different symbols, you soon realize that in terms of Unicode, they will come from multiple blocks. For instance double struck P is from the Letter Like Symbols block (starts at U+2100), while other math symbols may be in Arrows block, the Number Forms block, the Mathematical Operators Block or possibly the Dingbats Block. You can see from the Unicode Org Symbols and Punctuation Chart just how many blocks are involved.
Although a user doesn't normally have to know the Unicode point value, because many insertion tools such as the Windows Character Map, Mac Character Pallete or others are organized primarily by block, you do have to sort of have an idea of how blocks work.
Rarity
Fonts with a robust set of math symbols are still pretty rare, and sometimes the letter like symbols are even rarer. At one point I had ℙ (P) pulling from one font and ℚ (Q) from another...interesting. Below are some fonts I know have doublestruck letters like ℙ,ℚ.
- Windows/Mac Leopard - Arial Unicode MS
- Macintosh OS X - Apple Symbol, Hiragino Mincho Pro W3 (Japanese), Hiragino Mincho Pro W6 (Japanese ), Lucida Sans
- Unicode Symbols
- Hindsight Unicode
- Chrysanthi (Chryʃsanþi)
Formatting Issues
Normally I try to avoid font and size specifications, but double struck P is an interesting counterexample. One challenge is that because the legs are hollowed out, it has a much lighter visual appearance than say normal P. My base text is 12 px on the Web, but for the double struck P, I decided to bump up the size to about 16 px (in a standards-compliant way of 1.3 em).
The other issue was selecting font faces. I wanted one with thick double legs - if you look at the font chart below from my Mac, you'll see that some fonts had some very skinny legs.
I also prefer the serif fonts in this case since I personally believe serifs help inexperieced users in reading unfamiliar scripts (in this case undergraduate college students). For this course, I'll probably point students to some freeware fonts I like
Math symbols can stretch the boundaries of Unicode display technology, but not as much as some other related blocks like Letterlike Symbols the home of such symbols as ℙ (double struck P, see image below), ℚ (double struck Q), and even the pharmacy prescription symbol (℞).
Double struck letters in particular are used in different branches of mathematics to respresent, for instance, the set of all real numbers (double struck R) or in symbolic logic to symbolize any atomic proposition. See the table below for different double struck letters and their Unicode values. See the Penn State Math Symbol chart for other common letter like symbols of math.
| Character Name | Character | Entity | Num Entity |
Hex Entity |
|---|---|---|---|---|
| DOUBLE-STRUCK REAL NUMBER (Double R) | ℝ | -- | ℝ | ℝ |
| COMPLEX NUMBERS (Double C) | ℂ | -- | ℂ | ℂ |
| NATURAL NUMBERS (Double N) | ℕ | -- | ℕ | ℕ |
| PRIME NUMBERS (Double P) | ℙ | -- | ℙ | ℙ |
| RATIONAL NUMBERS (Double Q) | ℚ | -- | ℚ | ℚ |
| INTEGERS (Double Z) | ℤ | -- | ℤ | ℤ |
I was able to upgrade to Leopard recently on my Mac which means I'm able to manipulate a working version of Arial Unicode MS for the Mac...yeah.
Web Display
My blog actually switched to Arial Unicode because of the way I had coded the CSS. It was very legible, but the x-height seemed smaller in comparison to the Apple Lucida Grande - so I reordered the priority. I will have to see if I can download Lucida Grande onto Windows via the Windows Safari download.
Back to the Logic Symbols in Word
Most of my recent Unicode adventures have been about inserting logic symbols like (∨,∧,⊃) into Word (and later Excel). My main struggle has been that if I insert them from the Character Palette, the font switches to Symbol... which is OK until I start typing English. At that point I will stop outputting the English alphabet and σταρτ ουτπυτιν τηε γρεεκ αλπηαβετ. Greek is great...unless you're typing English text. I was using the left arrow key quite a bit.
Now that Microsoft has developed a working version of Arial Unicode MS, I can input the symbols without switching over to Greek. The only gotcha is that I have to shif old logic symbols out of their pre Arial Unicode fonts (thank goodness for keyboard shortcuts). What I'm hoping is that I can bypass the big font switch in Windows word too.
So I'm happy to say that we're adding another small step towards Unicode compatibility. Finally I can have logic symbols in a non-Greek, non-Japanese, non-Chinese font!
Recent Comments