{"id":4073,"date":"2016-03-19T11:33:39","date_gmt":"2016-03-19T11:33:39","guid":{"rendered":"https:\/\/really.zonky.org\/?p=4073"},"modified":"2016-03-19T11:34:44","modified_gmt":"2016-03-19T11:34:44","slug":"the-weird-characters-of-unicode","status":"publish","type":"post","link":"https:\/\/really.zonky.org\/?p=4073","title":{"rendered":"The Weird Characters Of Unicode"},"content":{"rendered":"

As you may or may not know, Unicode<\/a> is the standard for encoding text into ‘computer speak’. There have been many different encodings of characters (graphemes<\/a>, and other symbols) into “computer speak” by different manufacturers; all of which were severely limited. Almost all of them included the normal Roman alphabet plus a variety of other symbols.\u00a0Amongst other things, this had two main problems.<\/p>\n

Firstly, it excludes large parts of the world that use other writing systems from using computers sensibly – computers are difficult enough to use (at least at first) without learning a whole new writing system. And even then storing a document by transliterating it is not ideal – you have changed the original and introduced another possible source of errors.<\/p>\n

Secondly there is the problem of errors introduced into documents when moving them from one computer system to another – for example it was not unknown for a “#” to become a “\u00a3” (or very similarly,\u00a0“\u00a3” will appear as “\u00c2\u00a3”). Very much more extreme examples exist<\/a>.<\/p>\n

Of course because of the enormous number of symbols in Unicode, there is a great deal of fun to be had with Unicode – not infrequently poking fun<\/a> at Unicode for including ridiculous symbols. And why not? There’s no harm in having a bit of fun :-<\/p>\n

\u0265\u0287\u0131p\u01dd\u0279\u01dd\u026f \u01dd\u029e\u0131\u026f<\/a><\/p>\n

However it is worth pointing out that Unicode standards are a serious business and slipping in “fun” symbols is not likely to happen. Although I was not directly involved, I did help out one of the people who pushed for the inclusion of medieval Slavonic characters within the Unicode standard, and it is not a trivial process.<\/p>\n

And now for some “fun” Unicode characters…<\/p>\n

\u00fe<\/h1>\n

The Thorn<\/a>. The English letter that got away. Before the age of printing, English had an additional letter (I’m over-simplifying here) which was used instead of the digraph th<\/em>, so words such as the<\/em> would have been spelt \u00fee.\u00a0<\/em>By the time that printing had arrived, the shape of the thorn letter was becoming more like a “y” and because the printers imported their equipment from countries that did not have a thorn, the printed books tended to use “y” instead of \u00fe. Which is of course where we get “Ye olde Shoppe” from.<\/p>\n

Of course it was confusing printing\u00a0ye<\/em> when we said\u00a0\u00fee<\/em> (or\u00a0the<\/em>), so the printers settled on\u00a0the<\/em>.<\/p>\n

So why is \u00fe in Unicode? Because you cannot discuss the letter without including it, and perhaps more importantly cannot encode a historical document that used \u00fe without an encoding for it.<\/p>\n

\u2603 and probably \u2615<\/h1>\n

Ah! The snowman (and the cup of coffee). What sense of fun allowed these symbols into the standard?<\/p>\n

Well according to the Unicode standard, it is contained within a block of weather symbols so it was almost certainly contained within a TV station’s encoding standard for weather forecasts. And you cannot claim to be a universal standard for text encoding without including the symbols included in other encodings.<\/p>\n

\u203d<\/h1>\n

The interrobang<\/a>. The punctuation symbol used (if rarely) for signalling both a question and an exclamation: What the bleep are you doing\u203d<\/p>\n

Whilst not commonly used today, it was very commonly used in the 1960s and so there are many documents that need encoding that use this symbol.<\/p>\n

\u2620, \u26a0, \u2622<\/h1>\n

These look like fun don’t they? They certainly do to me, but in fact they are international symbols for various dangers – poison (\u2620), warning (\u26a0) of a general nature, and radioactivity (\u2622). All pretty serious stuff; and you\u00a0really<\/em> don’t want those symbols garbled in a document.<\/p>\n

\u0e5b<\/h1>\n

This is a Thai “letter” and I picked it out because it’s made fun of elsewhere, but it stands for all the non-European symbols used in language.<\/p>\n

It may look kind of funny, but it probably isn’t so much to someone who knows Thai. To put it another way, if you told me that we’re not going to include the “M” in a character encoding because it looks too silly, I’d be very, very annoyed (my name contains two of ’em).<\/p>\n

And yes I can type all of the above and the following into a text terminal \ud83d\ude03<\/p>\n

 <\/p>\n

\"2016-03-19_1119\"<\/p>\n

 <\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

As you may or may not know, Unicode is the standard for encoding text into ‘computer speak’. There have been many different encodings of characters (graphemes, and other symbols) into “computer speak” by different manufacturers; all of which were severely limited. Almost all of them included the normal Roman alphabet plus a variety of other […]<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_share_on_mastodon":"0"},"categories":[4],"tags":[1393,1395,1396,1394,1388,862,1390],"class_list":["post-4073","post","type-post","status-publish","format-standard","hentry","category-it","tag-1393","tag-1395","tag-1396","tag-interrobang","tag-thorn","tag-unicode","tag-th","category-4-id","post-seq-1","post-parity-odd","meta-position-corners","fix"],"share_on_mastodon":{"url":"","error":""},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p1f2KI-13H","_links":{"self":[{"href":"https:\/\/really.zonky.org\/index.php?rest_route=\/wp\/v2\/posts\/4073","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/really.zonky.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/really.zonky.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/really.zonky.org\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/really.zonky.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4073"}],"version-history":[{"count":4,"href":"https:\/\/really.zonky.org\/index.php?rest_route=\/wp\/v2\/posts\/4073\/revisions"}],"predecessor-version":[{"id":4078,"href":"https:\/\/really.zonky.org\/index.php?rest_route=\/wp\/v2\/posts\/4073\/revisions\/4078"}],"wp:attachment":[{"href":"https:\/\/really.zonky.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4073"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/really.zonky.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4073"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/really.zonky.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4073"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}