Re: Software
[way more than] everything you ever wanted to know about character encoding |
|
<ράντ type="οπτιοναλ"> Have you ever seen this, and wondered what the heck it means?
Read on! As you’ll see, this is a beautiful solution to a tough problem. In the beginning we needed to represent characters as numbers, and so there was ASCII, and it was good. (I’m skipping FIELDATA and let’s not mention EBCDIC in polite company.) ASCII was a way of representing every character in 7 bits, such that they fit comfortably in a byte, leaving a high-order parity bit. Where “every” includes all letters, numbers, and all the punctuation you could ever want, including stuff used for programming like quotes, equal signs, and angle brackets. And then C was created and the Unix runtime libraries, and they cleanly supported ASCII. Char was a native type, one byte, and strings could be stored in arrays of char, and by convention the character NULL = 00 was used to mark the end of a string. A metric ton of code was written which processed, managed, and manipulated such strings. Most of it ignores the content of strings, secure in the knowledge that every character resides comfortably in one byte, and the whole was terminated by a zero. Life was good. Gradually however it became apparent that “every” did not actually include all characters. There were these people in Europe who used áccènts and ümlaûts. They even had some different punctuation like upside down exclamation points¡ Who knew? And so the high order bit formerly used for parity was reused and 128 more characters were defined. Such a luxury, there was even room for graphics characters that could be used for drawing lines and stuff; ▀▄ yippee ▄▀. These characters still fit snugly into one byte, and that metric ton of code worked perfectly. But … turns out there are all these other people on Earth who don’t use the Roman alphabet, and they use computers too! We’re talking Greek, Cyrillic, Armenian, Hebrew, Arabic, and so on. (Armenians send email? Who knew?) So… now what? Well it was determined that for any one person or computer, they could live with an additional 128 characters, only different people needed different additional characters. So the concept of code pages was invented. Each code page was the definition of 128 characters which was used when that high-order bit was set. On any one computer the code page was fixed, but different computers could use different code pages. And characters *still* fit snugly into a byte, and 00 still meant the end of a string, and all that code still worked. Whew! But … turns out there are all these other people on Earth, who don’t use alphabets, they use symbols! We’re talking Chinese, Japanese, Korean, and so on. (Korean XML? Who knew?) So… now what? We need literally thousands of characters to store all these symbols… Oh no, Mr. Bill!
So… we have Unicode, but how do we represent these code points inside a computer? This is where character encoding comes in... there are different ways of encoding a series of Unicode code points as computer data. In this, as in just about everything else in computing, there is the crummy nonstandard technique Microsoft uses (called UTF-16), and the elegant cool technique everyone else uses (called UTF-8). BTW you will win bar bets if I tell you UTF stands for Unicode Transformation Format, please PayPal me 10% commission.
UTF-16 is the idea that all Unicode characters are stored in two byte “words”. Every code point is assigned a 16-bit value, characters are 16-bits wide, and strings are arrays of 16-bit words. The end of a string is indicated by a 16-bit zero. There are some problems with this; first, that metric ton of code written in the old days will no longer work, second, most strings are now twice as big in memory as they used to be, and third, there are more than 65,536 code points, so there are too many characters and symbols to fit into 16-bit words! Okay, so here’s what we’ll do; first, we’ll rewrite all the old code, create new subroutines for everything. No problem. Second, we don’t care about memory. Third, for code points too big to fit into one 16-bit word, we’ll use two 16-bit words. There will be a special range of values (D800-DBFF) which mean “I am the first word of a two-word (four-byte) sequence”. Of course if you look at a second word, you won’t know if it is just a 16-bit value, or the second word of a 32-bit value, but that’s a detail. Oh, and yeah there is a byte ordering problem, some computers represent 16-bit values with the high order byte first (big endian) and some with the low order byte first (little endian), so we will start every string with the value FEFF, so that everyone can tell. I am not making this up, that’s UTF-16, and that is the way all characters are stored and processed inside Windows. If you are reading this on a Windows PC, all these characters are coming to you via UTF-16. These are called “wide characters”, there are “wide” alternative versions of string manipulation function, and the whole thing is massively ugly.
UTF-8 is the idea that all Unicode characters are stored in 8-bit bytes, just like before. Some code points fit in one byte, some in two, some in three, etc.; as many bytes as are needed to represent the code point. The end of a string is indicated by a zero, just like before. The values 1-127 are standard ASCII, just like before. (In one fell swoop, perfect backward compatibility!) All the old code still works, and we don’t need new subroutines for everything. Some characters require more than one byte, but we only use the bytes we need, so no memory is wasted. When you see a byte, you can tell immediately whether it is the first byte of a multi-byte sequence. There are no endian issues. It is a beautiful solution to the problem. When you see a byte, you can what kind of byte it is by the value:
And you may ask yourself, If you do care about content, and the content is ASCII, not much has changed. You can scan for common parse characters like “<” or “=” inside a string, just like before. Finally if you care about content, and the content might not be ASCII then you have to be aware that the byte length of a string is not necessarily the same as the character length. To count bytes, you, um, just count bytes. To count characters, you count bytes which are in the range 01-7F and C0-FF, and skip bytes in the range 80-BF. Pretty simple. Copying and moving characters strings is exactly like before. Mostly stuff just works. To find a multi-byte value, you search for the multi-byte value in the string; the encoding ensures that a given sequence of characters only ever means one thing. </ράντ> :) |
programmers vs hardware |
|
Jeff Atwood writes Hardware is Cheap, Programmers are Expensive. "Given the rapid advance of Moore's Law, when does it make sense to throw hardware at a programming problem? As a general rule, I'd say almost always." Yeah, but... |
the world's dumbest installer |
|
You may not be aware, but there is intense competition between companies to create the world’s dumbest installer. I always thought Fog Creek deserved honorable mention for writing their own installer from scratch, but I have to admit it isn’t a bad installer, that was just a waste of their time. Sun’s Java upgrader is definitely in the running, for elevating what should be a silent little background thing into the foreground like some kind of major application that users care about. Real’s installer is a nightmare, with 400 options in seriously deeply nested menus, most of which have the wrong default settings (“Do you want to use Real for all C++ compiles? – default Yes”). But I have to give the award to Microsoft for their IE update to v7; in this, as in so many other things, I subtract all benefit of a doubt since they are so big and mighty and ought to know better. |
when new versions suck |
|
Don't you just hate it when new versions suck? You know what I'm talking about, I know you do... you have this software, and you use it every day, or weekly, or once in a while, and you like it, and you know it, and, well, it works for you. Maybe it has some bugs, maybe you wish it would be faster or better in some way, but it works for you. And then you find there is a new version. And you want to "stay current", and you're hoping if you upgrade maybe it will be faster or better in some way, and so you install the upgrade. At this point, one of two things happens... The good outcome is that the new version is like the previous version that you knew and liked, and maybe it fixes some bugs, and maybe it is faster or better in some way, and still it works for you. The bad, horrible, sucky outcome is that the new version is different, you no longer like it or know it, and now it doesn't work for you anymore. |
deHeisenbugging |
|
Word of the day: Heisenbug.
Their possible presence leads to the Heisenbug uncertainty principle: it is never possible to be sure there are no more bugs :) I am fighting a tenacious Heisenbug just now and wishing I could pause the universe temporarily to examine the server when an error occurs on the client. I wonder if the developers of The Matrix began this way? P.S. Yes of course there is a Wikipedia entry for Heisenbugs, as well as Bohrbugs, Mandelbugs, and Schroedinbugs. And my favorite, the Stotle, “the incorrect output of a computer program that contains no bug”. |
predicting bugs |
|
Today I spent considerable time on a problem that confronts all software developers: how can you predict bugs? I ended up deriving the following equation... Please click here to read how this works (!) ... |
everything you need to know about COM |
|
Everything you need to know about COM: For more than everything you need to know, please click here... |
the lost art of desk checking |
|
Okay, I’m going to date myself here. I’m 49 years old, and I started programming in Junior High, when I was 13, so my story begins 36 years ago. The dawn of time, metaphorically speaking. Please join me for a gentle rant about the lost art of desk checking... |
Oleosaurus: the dispose pattern |
|
You all know how much I love .NET. And you all pretty much figure I’m a hopeless dinosaur, and I just don’t get it. And you’re all pretty tired of hearing me rant about it. So, sorry… but yes, here’s another one. You may click "back" and get on with your life as appropriate. In which we investigate that wonder of .NET programming, the dispose pattern... |
managed memory leaks |
|
This is another in my series of foaming rants whereby you the reader become convinced of my status as a coding dinosaur. So be it. Today's subject is memory management, the old "bad" way, and the new "good" way using garbage collection in managed code. Please click to read more... |
pissed off in Peoria |
|
You all know I can't stand .NET’s virtual machine architecture, and you probably think I’m a hopeless dinosaur who just doesn’t get it. Everyone knows Microsoft is great, everyone knows .NET and Java are the future, etc. Someday Ole will retire from railing at progress. (And everyone will be spared Sunday morning rants :) Let me give you a clean example of what I can't stand about .NET’s CLR: Visual Studio 2005. |
as the memory turns |
|
You all know my status as a dinosaur; I can remember when all we had were zeros, and how great it was when we first got ones. (There are 10 kinds of people in the world, those who understand binary, and those who don’t.) Anyway here's a rant about virtual memory vs physical memory as the constraint du jour... |
the metric magic |
|
I've been spending a few days worrying about measuring stuff. Like productivity and predictabilty. Way back in late December, 2006, I worried about this, too, and wrote a long rambly email to my team about it. I just reread it, and thought it might be worth sharing. So here it is. |
dynamic page sizing |
|
This is a nerdy exposition, for those of you creating web apps and for me to be able to find it later :) The subject is dynamically sizing web page elements based on the size of the browser window. Please click to read more... |
almost famous design and stochastic debugging |
|
In which I screw up a programming change, devise an almost famous design, and engage in stochastic debugging, and philosophize... |
file permissions |
|
I have a question. Have you ever used OS file permissions? I have not. Ever. |
On the Persistence of Bad Design... |
|
As a programmer you are constantly making design decisions. Some are small, some are big. Some have little effect, some have larger effect. And every once in a while you make some decisions which seem small, but have a huge effect. If these decisions are made badly, then it affects many other people for years to come. |
Web 2.0, "Live" and other meaningless jargon |
|
Do you hate business jargon as much as I do? Blech. A classic example of meaningless jargon is "Web 2.0". Nobody knows what it means, it doesn't mean anything. And for an unbelievable example of jargon run amuck, consider Microsoft's recent "Live" announcement. Talk about meaningless blather. |
wrapping up Apple on Intel |
|
Okay, I was wrong. Completely. So be it. The "good call" award goes to Gary Lang, who emailed on Sunday:
Now, as someone working on porting Windows software to the Mac, I wonder, what does this mean? In the short term, not much. In the slightly longer term, use Xcode instead of CodeWarrior. And in the longer term, who knows? Stay tuned :) |
even more Apple on Intel |
|
I know you're probably sick of the Apple on Intel speculation and you don't care anymore or maybe you never did. Please skip the remainder of this post. Thank you. Here's the bottom line: I have to believe the next big play at Apple is online video. |
more apple on intel |
|
I have nothing brilliant to add to my musings yesterday about Apple on Intel. I still feel the key reason must have something to do with binary compatibility with Windows applications. All the other things Apple gets from this transition - efficiency, cost savings, branding - just don't see worth the trouble. Especially in games, Apple trails Windows dramatically in the availability of software titles; this factor is holding back the Mac platform more than any other. |
apple on intel |
|
Earlier today I posted about the rumors that Apple is planning to support Intel CPUs. I'd wrote "probably 'support' not 'switch to' but after one bike ride's worth of cogitation I think this is exactly wrong... Maybe it will be possible to run Windows programs "as is" without any changes on top of some kind of runtime emulation inside OS X? Now that would be a reason to do this! |
Shorthorn |
|
This is going to be a long post. I can tell. I have all these thoughts, jumbled together, and it is going to take a lot of words to get it all out. Sorry. The subject is Windows... Microsoft has now spent four years building Longhorn, the "next" version of Windows, and it looks to be spending two more years at least. When Longhorn is released, it will have been at least six years since XP came out. That is a long time in computer years. What will we get? If it were up to me, Microsoft would stick to its knitting, and instead of trying for more and more functionality - which is properly the province of application software anyway - it would fix paging and fix networking. Do the things Windows should do well. I won't get my wish, but that's what I want. |
solving bongard problems |
|
I found a great site from Harry Foundalis about his Research on the Bongard problems. What's a Bongard problem? Well, here's one: |
C++ method pointers |
|
Have you ever wanted to use a pointer to a class method? This might be basic C++ but I couldn’t remember how to do it, and spent some time Googling and messing around to figure it out. So here’s the way... |
RSS cookbook |
|
Okay, today we are going to get YOU to use RSS. Follow the simple steps, and you'll be using RSS, and loving it. I promise this is worth it. You will thank me. More... |
nph and mod_gzip |
|
If you write nph CGIs, be sure to exclude them from mod_gzip. That's the message of this post; feel free to skip the rest if this is geek to you. |
Simple is Good |
|
I always have so much to do in December, don't you? So many social events, Christmas shopping, year-end deadlines. But somehow I love it. The cold crisp air, lights everywhere, a sense of excitement, music... And since I have so much to do, naturally I'm procrastinating by working on something I don't have to do at all. Yep, I redesigned my blog. And you probably can't even tell! |
I Switched! [temporarily] |
|
So, for the past four days my [Windows] laptop was down, and I switched to using my iMac as my "main" computer for a few days. Overall the experience was pretty good. I thought you might find a brief review interesting. continue reading... |
The Little Boy Explains |
|
The other day I posted The Emperor's New Code, a heretical critique of "Longhorn" Microsoft's upcoming version of Windows. I expected to get a lot of links, and I did - thanks Dave, Tim, and Robert! - and I expected to get a lot of criticism, and I did. I am like the little boy who cried "the emperor's not wearing any clothes", and of course some noblemen cannot admit this; it would be too embarrassing. Or maybe the little boy just can't see the clothes :) So. Dialog is always healthy, right! I'd like to take a moment to discuss the most prevalent reactions... |
The Emperor's New Code |
|
... In which the author proves himself a hopeless heretic by disparaging Longhorn ... I attended the Microsoft Professional Developer's Conference in Los Angeles last week. Microsoft formally unveiled "Longhorn", the next version of Windows, along with a bunch of new underlying technology. My first day's reaction was PDC = Moo!; a positive impression of a lot of cool new stuff. But my takeaway is... there's a lot less here than it would at first appear... |
PDC = Moo! |
|
I'm at the Microsoft Professional Developer's Conference (PDC) in Los Angeles. Today Microsoft formally unveiled "Longhorn", the next version of Windows. I left for the PDC at 5AM, and got home at midnight. Cool. |
Really Moving Mount Fuj |
|
Everyone wants to know "How would you move Mount Fuji"? |
Third-Party C++ Children |
|
Suppose you have an application which provides a "core" for other developers. Suppose you wan to provide functionality as classes, rather than APIs, but enable extension. Supporting third-party children is not easy... |
The Lead Bullet - Writing Specs |
|
In which we talk about writing specs, the subject near and dear to every programmer's heart... |
Even More Bridgework |
|
Okay, here it is, the optimal solution for 25% flashlight carry... |
More Bridgework |
|
We're back at the bridge of the four programmers, as we consider variations in flashlight "carry". There is more complexity in this problem than you thought. Yay! |
Revisiting the Bridge of the Programmers |
|
Remember the bridge of the four programmers? An interesting technical interview problem, with an unexpected answer. It turns out there is more complexity in this problem than I had thought. Yay! |
The Two Switches |
|
I solved it! And it is great!! The infamous "two switches" puzzle does have a solution, and it isn't a trick; it is a pure logic puzzle. Read on for more... |
The $21 Question |
|
A few days ago I reviewed How Would You Move Mount Fuji, a great new book about the logic puzzles often used in technical interviews. I received a lot of feedback - thanks! - and herein consider Chris Lightfoot's different and better answer to the $21 question... |
Moving Mount Fuji |
|
I just read "How Would You Move Mount Fuji", a great new book about puzzles as technical interview questions. I enjoyed it a lot - it is an easy read, and kind of "fluffy"; I blew through it in two days. I'm often in the position of asking these questions, and I found it terrifically helpful. Here are my thoughts about the book, and also some examples taken from it... |
Emergent Properties |
|
In which we discuss emergent properties vs. explicit properties, take Marvin Minsky to task about artificial intelligence, diss RDF and the semantic web, and relate image processing to water. Read more... |
Google and Blogs |
|
There's been considerable discussion in the blogosphere about Google "dropping blogs" from search results. What's the truth? Here's some educated guesswork... |
Outbound Trackbacks |
|
I implemented "outbound trackbacks" today. Essentially a trackback is a way to tell someone: "hey, I linked to your site". |
Site Optimization |
|
If you're a web nerd like me, you might be interested in Site Optimization... |
Starting to Get .NET |
|
Everyone agrees that .NET is at least two things: a development environment and a marketing concept. The net net: .NET is a development environment positioned as a follow on platform to Windows... |
I Don't Get .Net |
|
Hello, my name is Ole, and I don't get .Net... |
"Tyranny" Revisited |
|
A week ago I posted a little article called The Tyranny of Email, giving some tips for improving personal productivity. It generated a terrific response, and I herewith post the most interesting observations and comments... |
How to Write C++ Classes II |
|
A little while back I posted How to Write C++ Classes. There was more which could be said, and so I did; here's How to Write C++ Classes II... |
The Tyranny of Email |
|
If you're reading this, you're online, which means you may be a victim of the tyranny of email... |
Frames, Too |
|
Today I made a couple of slight enhancements to the way this site uses frames, please click here if you're interested... |
Weighting Links |
|
Wow, so Google has bought Pyra, the company behind Blogger. This really puts 'blogging on the map. It also creates some potentially interesting conflicts of interest (will Google searches preferentially find Blogger 'blogs?). At first glance, actually, it doesn't seem a great fit. Google did buy Deja News about a year ago, but that made more sense; Deja archived all the Usenet groups, wereas Blogger only has about ¼ of the blogosphere... Perhaps the synergy is at the link level. Google indexes links, Blogger makes links. |
Simple Search |
|
Every website needs a simple search facility, including this one... |
How to Write C++ Classes |
|
If you're a programmer today, most likely you are writing in Java or C++. In which case you may be interested in How to Write C++ Classes... |
Tracking Visitors |
|
You are being watched... |
Turning Over Software |
|
The programmer's dilemma, turning over software... |
Frames |
|
Yeah, this site uses frames. I explain why - and how... |
Home
Archive
'11
'10
'09
'08
'07
'06
'05
'04
'03
flight
Re:Cycling
Re:The Book
Re:Software
Re:Philosophy
About Me
W=UH
Email
Greatest Hits
Correlation vs. Causality
The Tyranny of Email
Unnatural Selection
Lying
Aperio's Mission = Automating Pathology
On Blame
Try, or Try Not
Books and Wine
Google and Blogs
Emergent Properties
God and Beauty
Moving Mount Fuji
The Nest
Rock 'n Roll
IQ and Populations
Are You a Bright?
Adding Value
Confidence
The Joy of Craftsmanship
The Emperor's New Code
Toy Story
The Return of the King
Religion vs IQ
Most Spectacular Photos of 2003
In the Wet
the big day
solving bongard problems
visiting Titan
unintelligent design
Shorthorn
the nuclear option
second gear
On the Persistence of Bad Design...
Texas chili cookoff
the inflection point
almost famous design and stochastic debugging
may I take your order?
paper art
triple double
China's olympic gardens
New Yorker covers
Death Rider! (da da dum)
how did I get here (Mt.Whitney)?
the Law of Significance
Holiday Inn
Daniel Jacoby's photographs
room with a view
weird disaster update
in praise of paddle shifting
the first bird
Gödel Escher Bach: Birthday Cantatatata
shining a light
Father's Day (in pictures)
Tour de France 2009
Tour de France 2010
Jobsnotes of note
Tour de France 2011