Misdirected Advertising
As part of the new year clean up I logged into Facebook, one of the few social networking sites that I remain a member of. I think they’re telling me I’m getting older.
Memories of the Logo Turtle
I’ve been messing around a lot with Canvas and I keep adding the penUp method into my code. Firebug then quickly reminds me that Canvas is not equal to Logo. (I should preface my posts nowadays with a nerd alert….)
Anyone else remember Logo? When the nostalgia hit hard on Sunday I searched for a web version. It’s good to know the Logo Turtle is still alive and well.
See my pretty star that I drew (with the help of the cheat sheet at the bottom of the Logo page).
My coding mistakes: Time to stop using the escape function in JavaScript
Mistake: Just because the JavaScript escape function has worked well for my American- and European-centric tests doesn’t mean it works for Chinese characters.
Yep, after I figured out the French character problem along comes a problem with Chinese requests. Not really surprising: the code I inherited was well designed, but it was only ever really tested in the US, somewhat in Europe, and never really tested in Chinese applications.
I didn’t have any problem finding a test case. I just grabbed what I thought looked like a cool character from my test examples:
黄
I don’t speak Chinese, hell I almost didn’t make it to the Airport in Shanghai when I braved a taxi ride on my own, but Google found me a great site that had a load of Chinese characters along with their Unicode and UTF-8 character encodings at http://www.ansell-uebersetzungen.com/gbuni.html.
This one was a lot easier to figure out. The character:
黄
Should have been translated into the following hex codes:
%E9%BB%84
But nope, it was being translated into the string of the Unicode index:
%u9EC4
If I wanted a straight Unicode lookup, this would have been great, but the server expected UTF-8 encoding. The Mozilla Developer Guide says that escape shouldn’t be used. Well, escape worked just fine until we introduced Chinese characters, and since we needed to support Firefox usage, escape had to go. Looks like escape only works correctly for 1-byte (ASCII) and 2-byte UTF-8 characters, but breaks on 3-byte and larger characters.
Conclusion:
Use encodeURI or encodeURIComponent in modern browser code instead of escape. For a good comparison of these two functions, please see this article.
My coding mistakes: UTF-8 character encoding can’t be assumed
Sometimes I make mistakes that I know I shouldn’t. Rather than hide them, I will share them (when I find time to do so) in the hopes that others might learn from them.
Mistake: Assuming that all characters on a web page in a UTF-8 configurd browser are treated as UTF-8
Let’s start with the instigator of the problem, found in a French test word, the small e acute:
é
Damn French characters. My distilled process that recreated the problem:
- I was passing some French characters through the JavaScript API I am working on.
- The characters went through my JavaScript code to the backend server.
- The backend server did some crunching of my request.
- I got totally shitty results back that even me, the worst speaker on the entire planet, knew were wrong.
I got lucky and guessed the category of the problem correctly immediately: Character Encoding. But wasn’t everything supposed to be in UTF-8 these days? Damnit, how can I be sure?
I followed this path to figure out what was going on:
- First stop: Wikipedia on Character Encoding
- I know everything is supposed to be in UTF-8 in my browser, but damnit what the hell is UTF-8 really? Wikipedia to the rescue again with an entry on UTF-8
The idea of character encoding, or the storage of (usually) visual and human understandable glyphs by a (usually) computer system, is not magic.
I see the letter on my computer screen as:
é
Assuming UTF-8, the computer sees the following string of bits, from left to right:
11000011 10101001
Something was happening in the transfer between my web page to the backend server, that’s all that I could figure out. A little more investigation and I stumbled onto the second layer of the problem: what I thought was going on and what was really going on.
I see the letter on my computer screen as:
é
In UTF-8, the bits of this letter are:
11000011 10101001
But I wasn’t actually sending bits as bits, I was sending the bits as a hex string in a URL, encoded as follows:
%C3%A9
I looked closer at the results I should have been getting from the server and the results I was actually getting. The results were entirely fubared, totally and completely. I mean, the server was returning results, but it was as if I was asking a question in French and getting an answer back in Esperanto.
I could only guess that my original UTF-8 character, the small e acute, while seemingly being translated as a double-byte UTF-8 encoded character might actually be decoded on the server as two characters. It would be weird, but it would explain the most bizarre results I was getting back.
Take the:
é
That I sent to the server as:
%C3%A9
And what if the server was decoding it differently, say as part of the ISO-8859-1 extended set:
é
With fuzzy matching doing it’s magic on the server side, I guessed the resulting single byte characters might have led to the crappy results that were being returned. The server was supposed to work with UTF-8, but what if I was somehow signaling the server to treat my requests as Latin-1. Didn’t make a lot of sense, but I wasn’t in charge of the server.
Browsing around on the internet, I learned that the following element should be inserted into the head element of web pages to signal that everything should be dealt with as UTF-8:
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
After I made this setting to everyone of my web pages the requests in French returned results that made sense and the results I was expecting.
Conclusion:
On the web, don’t assume things are in UTF-8, or any encoding for that matter. Explicitly set the encoding on the web page.
To President Elect Obama on the Office of Deprogramming
Dear Mr. Obama,
Congratulations on kicking what’s his name and she-who-shall-not-be-named in the ass in November. I hear because your middle name is Hussein you must be Muslim. I bet people try to remind you to read the teleprompter from left to right, too. When you get sworn in, give me an Allah Uh Akbar just so I can watch some people I know squirm. And remember to thank the long string of white honkies who came before you. LOL!
Alright, joking aside, my mom and I were talking on the phone today about your missing department and cabinet member. That’s right, it’s the Secretary for the Department of Deprogramming. No really, I’m not kidding. Hell I’d even volunteer for this position.
Let’s review the First Amendment of the Bill of Rights:
Congress shall make no law respecting an establishment of religion, or prohibiting the free exercise thereof; or abridging the freedom of speech, or of the press; or the right of the people peaceably to assemble, and to petition the government for a redress of grievances.
and along with the famous letter written by Thomas Jefferson (linked here to online Library of Congress), I believe this Department of Deprogramming is quite important.
What needs Deprogramming? Religion from our government, of course. Would I dare suggest we deprogram religion from the individual engaged in politics? No and never! One of my favorite politicians, Ron Paul, scares my liberal friends because of his supposed personal views on abortion. But Ron seems quite clear about how to keep his personal morals out of our government, as I wish others would.
I would neither flatter myself that you will make your way to this blog to read this letter, nor would I insult you by being cynical that you don’t read simple little blogs like mine, so I’ll just toss this into the ether for fun. Maybe someday in my lifetime I’ll get to witness a real life Department of Deprogramming. Perhaps that group, what were they called, oh yeah the Supreme Court, might step up and fill the role and we can forget this whole idea.
Yours truly,
- Jeremy
Software Released
I did it, I finished my first software release on time! Somethings I did better than expected, somethings I didn’t do as well as I wanted. I did squash every bug that I had scheduled myself to squash, and as far as I know I didn’t create anymore.
Computer programming is a rush. That must make me a nerd.
10 Bugs Left to Fix
Only 10 bugs left on my queue of work for the year for stuff that “must be done.”
Not bad, considering the number I started with.
What I do while Netbeans loads
- Load up Pidgin and Thunderbird
- Open Firefox and peak at SLV, GLD, and the S&P 500
- Pull my personal email, delete the crap that still makes it through my spam filter
- Open Outlook and imagine Steve Ballmer and Jonathan Schwartz duking it out for the remaining resources on my system
- Cycle through the failblog posts
At this point, Netbeans has usually loaded up, but in that not quite ready for use state, so I shove it in the background one more time and do something like
- start a blog entry like this that I will post later
No more indistinguishable icons in the comments section
I really liked my previous theme, all except for the design of the comments section. Since I’m not going to alter the theme myself, I went looking for a new one.
And here it is. Thanks Brian Gardner for making this available.
Bank Fraud
Times have changed and so have problems.
In the old days I kept a chain on my wallet and simply avoided the bullies who wanted to make me buy them lunch. Now my wife and I keep extremely good records of our bank account, and like regular computer backups, the practice has paid off.
A week and a half ago there was an almost unnoticeable charge to a DEN ENTERPRISES for $9.60. Last Thursday, a week after this strange charge, there was a charge to FUNDABLE for $70. These two charges happened as point of service transactions with my wife’s debit card. Who knows what they would have been this coming week if we didn’t catch the charges earlier.
We’ve taken all the correct actions, the money is coming back to us, and it only took a little over half a day of my time to put some smackdown on the unknown internet annoyance.
I write this as a data point for others who might have dealt with fraud, and for public archival of what has happened so far. Hopefully there will be no follow up to this post.
