Next Day Video
2 HN recommenders · recommended in 4 months, 2015–2019
36:21 · 80K views · Mar 16, 2012
Ned Batchelder's PyCon 2012 talk gives a quick overview of what Unicode is, then focuses on what a Python programmer needs to handle text correctly. It presents the 'Unicode sandwich' strategy: decode bytes at input, work with Unicode internally, encode at output.
Commenters point to it as the clearest explanation of Python's bytes versus Unicode model and the Unicode sandwich, which finally makes the confusion click.
4 comments from Hacker News
You have to invest some time in understanding how strings (byte strings and unicode) work in Python. I put this off for way too long. After A TON OF PAIN I decided to put the time in to watch the talk "Pragmatic Unicode, or, How do I stop the pain?"[0] by Ned Batchelder and it all just clicked. Now, I make unicode sandwiches like a boss. [0] this video
From a quick skimming through the docs[0], Perl 5.8's Unicode support sounds a lot like Python 2's except that the default encoding is latin1 instead of ASCII - i.e. unlike the Python 3 way of explicitly decoding binary data into Unicode text at the point where it's read (where presumably the encoding of the data is known), it will defer the decoding until the data is used (at some completely unrelated point in the program) and decode it with some assumed globally agreed upon encoding. Since there is no globally agreed encoding (Windows even has different legacy encodings between Win32 and the command prompt!) this will appear to work as long as the data is ASCII but later, when you least expect it (and someone inputs a non-ASCII string), give UnicodeDecodeError in Python 2 or garbage data in Perl 5.8. Ned Batchelder gave an excellent talk[1] that explains how the Python 3 approach to Unicode works. I think it makes a lot more sense once you understand it; the Python 2 way was clearly broken, and it looks like Perl 5 has the same problem but hides it better. [0] [LINK] [1] this video
Python 3 is useful for a beginner to start with because they don't have to learn the deprecated cruft that Python 2 preserves for compatibility. If they just learn the Python 3 way of doing things, it generally works in both 2 and 3. In his explanation of classes he has several paragraphs about old-style vs. new-style classes, ending it with: "Just completely ignore the idea of old style versus new style classes and assume that Python always requires (object) when you make a class. Save your brain power for something important." Python 3 only has new-style classes, so the entire explanation could have been left out, allowing the beginner to, as he recommends, focus on more interesting things. Another example is his advice, in exercise 11, to avoid the input() function because of its security problems. Python 3 doesn't have that function, so the beginner doesn't have to remember to avoid it. And of course, there's Unicode, which this book seems to completely ignore. A beginner starting with Python 3 has to learn the "Unicode sandwich" approach[1], which applies very well to Python 2 code as well. But someone starting with Python 2 can easily be confused about the concept (because the language is confused) and will have a hard time getting things to work correctly. For example, the book recommends that people "from another country" set their source encoding to UTF-8 – good luck printing things on Windows. [1] this video
I don't know the details (because I don't understand unicode) but I am not sure what you said is possible. But to me ASCII seems like a hindrance. ``` fmt.Println("Hello, 世界") ``` This line of go lang looks beautiful. I bet they never have to worry about bytes vs Unicode. Here is python... this video I'll add one more assertion to the list. People are born and people die. It is not fair for us to impose our mistakes and sins to the next generation. With that being said, my thought was simply all the scripts that are in python (even the simple build for Firefox was python 2 only last time I checked). The way forward for anyone new to python is for us to tell them: Python 2 doesn't matter. Don't look at it. Learn python by which I mean 3.0+ I still don't get why Google of all places can't support python 3 in its standard app engine. To me it sounds like standard is deprecated for the flexible (which doesn't have a free tier).