UnicodeDecodeError

Типичная для нерусских нелатинских питонистов история. Все, кому для текстов недостаточно ASCII таблицы, наступают на грабли типа

UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)

Почему так, с чем это связано, какова механика и, главное, что с этим делать?

Покровы срывает Mikhail Korobov:

__str__ and __repr__ must return byte strings under Python 2.x, but it is not clear what encoding should they use. It is possible to have all the default encodings different on the same machine at the same time.

...

It is tempting to encode the values of __str__ and __repr__ to sys.stdout.encoding, and maybe set system default encoding to sys.stdout.encoding. This way REPL and print would work as expected and the result would be readable when it is possible.

...

The second option is to make __str__ and __repr__ return 7bit ASCII, using escaping and/or transliteration.

...

The third option is to always encode __str__ and __repr__ results to UTF-8.

...

All the tricks above are only necessary for Python 2.x because Python 3.x fixed the main issue: __repr__ and __str__ must return unicode in Python 3.x. Developers shouldn't worry about sys.stdout.encoding and many other things under Python 3.x because when the expected encoding is known, unicode would be encoded using this encoding.

Error handling become stricter; issues like 2.x %r issue are also not possible under Python 3.x.

I think that the best way is to write code with Python 3.x semantics (__str__ and __repr__ should return unicode) and have a decorator for fixing them for Python 2.x; this idea is from a thread at django-developers. Decorator then may be removed in a (far?) future alongside with dropping Python 2.x support.

http://kmike.ru/python-with-strings-attached/

Мой вариант — все переводить в UTF-8. Но, по любому, вариант с декоратором следует иметь в виду.

original post http://vasnake.blogspot.com/2013/01/unicodedecodeerror.html