[haiku-development] Re: BString and UTF-8

  • From: Michael Bridgers <mibrid@xxxxxxx>
  • To: haiku-development@xxxxxxxxxxxxx
  • Date: Fri, 02 Dec 2011 20:44:09 -0500

On 12/2/2011 11:08 AM, Axel Dörfler wrote:

This all sounds like very welcomed changes, but this one cannot be done
in a backward compatible manner, unfortunately.
As long as all *Chars() methods deal with invalid UTF-8 correctly (as
they should), there is no security risk either. I would just add a
method IsValidUTF8() or something to that degree which you can use to
test it.


I'm certain that it CAN be done in a backwardly compatible manner.

And *Chars methods, by definition, can't deal correctly with invalid UTF-8 strings. What should be done with a "replace char" method when, for example, the code point has one lead byte and 9 trailing bytes? There is no way that anything meaningful could be done.

If you look at how ICU handles things, it doesn't allow invalid strings. If you try to create a UnicodeString with invalid input, it replaces the invalid code points with the replacement character, 0xfffd. Other systems that use Unicode do similar things.

I know that some security exploits have passed executable code as a string as a means to breach security of an OS. At the "Internationalization and Unicode Conference" in 2002, there was a paper presented that talked about security considerations with UTF-8. (http://unicode.org/iuc/iuc22/a323.html)

If you will give my changes a chance, I think you will see that everything I'm doing will have a positive effect on the BString class and Haiku.

Also, I'm not a committer. Someone will have to verify what I'm doing before it will be committed. I'm not going to submit something that will break things, because I know it will never make it into the source tree.

And yes, I already have a static IsValidUTF8() method on the BString.


Other related posts: