FreeLists / openbeos / [openbeos] StyledEdit issues : text encodings, opentracker integration, etc.

["shatty" <shatty@xxxxxxxxxxxxx>]

Hello all,

I have been tweaking away at Mattias Sundblad's excellent StyledEdit 
implementation in order to increase its compatibility and conformance 
to the user interface experience in R5.  I'm happy to say that it's 
approaching a complete replacement.  Most people will probably say that 
it is complete already if they try it.

However, there are four issues I have hit up against.  I'd appreciate 
any feedback. [but especially insightful feedback :-) ]

Andrew

* Issue 1: text encodings

You knew it was going to happen sooner or later.  Some may not be aware 
that our friendly StyledEdit can handle multiple text encodings, and 
save to different text encodings.  Unfortunately, it is limited to a 
relatively small fixed set of encodings that doesn't even include some 
east asian encodings.

So, I was looking at UTF8.h and also at Font.h.  One approach would be 
to use the conversion routines provided in UTF8.h.  These routines are 
not extensible without recompilation.  In R5 there are about 24 
encodings, which don't even include Big-5 or GB2312. (the two major 
chinese encodings)  Also, if we used this approach, we would need to 
recompile both libtextencoding.so and StyledEdit in order to make new 
encodings available. (YUCK!)  This approach also has another problem.  
Consider a user that loads a file and resaves it as a new file with no 
changes.  Converting the file's text into UTF8 for editing and then 
back to the previous encoding for saving may produce a file that is not 
identical!  This is because some of the conversions are not reversible. 
:-(

A better approach solves the latter problem.  Instead of using the 
conversion routines, the file is loaded as is and the font is set to 
the encoding of the file using BFont::SetEncoding.  This removes the 
possibility of information loss due to encoding conversion.  However, 
this reduces the supported encodings to 12, which doesn't even include 
ShiftJIS, a major japanese encoding.  One must also be careful when 
using Find/Replace that those strings are in the same encoding or the 
search may fail where it should succeeed.  This approach still suffers 
from the recompilation problem of the above.

I would prefer us to take an approach where we use add-ons to perform 
encoding conversions.  Then new encodings can be added just by putting 
in a new add-on.  As part of this we should also have a way for 
encodings to provide name strings.  This will avoid the ugliness of 
hard-coding names into the application using the encodings.  It will 
also make it possible to not have to recompile an application to add 
user interface access to new encodings.

This brings up its own issues: how will these add-ons interact with the 
font rendering system?  What should the interface for these add-ons 
look like?  (it should be pretty simple I think.)

I was wondering if anyone has had experience with text encoding 
conversions on some other platform or using some popular libraries such 
as libiconv.so (can we use it? it's LGPL..) and if they could 
contribute some suggestions here.

* Issue 2: opentracker integration

Anybody who has looked at open tracker knows that it is big.  There's a 
lot of code there between the tracker and deskbar.  Unsurprisingly, 
some of it would also be useful for OpenBeOS.  In the case of R5's 
StyledEdit it seems to be using the ModelMenuItem class to present the 
recently used files menu.  This adds some nice icons to the left of the 
names that allow you to tell something about a files type in some 
situations.

As far as I know the opentracker license is compatible with OpenBeOS 
license so we could take the code for this and put it in StyledEdit.  
However, ModelMenuItem uses Model, Model uses an icon cache, etc.  This 
is more than just a snippet of code.  It seems like this functionality 
should be made available from libtracker.so for all applications to 
enjoy, inlcuding StyledEdit.

It seems like more and more of these kinds of code sharing situations 
are going to develop between OpenBeOS and open tracker.  Should we 
integrate our source control repositories?  (is it even possible 
legally?)  Or should we take pieces from OpenTracker that are useful 
and copy them into our cvs?  (same code two places = trouble)  Or maybe 
we require someone to download opentracker sources to compile openbeos?  
(I can't believe I even mentioned that idea. :-P )

* Issue 3: font sensitivity

This is another of those big issues that's going to crop up everywhere.  
The original R5 dialogs for Find/Replace in StyledEdit didn't look that 
good in my 12 pt Bistream Cyberbit, and the new ones look worse.  At 
some point I will probably go and figure out how to compute the font 
height properly and adjust the layout for the small dialogs in 
StyledEdit.  It would be nice to have some systematic approach to this 
rather than just doing it ad-hoc in each situation.  I'm not saying we 
need to implement layout management (yet) but perhaps there are some 
helper functions we can add to some of the text classes.  We could at 
least have some conventions about how to do appropriate layout.  (Ideas 
anybody?  somebody made their app nice and font sensitive?)

* Issue 4: performance

When opening a file our implementation seems slower than R5's.  It's 
unclear what is the reason for this.  It'd be nice if someone could 
look into it.