[haiku-development] What to do with termcap? Scene II.

Dear Colleagues,

silence gives consent so I have played a bit with hypotetical switching Haiku to terminfo. More info about my results and experience is coming below but, at first, let me repeat why I think the termcap should be buried:

Siarzhuk Zharski писал 30.01.2013 15:31:
during the last week I was playing in my sandbox and going to reduce
the population of Terminal bugs. Those bugs that are related to
various control sequences parsing, colors rendering etc. And I have
predictably landed in our current termcap database that looks
definitely outdated nowadays and was hacked some times for our needs.
Frankly speaking I convinced that we must adjust our Terminal
emulation functionality to mimic declared terminal type as close to
it's most used  capabilities set as possible. That is why I have
looked for termcap update first.

During seeking for more actual versions of termcap I visited
"Termcap/Terminfo Resources Page" at
http://www.catb.org/~esr/terminfo/ that looks like origin of our
termcap database version too. This resource pretends to be a "master"
of termcap/terminfo databases. The problem with termcap file provided
on this page is that it is broken. Both terminfo and termcap databases
available on this page are generated from so known master termtypes
file. Generation of termcap database is just compiling this file using
"tic -C" command. This process has one unpleasant "feature" - it
comments out the capabilities that cannot be converted from terminfo
to termcap format safely. I suspect just because of tic laziness.

On the good side the sanity check which performed by every
ncurses-based application can catch some logical problems in terminal
descriptions. Look into https://dev.haiku-os.org/ticket/1957 for
results of this check. But on the bad side a lot of commented-out
capabilities cannot be catch. And we have such broken termcap database
- some of "un-paired" capabilities were hacked during fixing #1957 but
there are still about 1100 commented out capabilities - so
corresponding terminal entries cannot be called valid.

The only actual source of terminal capabilities information I have
found, is the terminfo.src file provided with ncurses sources
distribution. This file was derived from the termtypes.master
published years ago at mentioned above "Termcap/Terminfo Resources
Page". Obviously it can be converted into termcap using tic compiler,
but the result is unacceptable because of many commented out
capabilities. But anyway I think it is a good idea to keep our termcap
database synchronized with the version provided by ncurses.
So the first opportunity to bring termcap in order is just to use
manually corrected database generated from the ncurses'
termtypes.master. Another one - switch to using terminfo. As far as I
have shortly seen, ncurses can supersede the libtermcap functionality
completely.

Note that for interactive terminal applications the start time will be reduced using terminfo.

Well, the status quo is following:

Following modules are linked to libtermcap:
- bin/top
- bin/tput
- apps/debugger
- bin/bash
- bin/less
- bin/network/telnet
- bin/network/telnetd

Following ones are linked to libncurses:
- bin/watch
- bin/gdb
- bin/network/ftp
- bin/network/telnet

At first step I have removed libtermcap module and switched all modules linked to it to use ncurses' libtermcap emulation. Quick test of the new build show no visible problems in corresponding software. Additionally I have replaced bin/tput (originated from NetBSD AFAIK) with the same tool provided by ncurses. More info you can see in corresponding commit: https://github.com/siarzhuk/haiku/commit/aba22c75f14424e6466c53d2688fb9717de4c288. Note that <etc>termcap database itself will be still required for currently available optional packages and, probably, for R5 binary compatibility, so it should be provided with the Haiku installation either until the next "lets recompile all packages" campaign or until the Holy Cow of binary compatibility will be slaughted.

The second step was updating our stock ncurses sources from version 5.5 up to version 5.9. It was required to bring the terminfo.src to actual revision and avoid possible compatibility problems using 5.5 library with modern terminfo.src. There was nothing interesting - just some "passing" questions: a) Are there any reasons to install libtermcap.a with the Development optional package? I think anyone going to develop or build ncurses-based applications on Haiku should be forced to use normal ncurses installation. And having our restricted libtermcap.a may confuse configure scripts for 3rd-party aoftware. b) Currently we have ncurses without wide characters support that can produce problems in hypothetical case of localizing corresponding tools. May be we should activate wide characters support for the stock ncurses now?

The third step was to add terminfo database distribution support for every type of Haiku builds. It was a bit tricky because ncurses distribution model requires issuing so known terminfo compiler (tic) to compile the database directly on the target system using terminfo.src as source. Keeping in mind all cases of cross-compiling I have seen following variants:

a) Use the ncurses' terminfo compiler tool (tic) provided by build host system. Note that tic is heavily dependent on libncurses functionality so full-blown ncurses installation will become pre-requisite for all host build systems. For Haiku that means creating ncurses optional package, outsorcing the stock version of ncurses and modifying build system correspondently. Sooner or later it will be implemented in this way, but at the moment I had not enough time resources for this variant :-(

b) Provide precompiled version of terminfo database as optional package. It looks like the easiest one but my self-respect doesn't allow me to go such way. If something can be automated at reasonable cost - it must be automated;

c) Install only terminfo.src and compile the database on the first start of system. I found burdening the user with yet another notify window for visibly long time unacceptable - no, no, no.

d) Implement the light-weight and portable terminfo compiler tool and use it during system build. Binary format of terminfo database entries is straightforward so the task has looked easy for me. Programmers love to reinvent bicycles and writing my own one - rtic (reduced terminfo compiler) was not tedious. The rtic is checked on FreeBSD/Linux and produces terminfo database that is functionally compatible with one from the ncurses' tic output. In corresponding commit (https://github.com/siarzhuk/haiku/commit/568e8bd8b30ee76ad1f4787caa94537b29e76f89) you can look onto rtic sources and my silly attempts to integrate it into Haiku build process.

So, I would like to propose switching to terminfo in this way in case no serious objections from your side. I have checked system modules that may be affected by this switch and observe no visible issues. In some cases, like "gdb -tui" they behave even more clear - without start delays and massive name collision complains.

Please share your suggestions, objections and advices. Thank you for the patience and have a nice Weekend.

--
Kind Regards,
   S.Zharski

Other related posts: