#16106: [kernel] vfork: Out of memory when building haiku with plenty of RAM
----------------------------+----------------------------
Reporter: diver | Owner: nobody
Type: bug | Status: new
Priority: normal | Milestone: Unscheduled
Component: System/Kernel | Version: R1/Development
Resolution: | Keywords:
Blocked By: | Blocking:
Platform: All |
----------------------------+----------------------------
Comment (by pulkomandy):
The two issues are probably unrelated. The design of vfork() is that it
essentially makes a copy of the existing running program. All the
variables, etc need to be copied. Quite often, this is very temporary
because after a vfork(), the new team does an exec() and replaces all this
with the new executable to run.
This is a well-known limitation of the way to run things in UNIX. In the
case of jam in particular, the whole context of jam (which is large
because it stores all the build tree in memory) has to be forked everytime
it wants to start a new job. So the build process is known to require a
lot of RAM (2GB plus swap is not unheard of and it has been so for a long
time, I think this was already mentionned in alpha1 release notes, even).
Linux works around it by using overcommit memory for this and hoping for
the best (that is, that the program will not try to access too much
variables after forking, otherwise it would crash in an unrecoverable way
there). In BeOS and Haiku, this is handled by normally not using fork/exec
but starting apps directly from a clean context (BRoster::Launch does
this). AboutSystem is started using the latter way, so it can't be a
problem with vfork.
Here is the listimage for AboutSystem:
{{{
TEAM 2057 (/boot/system/apps/AboutSystem):
ID Text Data Seq# Init# Name
--------------------------------------------------------------------------------
5077 0x01333000 0x01353000 0 0 /boot/system/apps/AboutSystem
5075 0x60e7e000 0x00000000 0 0 commpage
5076 0x01688000 0x016a7000 0 0 /boot/system/runtime_loader
5078 0x01978000 0x019aa000 0 0
/boot/system/lib/libstdc++.r4.so
5079 0x00d43000 0x01013000 0 0 /boot/system/lib/libbe.so
5080 0x002e8000 0x002ff000 0 0
/boot/system/lib/libtranslation.so
5081 0x00517000 0x005f2000 0 0 /boot/system/lib/libroot.so
5082 0x025cf000 0x03e4f000 0 0
/boot/system/lib/libicudata.so.57.2
5083 0x01d9f000 0x02030000 0 0
/boot/system/lib/libicui18n.so.57.2
5084 0x012b4000 0x012bd000 0 0
/boot/system/lib/libicuio.so.57.2
5085 0x009c5000 0x00a18000 0 0
/boot/system/lib/libicule.so.57.2
5086 0x0188c000 0x01897000 0 0
/boot/system/lib/libiculx.so.57.2
5087 0x008dc000 0x00915000 0 0
/boot/system/lib/libicutu.so.57.2
5088 0x00b68000 0x00cda000 0 0
/boot/system/lib/libicuuc.so.57.2
5089 0x02151000 0x0216b000 0 0
/boot/system/lib/libz.so.1.2.11
5090 0x010f4000 0x011eb000 0 0
/boot/system/lib/libtextencoding.so
5092 0x0182f000 0x0183f000 0 0 /boot/system/lib/libroot-
addon-icu.so
5093 0x010c2000 0x010df000 0 0 /boot/system/add-
ons/Translators/NanoSVGTranslator
5094 0x01918000 0x01928000 0 0 /boot/system/add-
ons/Translators/WonderBrushTranslator
5095 0x020d5000 0x020e1000 0 0 /boot/system/add-
ons/Translators/WebPTranslator
5096 0x00964000 0x009b5000 0 0
/boot/system/lib/libwebp.so.7.1.0
5097 0x0089c000 0x008a8000 0 0 /boot/system/add-
ons/Translators/TIFFTranslator
5098 0x02305000 0x0236e000 0 0
/boot/system/lib/libtiff.so.5.5.0
5099 0x01c9b000 0x01d25000 0 0
/boot/system/lib/libjpeg.so.62.3.0
5100 0x010ae000 0x010bb000 0 0 /boot/system/add-
ons/Translators/TGATranslator
5101 0x004c4000 0x004d1000 0 0 /boot/system/add-
ons/Translators/SGITranslator
5102 0x00785000 0x00796000 0 0 /boot/system/add-
ons/Translators/RTFTranslator
5103 0x0126d000 0x0128e000 0 0 /boot/system/add-
ons/Translators/RAWTranslator
5104 0x01628000 0x01637000 0 0 /boot/system/add-
ons/Translators/PSDTranslator
5105 0x0179e000 0x017a9000 0 0 /boot/system/add-
ons/Translators/PPMTranslator
5106 0x00754000 0x00760000 0 0 /boot/system/add-
ons/Translators/PNGTranslator
5107 0x011ee000 0x01218000 0 0
/boot/system/lib/libpng16.so.16.37.0
5108 0x015dd000 0x015e7000 0 0 /boot/system/add-
ons/Translators/PCXTranslator
5109 0x00882000 0x00892000 0 0 /boot/system/add-
ons/Translators/JPEG2000Translator
5110 0x012c2000 0x01329000 0 0
/boot/system/lib/libjasper.so.4.0.0
5111 0x01223000 0x01237000 0 0 /boot/system/add-
ons/Translators/JPEGTranslator
5112 0x01d90000 0x01d9b000 0 0 /boot/system/add-
ons/Translators/ICNSTranslator
5113 0x021d7000 0x021e6000 0 0
/boot/system/lib/libicns.so.1.2.0
5114 0x006b5000 0x006fb000 0 0
/boot/system/lib/libopenjp2.so.2.1.2
5115 0x01bd3000 0x01bdf000 0 0 /boot/system/add-
ons/Translators/ICOTranslator
5116 0x0042f000 0x00439000 0 0 /boot/system/add-
ons/Translators/HVIFTranslator
5117 0x00a68000 0x00a7c000 0 0 /boot/system/add-
ons/Translators/GIFTranslator
5118 0x020bc000 0x020c7000 0 0 /boot/system/add-
ons/Translators/EXRTranslator
5119 0x03f29000 0x041e2000 0 0 /boot/system/lib/libIlmImf-
2_2.so.23.0.0
5120 0x003b1000 0x0040e000 0 0 /boot/system/lib
/libIlmImfUtil-2_2.so.23.0.0
5121 0x01d4a000 0x01d8c000 0 0
/boot/system/lib/libHalf.so.23.0.0
5122 0x020f0000 0x0211a000 0 0 /boot/system/lib/libIex-
2_2.so.23.0.0
5123 0x01774000 0x0177a000 0 0 /boot/system/lib/libIexMath-
2_2.so.23.0.0
5124 0x0076b000 0x00776000 0 0 /boot/system/lib
/libIlmThread-2_2.so.23.0.0
5125 0x0064a000 0x00669000 0 0 /boot/system/lib/libImath-
2_2.so.23.0.0
5126 0x0171b000 0x01727000 0 0 /boot/system/add-
ons/Translators/BMPTranslator
5127 0x020a6000 0x020b1000 0 0 /boot/system/add-
ons/Translators/STXTTranslator
}}}
I already see that all translators are loaded, which seems not really
needed. That would save quite a lot on both memory usage and loading time.
The translation kit is used to load the logo, which we could instead
convert to raw BBitmap data to not need the translation kit. Or, we could
make it so that the translation kit does not load all translators when a
specific format is asked for the translation (AboutSystem knows and
specifies that the logo is in PNG format).
the next contributor to memory use is probably libicudata, which is 20MB
on its own. There are also two versions of it for gcc2 and gcc8. We could
try to use a .dat file instead which could be shared by the two
architectures, however I don't know if that will still work when mixing
icu version (the format of the files changed sometimes, so we'll have to
review this for each version we want to update to if we go this way). We
can also probably remove a few things from the lib/datafile to save space
and memory (see http://userguide.icu-project.org/icudata).
--
Ticket URL: <https://dev.haiku-os.org/ticket/16106#comment:2>
Haiku <https://dev.haiku-os.org>
The Haiku operating system.
