Well I thought I had a brilliant idea, and I spent a number of hours this
evening trying to bake in a solution. I eventually had to throw my hands
up in the air.
I can see that it *is* possible to build a solution that leaks *only* any
memory used by mutexes and condvars. That’s definitely possible. The
problem is, the work you have to do for this is extreme, and it requires
you to basically build the equivalent of an operating system in some ways.
I had a scheme to suspend threads, and mark regions fork-safe vs. unsafe,
etc. The problem is that in order to avoid leaking memory, you pretty
*have* to manage your own heap — as in every single memory object in your
system has to be globally discoverable. This turns out to be rather
inconvenient if you don’t also want to build your own memory manager, since
some memory objects are going to be used by threads, and frankly I had
objects that were “orphaned” in that they didn’t have any global state to
them, only locally used inside functions in threads.
One day I may come back to this, by supplying my own memory manager that
will let me reclaim every allocated object in the system (perhaps simply by
reclaiming the entire heap in one fell swoop). I’d also need a way to
reclaim files, and handle mutexes and condvars “magically”. I’m pretty
sure I know how to do that, and that it can be done in the platform layer.
Which means it can be done in the future, as a fairly straight-forward
retrofit, once I decide I’m willing to take the larger action to stop using
“ordinary” memory management. I’ve got enough other stuff to do in the
meantime, that I’m taking my earlier action, which is to panic when the
user attempts to reenter the library from the child after fork().
- Garrett
On Tue, Dec 13, 2016 at 8:51 AM, Garrett D'Amore <garrett@xxxxxxxxxx> wrote:
Thanks. I had planned to design a fork safe version of things in the new
design. I had implemented freeze and thaw and reset entry points at various
points and was pretty sure that this would have worked well. Until I
discovered that the child side version was not allowed to call any mutex
functions or to call free.
I will think about this some more. Delaying the child side action might
be reasonable and lead to a working solution.
Sent from my iPhone
On Dec 13, 2016, at 12:20 AM, Franklin Mathieu <franklinmathieu@xxxxxxxxx> wrote:
initially
I'm going to give my 2 cents on the matter as I was the one that
opened the github issue regarding fork()-safety and I had the timethe
to work with different approaches on the matter.
I've been maintaining an unit testing framework for C that relies on
worker processes to run tests safely, and as such, for the longest
time, this had been implemented with fork() without a subsequent exec().
I recently switched the I/O layer of the framework to use nanomsg
because it was simple, and it was much more "correct" than what
I had been doing before with pipe() shenanigans.
However, as nanomsg isn't fork()-safe, I took a swab at implementing
a fork()-safety mechanism, which ended up being brittle but was
"good enough" for my purposes, and I reworked other dependencies
to make sure they handled forks correctly.
The problem with fork()-safety is that unless you think of it right at
design of the software, you're going to end up doing something hack-ish;reinitialize
which means that the rewrite could be a good starting point to actually
implement the structural basis towards fork()-safety. POSIX might be
right on target with the problems caused by pthread_atfork(), but in
practice there is a lot of wiggle room to do what we must to make
things work at fork.
With all of that being said, I've given up myself on fork()-safety.
The fact is that there is no single silver bullet to address this,
that a lot of software is expecting exec() to be called after a fork(),
and that there aren't many use cases in having worker processes.
I ended up writing a library dedicated to spawning worker
processes [1] in a manner that calls fork() then re-exec()s the current
executable with a patched main function, which while not ideal, is
in my opinion less of a hack than having to make the software and
all of its dependencies fork-safe().
This is why I understand your decision of giving up and panicking
the process on fork-reentry. You might also be able to compromise
by only allowing calls to nng_socket_create after fork, which could under
the covers completely drop the current invalid state and just
the library. This would cause a resource leak, but allow the usagefuture
of sockets in the child for those that really want it.
[1]: https://github.com/diacritic/BoxFort
2016-12-12 19:31 GMT+01:00 Garrett D'Amore <garrett@xxxxxxxxxx>:
The following conversation relates to using fork() with nanomsg (or
workerrewrites), where you do *not* immediately call exec(). Using fork() and
then immediately calling exec() is fine, and will continue to work as it
always.
But some people want to use fork() to spawn children, e.g. a child
goingprocess, that communicates back to the parent somehow. This is never
lessto work.
I’ve been doing a bit more research into pthread_atfork() as part of an
attempt to make my new nng library properly fork()-safe. I’ve more or
functions/pthread_atfork.htmlgiven up though.
The reason for this is that even the OpenGroup has given up — see
http://pubs.opengroup.org/onlinepubs/9699919799/
altogether.— and especially the RATIONALE section, for the logic behind this. They
have even indicated plans to deprecate the pthread_atfork() API
safe
Essentially, it isn’t possible to make a version of the library fork()
thoseas it would be necessary to free resources, do locks, etc. — i.e. all
afterAsync-Signal-Unsafe calls.
So, for libnng, and possibly in the future for libnanomsg, I will be
changing the API so that if you attempt to callback into the library
anyfork(), it will actually panic the process.
I probably will also arrange for pthread_atfork() to be called to close
file descriptors that were not marked close-on-exec…
Stay tuned for more details.
- Garrett
--
Franklin "Snaipe" Mathieu
🝰 https://diacritic.io
