Initially, I’m not handling it at all… the expectation is that the thing
that happens after fork() is exec(). (In which case the whole image is
replaced, so it does not matter.) If you reenter the library in the child
after the fork, the library will “panic” — emit an error message to stderr,
and call abort(). Essentially I’m making it explicit that this operation
is verboten.
In the future, if I ever do the work to make this really fork safe, where
we completely reinitialize the library in the child, I’ll need to use a
different allocator. I can either track all memory objects allocated via
my nni_alloc()
routine on a linked list, or if I have an allocator that lets me manage the
entire arena, I can just discard the arena. The latter is preferable in
that it eliminates a lot of overhead, and I don’t have to figure out how to
keep track of the structures allocated separately.
If I have an arena based approach, or slab based approach, I can avoid
having to block fork operation except when I grow the arena. But when I
do, or on every alloc/free if I keep a global linked list, I’d have to
acquire a resource that blocks fork (such as a reader writer lock, that is
acquired in write mode in prefork(), and released in parent().)
This is a fair bit of complexity, for an uncommon use case, so I’m not
investing any time in it at present, now that I’ve verified that there is a
clear path to making this work in my implementation if there is a need.
Btw, this same approach could be used with nanomsg currently — you’d just
basically have to provide an alternate implementation for nn_alloc() and
nn_free(). With these the linked list approach looks not so bad since they
are already doing some fairly dirty things under the covers.
Note that this will do *nothing* to recover memory allocated by the C
library for underlying things like pthread structures or internal buffers
used for open files. You can see why the problem is rather thorny — so
much so that the POSIX committee was convinced to abandon efforts to solve
it.
On Wed, Dec 14, 2016 at 10:37 AM, Karan, Cem F CIV USARMY RDECOM ARL (US) <
cem.f.karan.civ@xxxxxxxx> wrote:
As you wish. Can you post how you handle the memory leaks after the
fork()? I'm curious to see what you're going to do.
Thanks,
Cem Karan
-----Original Message-----On Behalf Of Garrett D'Amore
From: nanomsg-bounce@xxxxxxxxxxxxx [mailto:nanomsg-bounce@xxxxxxxxxxxxx]
Sent: Wednesday, December 14, 2016 10:57 AMfork()-safe implementation
To: nanomsg@xxxxxxxxxxxxx
Subject: [nanomsg] Re: [Non-DoD Source] Re: On pthread_atfork(), and
the identity of the sender, and confirm the authenticity of all links
All active links contained in this email were disabled. Please verify
contained within the message prior to copying and pasting the address toa Web browser.
track of my own objects. (Kind of critical when you work inside
________________________________
Yeah…um… as an operating system engineer, I generally believe I can keep
a kernel.) In fact, in libnng I require that the system provide the*size* of the object with the object at free() time. This is to permit
porting to platforms where this is a requirement. (Such a requirementexists in the Solaris kernel.) It also allows the use of much much
more efficient allocators, like slab allocators, and not having to stashthe size with the object (which is often automatically known at
*compile* time), so you save quite a bit of lookup, and can improve theodds of having your object aligned on a natural boundary (such as
a page).effort, and so a GC is kind of out of the question.
Portability to embedded systems is really important to me in this
< Caution-mailto:mwpowellhtx@xxxxxxxxx ;> > wrote:
On Wed, Dec 14, 2016 at 6:50 AM, Michael Powell <mwpowellhtx@xxxxxxxxx
civ@xxxxxxxx > > wrote:
On Wed, Dec 14, 2016 at 9:07 AM, Karan, Cem F CIV USARMY RDECOM ARL
(US) <cem.f.karan.civ@xxxxxxxx < Caution-mailto:cem.f.karan.
> Have you considered using a garbage collector? E.g. Caution-http://www.hboehm.info/gc/ ;< Caution-
http://www.hboehm.info/gc/ ;> . Looking through the header file, itappears that there are calls specifically for handling forks
(GC_set_handle_fork(), GC_atfork_prepare(), GC_atfork_parent(),GC_atfork_child(), and GC_start_mark_threads()). Based on the
documentation surrounding GC_start_mark_threads(), it appears that thecollector can handle fork()s that are not followed by an exec().
That may solve the memory leak issues cleanly. There are also functionsto register finalization methods, so that should handle dealing
with file pointers, etc. that you want to close eventually.coding
What does a GC gain you, but to make further excuses for poor
practices, in the first place? Been there, done that, don't needfast; I allocate a ridiculous number of short-lived objects, and even
another T-shirt.
> I use that particular collector in my own work, and it is quite
then, the profiler shows that garbage collection takes less than 1% ofthe runtime. I've never tried forking a child though, so I don't know
how well that part works.nanomsg-bounce@xxxxxxxxxxxxx > [Caution-mailto:nanomsg-
>
> Thanks,
> Cem Karan
>
>> -----Original Message-----
>> From: nanomsg-bounce@xxxxxxxxxxxxx < Caution-mailto:
bounce@xxxxxxxxxxxxx < Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> ]On Behalf Of Garrett D'Amore
>> Sent: Wednesday, December 14, 2016 1:53 AMfreelists.org >
>> To: nanomsg@xxxxxxxxxxxxx < Caution-mailto:nanomsg@
>> Subject: [Non-DoD Source] [nanomsg] Re: On pthread_atfork(),and fork()-safe implementation
>>verify the identity of the sender, and confirm the authenticity of all
>> All active links contained in this email were disabled. Please
linksaddress to a Web browser.
>> contained within the message prior to copying and pasting the
>>hours this evening trying to bake in a solution. I eventually had to
>>
>> ________________________________
>>
>>
>>
>> Well I thought I had a brilliant idea, and I spent a number of
throw my*only* any memory used by mutexes and condvars. That’s definitely
>> hands up in the air.
>>
>> I can see that it *is* possible to build a solution that leaks
possible.and it requires you to basically build the equivalent of an operating
>> The problem is, the work you have to do for this is extreme,
system infork-safe vs. unsafe, etc. The problem is that in order to
>> some ways. I had a scheme to suspend threads, and mark regions
avoid leakingsingle memory object in your system has to be globally
>> memory, you pretty *have* to manage your own heap — as in every
discoverable.to build your own memory manager, since some memory
>> This turns out to be rather inconvenient if you don’t also want
objects are going“orphaned” in that they didn’t have any global state to them, only
>> to be used by threads, and frankly I had objects that were
locally usedmanager that will let me reclaim every allocated object in the
>> inside functions in threads.
>>
>> One day I may come back to this, by supplying my own memory
systemswoop). I’d also need a way to reclaim files, and handle mutexes and
>> (perhaps simply by reclaiming the entire heap in one fell
condvarsit can be done in the platform layer. Which means it can be done
>> “magically”. I’m pretty sure I know how to do that, and that
in theI’m willing to take the larger action to stop using “ordinary” memory
>> future, as a fairly straight-forward retrofit, once I decide
>> management. I’ve got enough other stuff to do in the meantime,that I’m taking my earlier action, which is to panic when the
usergarrett@xxxxxxxxxx < Caution-mailto:garrett@xxxxxxxxxx ;> < Caution-
>> attempts to reenter the library from the child after fork().
>>
>> - Garrett
>>
>>
>> On Tue, Dec 13, 2016 at 8:51 AM, Garrett D'Amore <
Caution-mailto:garrett@xxxxxxxxxx ;< Caution-mailto:garrett@xxxxxxxxxxthings in the new design. I had implemented freeze and thaw and
> > wrote:
>>
>>
>> Thanks. I had planned to design a fork safe version of
resetwould have worked well. Until I discovered that the child side
>> entry points at various points and was pretty sure that this
version was notside action might be reasonable and lead to a working solution.
>> allowed to call any mutex functions or to call free.
>>
>> I will think about this some more. Delaying the child
>>franklinmathieu@xxxxxxxxx < Caution-
>> Sent from my iPhone
>>
>>
>> > On Dec 13, 2016, at 12:20 AM, Franklin Mathieu <
mailto:franklinmathieu@xxxxxxxxx ;> < Caution-Caution-mailto:franklinmathieu@xxxxxxxxx < Caution-
mailto:franklinmathieu@xxxxxxxxx ;> >one that initially
>> > wrote:
>> >
>> > I'm going to give my 2 cents on the matter as I was the
>> > opened the github issue regarding fork()-safety and Ihad the time
>> > to work with different approaches on the matter.that relies on
>> >
>> > I've been maintaining an unit testing framework for C
>> > worker processes to run tests safely, and as such, forthe longest
>> > time, this had been implemented with fork() without asubsequent exec().
>> > I recently switched the I/O layer of the framework touse nanomsg
>> > because it was simple, and it was much more "correct"than what
>> > I had been doing before with pipe() shenanigans.implementing
>> >
>> > However, as nanomsg isn't fork()-safe, I took a swab at
>> > a fork()-safety mechanism, which ended up being brittlebut was
>> > "good enough" for my purposes, and I reworked otherdependencies
>> > to make sure they handled forks correctly.of it right at the
>> >
>> > The problem with fork()-safety is that unless you think
>> > design of the software, you're going to end up doingsomething hack-ish;
>> > which means that the rewrite could be a good startingpoint to actually
>> > implement the structural basis towards fork()-safety.POSIX might be
>> > right on target with the problems caused bypthread_atfork(), but in
>> > practice there is a lot of wiggle room to do what wemust to make
>> > things work at fork.fork()-safety.
>> >
>> > With all of that being said, I've given up myself on
>> > The fact is that there is no single silver bullet toaddress this,
>> > that a lot of software is expecting exec() to be calledafter a fork(),
>> > and that there aren't many use cases in having workerprocesses.
>> >worker
>> > I ended up writing a library dedicated to spawning
>> > processes [1] in a manner that calls fork() thenre-exec()s the current
>> > executable with a patched main function, which whilenot ideal, is
>> > in my opinion less of a hack than having to make thesoftware and
>> > all of its dependencies fork-safe().panicking
>> >
>> > This is why I understand your decision of giving up and
>> > the process on fork-reentry. You might also be able tocompromise
>> > by only allowing calls to nng_socket_create after fork,which could under
>> > the covers completely drop the current invalid stateand just reinitialize
>> > the library. This would cause a resource leak, butallow the usage
>> > of sockets in the child for those that really want it.github.com/diacritic/BoxFort < Caution-https://github.com/
>> >
>> > [1]: Caution-Caution-https://
diacritic/BoxFort > < Caution-
Caution-https://github.com/diacritic/BoxFort ;< Caution-https://github.com/diacritic/BoxFort ;> >
>> >garrett@xxxxxxxxxx < Caution-mailto:garrett@xxxxxxxxxx ;> <
>> > 2016-12-12 19:31 GMT+01:00 Garrett D'Amore <
Caution-Caution-mailto:garrett@xxxxxxxxxx ;< Caution-mailto:garrett@xxxxxxxxxx > > >:
>> >> The following conversation relates to using fork()with nanomsg (or future
>> >> rewrites), where you do *not* immediately callexec(). Using fork() and
>> >> then immediately calling exec() is fine, and willcontinue to work as it
>> >> always.e.g. a child worker
>> >>
>> >> But some people want to use fork() to spawn children,
>> >> process, that communicates back to the parentsomehow. This is never going
>> >> to work.pthread_atfork() as part of an
>> >>
>> >> I’ve been doing a bit more research into
>> >> attempt to make my new nng library properlyfork()-safe. I’ve more or less
>> >> given up though.given up — see
>> >>
>> >> The reason for this is that even the OpenGroup has
>> >> Caution-Caution-http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_atfork.html < Caution-
http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_atfork.html > < Caution-
>> Caution-http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_atfork.html < Caution-
http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_atfork.html > >
>> >> — and especially the RATIONALE section, for the logicbehind this. They
>> >> have even indicated plans to deprecate thepthread_atfork() API altogether.
>> >>the library fork() safe
>> >> Essentially, it isn’t possible to make a version of
>> >> as it would be necessary to free resources, do locks,etc. — i.e. all those
>> >> Async-Signal-Unsafe calls.libnanomsg, I will be
>> >>
>> >> So, for libnng, and possibly in the future for
>> >> changing the API so that if you attempt to callbackinto the library after
>> >> fork(), it will actually panic the process.be called to close any
>> >>
>> >> I probably will also arrange for pthread_atfork() to
>> >> file descriptors that were not marked close-on-exec…https://diacritic.io ;> < Caution-Caution-https://diacritic.io ;< Caution-
>> >>
>> >> Stay tuned for more details.
>> >>
>> >> - Garrett
>> >
>> >
>> > --
>> > Franklin "Snaipe" Mathieu
>> > 🝰 Caution-Caution-https://diacritic.io ;< Caution-
https://diacritic.io ;> >
>> >
>>
>>
>
