[nanomsg] Re: threading guidelines, and is traditional unix single threaded mode possible?

  • From: Nico Williams <nico@xxxxxxxxxxxxxxxx>
  • To: "nanomsg@xxxxxxxxxxxxx" <nanomsg@xxxxxxxxxxxxx>
  • Date: Tue, 24 Sep 2013 13:16:40 -0500

On Tue, Sep 24, 2013 at 1:21 AM, Martin Sustrik <sustrik@xxxxxxxxxx> wrote:
> On 23/09/13 22:44, Jason E. Aten wrote:
>> Obviously fork doesn't play well with multiple threads; i.e. [...]

fork() is rather evil as you can tell.

> I don't think this is possible. Unfortunately, POSIX doesn't provide a way
> to clean up resources after fork, except for exec. You'll end up with
> pending file descriptors etc.

Well, there's pthread_atfork().

> There are two options IMO:

My advice:

1. Use posix_spawn(), not fork()
2. If you can't posix_spawn() then use vfork() and then exec*() or
_exit() soon after (remember to use only async-signal-safe functions
in between)
3. If you can't posix_spawn() nor vfork() then fork() and exec*() or
_exit() soon after

4. If you must fork() and stick around (e.g., to daemonize) then do
this very soon after the parent starts and before you setup any
significant state (i.e., before starting to use nanomsg, or anything
else like, say OpenSSL or PKCS#11).  This is important.  I've seen
programs that setup state, then call daemon() -- this is a very
dangerous pattern.  Instead you should setup a pipe(), fork(), have
the parent block reading on the read end of the pipe, have the child
do everything it needs to do in order to get ready, then write() to
(or close()) the write end of the pipe to signal the parent that it
should exit.

Libraries should use pthread_atfork() to cleanup on the child side of
fork().  Keep in mind that when posix_spawn() is implemented in terms
of fork() instead of a true vfork() or true posix_spawn() system call
then atfork handlers will run when you spawn.

> 1. fork+exec
> 2. close all nanomsg sockets before forking

Yes, but it'd be nice if nanomsg could just close its internal
resources on the child side of fork() with an atfork handler and
arrange for existing open nanomsg descriptors to return EIO (or close
them too so they return EBADF?  but EIO seems safer).

(Also, assume forkall() doesn't exist and isn't a problem.  If fork()
is evil, there are no words to describe forkall().)

Nico
--

Other related posts: