On Mon, Dec 12, 2016 at 12:05 PM, Garrett D'Amore <garrett@xxxxxxxxxx> wrote:
Yes. there is a platform porting layer, and one of the functions that the
“platform” implements is “nni_thread_create()”. This creates a thread, or
coroutine, or whatever. Not every platform has pthreads.
I actually intend to build a version of this that uses C11 threads and
synchronization objects at some point.
On Mon, Dec 12, 2016 at 8:55 AM, Karan, Cem F CIV USARMY RDECOM ARL (US)
<cem.f.karan.civ@xxxxxxxx> wrote:
Are you going to create your own thread/process API? That is, something
where all your code calls through a small set of well-known POSIX
thread-like functions that can be easily replaced by something else? That
will make it much easier to test out green threads vs. regular threads vs.
full processes.
Thanks,
Cem Karan
-----Original Message-----
From: nanomsg-bounce@xxxxxxxxxxxxx [mailto:nanomsg-bounce@xxxxxxxxxxxxx]
On Behalf Of Garrett D'Amore
Sent: Monday, December 12, 2016 11:19 AM
To: nanomsg@xxxxxxxxxxxxx
Subject: [Non-DoD Source] [nanomsg] Re: nanomsg rewrite - new API
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address to
a Web browser.
________________________________
Yes, I plan to use a single (or even two) threads per connection
endpoint. Plus one per socket at large. This allows me to use simple
blocking system calls, without having to worry about the vagaries of
async I/O.
Modern operating systems (or rather modern thread implementations) can
generally scale to huge numbers of threads. Recall that a
thread is really just a scheduling context with some stack space. To
get to 100k+ threads you’re going to need adequate RAM to hold the
stacks (probably several GB in this case — e.g. 100k 8k stacks is just
under a GB). It’s not hard to build meaningful systems that have vast
numbers of threads, especially when they don’t contend on the same data
(this is very important to scalability). You’re still going to run
out of ephemeral TCP ports before you hit this number.
I don’t necessarily believe the current hype to turn away from threads
and make everything an event loop is “correct” — the problem is
that most of the threaded code that people look at (like say Apache)
uses really heavy weight processes (where context switching requires
changing MMU tables which is really really expensive), or contend on
shared resources (which is the situation where many threads
perform worse than one), or where the threading framework is
inefficient. (This used to the situation long ago. These days thread
libraries are *very* efficient.)
Now, all that said, the code is being built with a very open design to
support porting to different platforms, and will be able to accept
solutions that make use of coroutines or green threads. The only thing
that is *required* is that the implementation provide high level
blocking calls for I/O, and thread creation, and a few synchronization
primitives. (mutexes and condition variables).
- Garrett
On Mon, Dec 12, 2016 at 6:33 AM, Jan Bramkamp <crest@xxxxxxxxx <
Caution-mailto:crest@xxxxxxxxx ;> > wrote:
On 10/12/2016 08:25, Garrett D'Amore wrote:
I have embarked on the creation of a complete rewrite of
nanomsg. This
rewrite utilizes threads to obtain concurrency. (In
theory “green”
threads or coroutines could be used as well, if the
underlying platform
supports the necessary I/O primitives. I’m not interested
in discussion
of these internal details at this point, as I strongly
believe that it
is possible to make a very performant library that uses
threading for
asynchronous I/O and to engage with greater levels of
concurrency.)
In some ways using kernel threads simplifies the concurrency as
you have proper threads that can block on everything and aren't
restricted to operations which are available as non-blocking. Targeting
more than one platform increases the the burden even further
because each platform (Linux, *BSD, OSX, Windows) offers a different set
of non-blocking operations e.g. FreeBSD offers working POSIX
AIO on local file systems, but Linux AIO isn't worth the effort for most
applications.
The reason for my message here is to present a revised API
for your
consideration. This API is quite different in many
respects from
nanomsg — in particular it presents a more “object
oriented” API that
uses “natural” objects rather than attempting to strictly
follow POSIX
file descriptor semantics. (So for example, sockets are
actual pointers
to structures rather than integers, and you can obtain
information about
the sockets using pointers to opaque structures instead of
trying to use
the CMSG API. There is no attempt to provide a “file
descriptor” type
API as a first class citizen, although for those that need
to use
select() or poll() there will be a way to obtain
notification file
descriptors. (Under the hood, those descriptors will be
lazy allocated,
and not used at all unless someone demands this. I
believe this will
have a dramatic performance benefit for the library,
greatly reducing
the total number of system calls required to send and
receive messages.
The API will also open up some new richer abilities, like
the ability to
introspect into pipes, and to obtain information about
where a message
came from, or set different options for different
endpoints. Much of
this is modeled on work I’ve done in mangos, where I’ve
been able to add
functionality that folks have requested for nanomsg, but
which I cannot
easily add to nanomsg.
Anyway, I’m opening this for discussion on the API.
(Again, let’s not
get bogged down in *how* I’m building this — I’m not
interested in the
various eventing frameworks or async I/O engines — I’m
doing this using
threads.)
Do you really plan to use one thread per endpoint? This would be a
real scalability problem. Do know any up to date
documentation how far which kernel can scale in this regard? I doubt
Linux or FreeBSD can handle 100k threads as well as they can handle
just 100k sockets.
Oh, and just to be clear, the new code will still be
written in C (not
C++), and retain the MIT license.
Pleased to hear that. C is a lot more suitable this because its
ABI is the lowest common denominator in *nix systems and the MIT
license should allow anyone to integrate nanomsg into their application.
This rewrite will facilitate a number of changes with
respect to adding
new protocols, and adding new transports. I look forward
to being able
to add ZeroTier, UDP, TLS, and Websocket/HTTPS to this
implementation —
something that is *extraordinarily* difficult to do in the
existing
implementation.
Utilizing threads also simplifies reusing existing libraries in
nanomsg transports.
Without further ado, here’s the header that defines the
nng.h API
(nanomsgNG):
[truncated]
The API looks clean and easy to use without constraining
applications. Will the objects be thread-safe or does the application have
to provide its own locking around the API like in ZeroMQ? The ZeroMQ API
without locking makes it very hard to provide sane bindings to
some languages.
