On Mon, Dec 12, 2016 at 9:12 AM, Karan, Cem F CIV USARMY RDECOM ARL
(US) <cem.f.karan.civ@xxxxxxxx> wrote:
Hi Garrett, I like what you're trying to do! Two thoughts:
First, are you making changes to how new transports/protocols are added to
the library? I'd really like to be able to use what you're doing on top of
my simulator framework, but would need a stable, well-documented API that
operates on UDP-like packets rather than TCP streams to really use it.
Second, I'm going to make an argument for integers rather than pointers for
socket descriptors. I ran into some weird ABA-like problems
(https://en.wikipedia.org/wiki/ABA_problem) in some code that I was using at
one point that I only figured out after a month's worth of work. The problem
was that one thread created a struct, handed it off to a second thread, and
then deleted the struct, leaving the second thread with a dangling pointer.
However, before the second thread woke up, the first thread created another
struct on the heap of the same type, which, thanks to the magic of malloc()
on that particular machine, happened to be in exactly the same location as
the first struct. This only worked on my coworker's machine, where it never
crashed. On any other machine, it crashed.
To avoid similar problems in the library, I converted everything to use 32
bit IDs instead. The library owned all memory it allocated, and clients
could ask about particular information via the 32 bit IDs. The trick was the
library never reused an ID within the same process; it just kept incrementing
a counter to generate IDs. If a datastructure was deallocated, then its ID
was purged from the hashmap, and any requests for it would cause an immediate
crash. The only downside was that the library had to lookup information
within the hashmap each time, which caused a tiny slowdown, but it was small
enough that it wasn't noticeable in that particular application.
I'll admit that pointers are MUCH easier to use, but that was my experience
with sharing pointers across an API interface. Inside the library, we used
pointers everywhere, but it was easier to deal with any headaches from that
there.
Thanks, and I'm looking forwards to the rewrite; where are you putting the
code? I'd like to see it.
Thanks,
Cem Karan
-----Original Message-----
From: nanomsg-bounce@xxxxxxxxxxxxx [mailto:nanomsg-bounce@xxxxxxxxxxxxx] On ;
Behalf Of Garrett D'Amore
Sent: Saturday, December 10, 2016 10:46 AM
To: nanomsg@xxxxxxxxxxxxx
Subject: [Non-DoD Source] [nanomsg] Re: nanomsg rewrite - new API
All active links contained in this email were disabled. Please verify the
identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address to a
Web browser.
________________________________
Omid,
You are just going to have to trust that I know what I am doing here. The
model will involve threads performing synchronous operations
and will probably closely match what I put together in mangos.
Sent from my iPhone
On Dec 10, 2016, at 2:55 AM, omid shahraki <dreamstechgroup@xxxxxxxxx <
Caution-mailto:dreamstechgroup@xxxxxxxxx ;> > wrote:
Hi garrett,
I think this is a good opportunity to reconsider everything and better
shape nano library regardless of the design considerations at
the very beginning of the nano life.
I think concurrncy is one of the challenges which should be payed
close attention in its place throughout the project life cycle. As
you mentioned, you are going to utilize threads to obtain concurrency. Well,
thread is a very basic element, at a level, in concurrncy
construction. Actually it is one of the basic elements of Mendeleev Table in
software architecture, as a comparison. But, how it is going to
take place in such crucial role which could shine in different scenarios?
Without taking this in mind, there would be a risk of getting
confused about concurrency and preparing the scene for probable questions
which could be late to answer. That is why i'd like whiteboard
instead of keyboard at this level.
So, I think it is better to think of a concurrency model instead. This
way we can make it more mission critical which stacked on
experience and expertise and provides value to other layers.
Well, many things I would like to discuss about, but I think this
would be enough for the time being to keep it running.
Omid Shahraki
#############################
On Dec 10, 2016 10:56, "Garrett D'Amore" <garrett@xxxxxxxxxx <
Caution-mailto:garrett@xxxxxxxxxx ;> > wrote:
I have embarked on the creation of a complete rewrite of
nanomsg. This rewrite utilizes threads to obtain concurrency.
(In theory “green” threads or coroutines could be used as well, if the
underlying platform supports the necessary I/O primitives. I’m not
interested in discussion of these internal details at this point, as I
strongly believe that it is possible to make a very performant library that
uses threading for asynchronous I/O and to engage with greater levels of
concurrency.)
The reason for my message here is to present a revised API for
your consideration. This API is quite different in many
respects from nanomsg — in particular it presents a more “object oriented”
API that uses “natural” objects rather than attempting to
strictly follow POSIX file descriptor semantics. (So for example, sockets
are actual pointers to structures rather than integers, and you can
obtain information about the sockets using pointers to opaque structures
instead of trying to use the CMSG API. There is no attempt to
provide a “file descriptor” type API as a first class citizen, although for
those that need to use select() or poll() there will be a way to
obtain notification file descriptors. (Under the hood, those descriptors
will be lazy allocated, and not used at all unless someone demands
this. I believe this will have a dramatic performance benefit for the
library, greatly reducing the total number of system calls required to
send and receive messages.
The API will also open up some new richer abilities, like the
ability to introspect into pipes, and to obtain information
about where a message came from, or set different options for different
endpoints. Much of this is modeled on work I’ve done in mangos,
where I’ve been able to add functionality that folks have requested for
nanomsg, but which I cannot easily add to nanomsg.
Anyway, I’m opening this for discussion on the API. (Again,
let’s not get bogged down in *how* I’m building this — I’m
not interested in the various eventing frameworks or async I/O engines — I’m
doing this using threads.)
Oh, and just to be clear, the new code will still be written
in C (not C++), and retain the MIT license.
This rewrite will facilitate a number of changes with respect
to adding new protocols, and adding new transports. I look
forward to being able to add ZeroTier, UDP, TLS, and Websocket/HTTPS to this
implementation — something that is *extraordinarily*
difficult to do in the existing implementation.
Without further ado, here’s the header that defines the nng.h
API (nanomsgNG):
/*
* Copyright 2016 Garrett D'Amore <garrett@xxxxxxxxxx <
Caution-mailto:garrett@xxxxxxxxxx ;> >
*
* Permission is hereby granted, free of charge, to any person
obtaining a copy
* of this software and associated documentation files (the
"Software"),
* to deal in the Software without restriction, including
without limitation
* the rights to use, copy, modify, merge, publish,
distribute, sublicense,
* and/or sell copies of the Software, and to permit persons
to whom
* the Software is furnished to do so, subject to the
following conditions:
*
* The above copyright notice and this permission notice shall
be included
* in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO
EVENT SHALL
* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE
OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#ifndef NNGH_H
#define NNGH_H
/*
* NNG (nanomsg-ng) is a next generation implementation of the
SP protocols.
* The APIs have changed, and there is no attempt to provide
API compatibility
* with legacy libnanomsg. This file defines the library
consumer-facing
* Public API. Use of definitions or declarations not found in
this header file
* is specfically unsupported and strongly discouraged.
*/
#ifdef __cplusplus
extern "C" {
#endif
#include <errno.h>
#include <stddef.h>
#include <stdint.h>
/*
* NNG_DECL is used on declarations to deal with scope.
* For building Windows DLLs, it should be the appropriate
* __declspec(). (We recommend *not* building this library
* as a DLL, but instead linking it statically for your
projects
* to minimize questions about link dependencies later.)
*/
#ifndef NNG_DECL
#define NNG_DECL extern
#endif
/*
* A socket is the logical handle to the underlying topology.
Sockets
* are of a specific class (bus, req, etc.) and mode (raw or
cooked).
* This defines the behavior, and a given socket must only
have one such
* mode. This is analogous to a file descriptor.
*
* They may have any number of underlying physical connections
underneath.
*/
typedef struct nng_socket *nng_socket_t;
/*
* An endpoint (nn_endpt_t) is a the logical entity underneath
a socket,
* that handles creating individual connections. It can be
either a listener
* (does the bind() and accept() dance, creating many
connections), or a
* dialer (does connect(), and handles reconnection logic.)
*/
typedef struct nng_endpt *nng_endpt_t;
/*
* A pipe is logical communications channel used underneath
the socket.
* Usually (but not always!) this will be an actual
connection, e.g.
* a TCP connection.
*/
typedef struct nn_pipe *nng_pipe_t;
/*
* nng_socket simply creates a socket of the given class. It
returns an
* error code on failure, or zero on success. The socket
starts in cooked
* mode.
*/
NNG_DECL int nng_socket_create(nng_socket_t *, int proto);
/*
* nng_socket_close closes the socket, terminating all
activity and
* closing any underlying connections and releasing any
associated
* resources. Memory associated with the socket is freed, so
it is an
* error to reference the socket in any way after this is
called. Likewise,
* it is an error to reference any resources such as end
points associated
* with the socket.
*/
NNG_DECL int nng_socket_close(nng_socket_t);
/*
* nng_socket_setopt sets an option for a specific socket.
*/
NNG_DECL int nng_socket_setopt(nng_socket_t, int, void *,
size_t);
/*
* nng_socket_getopt obtains the option for a socket.
*/
NNG_DECL int nng_socket_getopt(nng_socket_t, int, void *,
size_t *);
/*
* nng_notify_register sets a notification callback. The
callback will be
* called for any of the requested events. The callback can
be deregistered
* by calling nng_notify_unregister with the same handle.
These notification
* callbacks are executed on a separate thread, to avoid
potential lock
* recursion.
*/
typedef struct nng_notify nng_notify_t;
typedef (void *nng_event_cb_t)(nng_socket_t, nng_event_t, void
*);
NNG_DECL nng_notify_t nng_notify_register(nng_socket_t, int,
nng_event_cb_t, void *);
NNG_DECL int nng_notify_unregister(nng_socket_t, nng_notify_t);
/*
* Event types. Sockets can have multiple different kind of
events.
* Note that these are edge triggered -- therefore the status
indicated
* may have changed since the notification occurred.
*
* NNG_EVENT_RECV - A message is ready for receive.
* NNG_EVENT_SEND - A message can be sent.
* NNG_EVENT_ERROR - An error condition on the socket occurred.
* NNG_EVENT_PIPE_ADD - A new pipe (connection) is added to
the socket.
* The argument is an nn_pipe_t.
* NNG_EVENT_PIPE_RM - A pipe (connection) is removed from the
socket.
* The argument is an nn_pipe_t.
* NNG_EVENT_ENDPT_ADD - An endpoint is added to the socket.
* The argument is an nn_endpt_t.
* NNG_EVENT_ENDPT_RM - An endpoint is removed from the socket.
* The argument is an nn_endpt_t.
*/
#define NNG_EVENT_BIT(x) (1U << (x))
#define NNG_EVENT_RECV NNG_EVENT_BIT(0)
#define NNG_EVENT_SEND NNG_EVENT_BIT(1)
#define NNG_EVENT_ERROR NNG_EVENT_BIT(2)
#define NNG_EVENT_PIPE_ADD NNG_EVENT_BIT(3)
#define NNG_EVENT_PIPE_RM NNG_EVENT_BIT(4)
#define NNG_EVENT_ENDPT_ADD NNG_EVENT_BIT(5)
#define NNG_EVENT_ENDPT_RM NNG_EVENT_BIT(6)
/*
* The following functions return more detailed information
about the event.
* Some of the values will not make sense for some event
types, in which case
* the value returned will be NULL.
*/
NNG_DECL int nng_event_type(nng_event_t);
NNG_DECL nng_socket_t nng_event_socket(nng_event_t);
NNG_DECL nng_endpt_t nng_event_endpt(nng_event_t);
NNG_DECL nng_pipe_t nng_event_pipe(nng_event_t);
NNG_DECL const char *nng_event_reason(nng_event_t);
/*
* nng_socket_listen creates a listening endpoint with no
special options,
* and starts it listening. It is functionally equivalent to
the legacy
* nn_bind(). The underlying endpoint is returned back to the
caller.
*/
NNG_DECL int nng_socket_listen(nng_endpt_t *, nng_socket_t,
const char *);
/*
* nng_socket_dial creates a dialing endpoint, with no special
options,
* and starts it dialing. Dialers have at most one active
connection at a
* time. This is similar to the legacy nn_connect(). The
underlying endpoint
* is returned back to the caller.
*/
NNG_DECL int nng_socket_dial(nng_endpt_t *, nng_socket_t,
const char *);
/*
* nng_socket_endpt creates an endpoint on the socket, but
does not
* start it either dialing or connecting.
*/
NNG_DECL int nng_socket_endpt(nng_endpt_t *, nng_socket_t,
const char *);
/*
* nng_endpt_dial starts the endpoint dialing. This is only
possible if
* the endpoint is not already dialing or listening.
*/
NNG_DECL int nng_endpt_dial(nng_endpt_t);
/*
* nng_endpt_listen starts the endpoint listening. This is
only possible if
* the endpoint is not already dialing or listening.
*/
NNG_DECL int nng_endpt_listen(nng_endpt_t);
/*
* nng_endpt_close closes the endpt, shutting down all
underlying
* connections and releasing all associated resources. It is
an error to
* refer to the endpoint after this is called.
*/
NNG_DECL int nng_endpt_close(nng_endpt_t);
/*
* nng_endpt_setopt sets an option for a specific endpoint.
Note
* endpoint options may not be altered on a running endpoint.
*/
NNG_DECL int nng_endpt_setopt(nng_endpt_t, int, void *,
size_t);
/*
* nng_endpt_getopt obtains the option for an endpoint.
*/
NNG_DECL int nng_endpt_getopt(nng_endpt_t, int, void *, size_t
*);
/*
* nng_strerror returns a human readable string associated
with the error
* code supplied.
*/
NNG_DECL const char *nng_strerror(int);
/*
* nng_send sends (or arranges to send) the data on the
socket. Note that
* this function may (will!) return before any receiver has
actually
* received the data. The return value will be zero to
indicate that the
* socket has accepted the entire data for send, or an errno
to indicate
* failure. The flags may include NNG_FLAG_NONBLOCK.
*/
NNG_DECL int nng_send(nng_socket_t, const void *, size_t, int);
/*
* nng_recv receives message data into the socket, up to the
supplied size.
* The actual size of the message data will be written to the
value pointed
* to by size. The flags may include NNG_FLAG_NONBLOCK and
NNG_FLAG_ALLOC.
* If NNG_FLAG_ALLOC is supplied then the library will
allocate memory for
* the caller. In that case the pointer to the allocated will
be stored
* instead of the data itself. The caller is responsible for
freeing the
* associated memory with free().
*/
NNG_DECL int nng_recv(nnt_socket_t, void *, size_t *, int);
/*
* nng_sendmsg is like nng_send, but offers up a message
structure, which
* gives the ability to provide more control over the message,
including
* providing backtrace information. It also can take a
message that was
* obtain via nn_recvmsg, allowing for zero copy forwarding.
*/
NNG_DECL int nng_sendmsg(nng_socket_t, nng_msg_t, int);
/*
* nng_recvmsg is like nng_recv, but is used to obtain a
message structure
* as well as the data buffer. This can be used to obtain
more information
* about where the message came from, access raw headers, etc.
It also
* can be passed off directly to nng_sendmsg.
*/
NNG_DECL int nng_recvmsg(nng_socket_t, nng_msg_t *, int);
/*
* Message API.
*/
NNG_DECL int nng_msg_alloc(nng_msg_t *, size_t);
NNG_DECL void nng_msg_free(nng_msg_t);
NNG_DECL const char *nng_msg_data(nng_msg_t);
NNG_DECL int nng_msg_realloc(nng_mst_t, size_t);
NNG_DECL void *nng_msg_header(nng_msg_t, size_t *);
NNG_DECL void *nng_msg_body(nng_msg_t, size_t *);
NNG_DECL int nng_msg_port(nng_msg_t, nng_pipe_t *);
/*
* Pipe API. Generally pipes are only "observable" to
applications, but
* we do permit an application to close a pipe. This can be
useful, for
* example during a connection notification, to disconnect a
pipe that
* is associated with an invalid or untrusted remote peer.
*/
NNG_DECL int nng_pipe_getopt(nng_pipe_t, int, void *, size_t
*);
NNG_DECL int nng_pipe_close(nng_pipe_t);
/*
* Protocol numbers. These are to be used with
nng_socket_create().
* These values are used on the wire, so must not be changed.
The major
* number of the protocol is shifted left by 4 bits, and a
subprotocol is
* assigned in the lower 4 bits.
*
* There are gaps in the list, which are obsolete or
unsupported protocols.
* For now we assume that protocol numbers are never more than
16 bits.
*/
#define NNG_PROTO(major, minor) (((major) * 16) + (minor))
#define NNG_PROTO_PAIR NNG_PROTO(1, 0)
#define NNG_PROTO_PUB NNG_PROTO(2, 0)
#define NNG_PROTO_SUB NNG_PROTO(2, 1)
#define NNG_PROTO_REQ NNG_PROTO(3, 0)
#define NNG_PROTO_REP NNG_PROTO(3, 1)
#define NNG_PROTO_PUSH NNG_PROTO(5, 0)
#define NNG_PROTO_PULL NNG_PROTO(5, 1)
#define NNG_PROTO_SURVEYOR NNG_PROTO(6, 2)
#define NNG_PROTO_RESPONDENT NNG_PROTO(6, 3)
#define NNG_PROTO_BUS NNG_PROTO(7, 0)
#define NNG_PROTO_STAR NNG_PROTO(100, 0)
/*
* Options. We encode option numbers as follows:
*
* <level> - 0: socket, 1: transport
* <type> - zero (socket), or transport (8 bits)
* <code> - specific value (16 bits)
*
*/
#define NNG_OPT_SOCKET(c) (c)
#define NNG_OPT_TRANSPORT(t, c) (0x10000 | ((p) << 16) | (c))
#define NNG_OPT_RAW NNG_OPT_SOCKET(0)
#define NNG_OPT_LINGER NNG_OPT_SOCKET(1)
#define NNG_OPT_RCVBUF NNG_OPT_SOCKET(2)
#define NNG_OPT_SNDBUF NNG_OPT_SOCKET(3)
#define NNG_OPT_RCVTIMEO NNG_OPT_SOCKET(4)
#define NNG_OPT_SNDTIMEO NNG_OPT_SOCKET(5)
#define NNG_OPT_RECONN_TIME NNG_OPT_SOCKET(6)
#define NNG_OPT_RECONN_MAXTIME NNG_OPT_SOCKET(7)
#define NNG_OPT_RCVMAXSZ NNG_OPT_SOCKET(8)
#define NNG_OPT_MAXTTL NNG_OPT_SOCKET(9)
#define NNG_OPT_PROTOCOL NNG_OPT_SOCKET(10)
#define NNG_OPT_SUBSCRIBE NNG_OPT_SOCKET(11)
#define NNG_OPT_UNSUBSCRIBE NNG_OPT_SOCKET(12)
#define NNG_OPT_SURVEYTIME NNG_OPT_SOCKET(13)
#define NNG_OPT_RESENDTIME NNG_OPT_SOCKET(14)
#define NNG_OPT_TRANSPORT NNG_OPT_SOCKET(15)
#define NNG_OPT_LOCALADDR NNG_OPT_SOCKET(16)
#define NNG_OPT_REMOTEADDR NNG_OPT_SOCKET(17)
#define NNG_OPT_RECVFD NNG_OPT_SOCKET(18)
#define NNG_OPT_SENDFD NNG_OPT_SOCKET(19)
/* XXX: TBD: priorities, socket names, ipv4only */
/*
* Statistics. These are for informational purposes only, and
subject
* to change without notice. The API for accessing these is
stable,
* but the individual statistic names, values, and meanings
are all
* subject to change.
*/
/*
* nng_snapshot_create creates a statistics snapshot. The
snapshot
* object must be deallocated expressly by the user, and may
persist beyond
* the lifetime of any socket object used to update it. Note
that the
* values of the statistics are initially unset.
*/
NNG_DECL int nng_snapshot_create(nng_snapshot_t *);
/*
* nng_snapshot_free frees a snapshot object. All statistic
objects
* contained therein are destroyed as well.
*/
NNG_DECL void nng_snapshot_free(nng_snapshot_t);
/*
* nng_snapshot_update updates a snapshot of all the statistics
* relevant to a particular socket. All prior values are
overwritten.
* It is acceptable to use the same snapshot object with
different
* sockets.
*/
NNG_DECL int nng_snapshot_update(nng_socket_t, nng_snpshot_t);
/*
* nng_snapshot_iterate is used to iterate over the individual
statistic
* objects inside the snapshot. Note that the statistic
object, and the
* meta-data for the object (name, type, units) is fixed, and
does not
* change for the entire life of the snapshot. Only the value
* is subject to change, and then only when a snapshot is
updated.
*
* Iteration begins by providing NULL in the value referenced.
Successive
* calls will update this value, returning NULL when no more
statistics
* are available in the snapshot.
*/
NNG_DECL nng_stat_t nng_snapshot_iterate(nng_snapshot_t,
nng_stat_t);
/*
* nng_stat_name is used to determine the name of the
statistic.
* This is a human readable name. Statistic names, as well as
the presence
* or absence or semantic of any particular statistic are not
part of any
* stable API, and may be changed without notice in future
updates.
*/
NNG_DECL const char *nng_stat_name(nng_stat_t);
/*
* nng_stat_type is used to determine the type of the
statistic.
* At present, only NNG_STAT_TYPE_LEVEL and and
NNG_STAT_TYPE_COUNTER
* are defined. Counters generally increment, and therefore
changes in the
* value over time are likely more interesting than the actual
level. Level
* values reflect some absolute state however, and should be
presented to the
* user as is.
*/
NNG_DECL int nng_stat_type(nng_stat_t);
#define NNG_STAT_LEVEL 0
#define NNG_STAT_COUNTER 1
/*
* nng_stat_unit provides information about the unit for the
statistic,
* such as NNG_UNIT_BYTES or NNG_UNIT_BYTES. If no specific
unit is
* applicable, such as a relative priority, then NN_UNIT_NONE
is
* returned.
*/
NNG_DECL int nng_stat_unit(nng_stat_t);
#define NNG_UNIT_NONE 0
#define NNG_UNIT_BYTES 1
#define NNG_UNIT_MESSAGES 2
#define NNG_UNIT_BOOLEAN 3
#define NNG_UNIT_MILLIS 4
#define NNG_UNIT_EVENTS 5
/*
* nng_stat_value returns returns the actual value of the
statistic.
* Statistic values reflect their value at the time that the
corresponding
* snapshot was updated, and are undefined until an update is
performed.
*/
NNG_DECL int64_t nng_stat_value(nng_stat_t);
/*
* Device functionality. This connects two sockets together
in a device,
* which means that messages from one side are forwarded to
the other.
*/
NNG_DECL int nng_device(nng_socket_t, nng_socket_t);
/*
* Pollset functionality. TBD. (Note that I'd rather avoid
this
* altogether, because I believe that the notification
mechanism I've
* created offers a superior way to handle this. I don't think
many
* direct consumers of nn_poll existed in the wild, except via
nn_device().
* I suspect that there not even many nn_device() consumers.)
*/
/*
* Symbol name and visibility. TBD. The only symbols that
really should
* be directly exported to runtimes IMO are the option
symbols. And frankly
* they have enough special logic around them that it might be
best not to
* automate the promotion of them to other APIs. This is an
area open
* for discussion.
*/
#ifdef __cplusplus
}
#endif
#endif /* NNG_H */
