Others have pointed you at the work in progress. Its still very very much
in its infancy.
You could have the the same problem with surprise reuse with integer file
descriptors, because they wind up using small values (array indices) as
well. That won’t magically help you. nng internally is threadsafe, so
that you won’t need to worry about this overly much. However, the
application is going to be responsible for ensuring that it does not
attempt to access a Socket after it closes it. That is, you won’t be able
to continue to do read() operations on a socket after you call close().
(Any concurrent outstanding operation will return NNG_ECLOSED in response
to the socket being closed.)
On Mon, Dec 12, 2016 at 6:12 AM, Karan, Cem F CIV USARMY RDECOM ARL (US) <
cem.f.karan.civ@xxxxxxxx> wrote:
Hi Garrett, I like what you're trying to do! Two thoughts:
First, are you making changes to how new transports/protocols are added to
the library? I'd really like to be able to use what you're doing on top of
my simulator framework, but would need a stable, well-documented API that
operates on UDP-like packets rather than TCP streams to really use it.
Second, I'm going to make an argument for integers rather than pointers
for socket descriptors. I ran into some weird ABA-like problems (
https://en.wikipedia.org/wiki/ABA_problem) in some code that I was using
at one point that I only figured out after a month's worth of work. The
problem was that one thread created a struct, handed it off to a second
thread, and then deleted the struct, leaving the second thread with a
dangling pointer. However, before the second thread woke up, the first
thread created another struct on the heap of the same type, which, thanks
to the magic of malloc() on that particular machine, happened to be in
exactly the same location as the first struct. This only worked on my
coworker's machine, where it never crashed. On any other machine, it
crashed.
To avoid similar problems in the library, I converted everything to use 32
bit IDs instead. The library owned all memory it allocated, and clients
could ask about particular information via the 32 bit IDs. The trick was
the library never reused an ID within the same process; it just kept
incrementing a counter to generate IDs. If a datastructure was
deallocated, then its ID was purged from the hashmap, and any requests for
it would cause an immediate crash. The only downside was that the library
had to lookup information within the hashmap each time, which caused a tiny
slowdown, but it was small enough that it wasn't noticeable in that
particular application.
I'll admit that pointers are MUCH easier to use, but that was my
experience with sharing pointers across an API interface. Inside the
library, we used pointers everywhere, but it was easier to deal with any
headaches from that there.
Thanks, and I'm looking forwards to the rewrite; where are you putting the
code? I'd like to see it.
Thanks,
Cem Karan
-----Original Message-----On Behalf Of Garrett D'Amore
From: nanomsg-bounce@xxxxxxxxxxxxx [mailto:nanomsg-bounce@xxxxxxxxxxxxx]
Sent: Saturday, December 10, 2016 10:46 AMthe identity of the sender, and confirm the authenticity of all links
To: nanomsg@xxxxxxxxxxxxx
Subject: [Non-DoD Source] [nanomsg] Re: nanomsg rewrite - new API
All active links contained in this email were disabled. Please verify
contained within the message prior to copying and pasting the address toa Web browser.
The model will involve threads performing synchronous operations
________________________________
Omid,
You are just going to have to trust that I know what I am doing here.
and will probably closely match what I put together in mangos.Caution-mailto:dreamstechgroup@xxxxxxxxx ;> > wrote:
Sent from my iPhone
On Dec 10, 2016, at 2:55 AM, omid shahraki <dreamstechgroup@xxxxxxxxx <
better shape nano library regardless of the design considerations at
Hi garrett,
I think this is a good opportunity to reconsider everything and
the very beginning of the nano life.close attention in its place throughout the project life cycle. As
I think concurrncy is one of the challenges which should be payed
you mentioned, you are going to utilize threads to obtain concurrency.Well, thread is a very basic element, at a level, in concurrncy
construction. Actually it is one of the basic elements of MendeleevTable in software architecture, as a comparison. But, how it is going to
take place in such crucial role which could shine in differentscenarios? Without taking this in mind, there would be a risk of getting
confused about concurrency and preparing the scene for probablequestions which could be late to answer. That is why i'd like whiteboard
instead of keyboard at this level.This way we can make it more mission critical which stacked on
So, I think it is better to think of a concurrency model instead.
experience and expertise and provides value to other layers.would be enough for the time being to keep it running.
Well, many things I would like to discuss about, but I think this
Caution-mailto:garrett@xxxxxxxxxx ;> > wrote:
Omid Shahraki
#############################
On Dec 10, 2016 10:56, "Garrett D'Amore" <garrett@xxxxxxxxxx <
nanomsg. This rewrite utilizes threads to obtain concurrency.
I have embarked on the creation of a complete rewrite of
(In theory “green” threads or coroutines could be used as well, if theunderlying platform supports the necessary I/O primitives. I’m not
interested in discussion of these internal details at this point, as Istrongly believe that it is possible to make a very performant library that
uses threading for asynchronous I/O and to engage with greater levels ofconcurrency.)
for your consideration. This API is quite different in many
The reason for my message here is to present a revised API
respects from nanomsg — in particular it presents a more “objectoriented” API that uses “natural” objects rather than attempting to
strictly follow POSIX file descriptor semantics. (So for example,sockets are actual pointers to structures rather than integers, and you can
obtain information about the sockets using pointers to opaque structuresinstead of trying to use the CMSG API. There is no attempt to
provide a “file descriptor” type API as a first class citizen, althoughfor those that need to use select() or poll() there will be a way to
obtain notification file descriptors. (Under the hood, thosedescriptors will be lazy allocated, and not used at all unless someone
demands
this. I believe this will have a dramatic performance benefit for thelibrary, greatly reducing the total number of system calls required to
send and receive messages.the ability to introspect into pipes, and to obtain information
The API will also open up some new richer abilities, like
about where a message came from, or set different options for differentendpoints. Much of this is modeled on work I’ve done in mangos,
where I’ve been able to add functionality that folks have requested fornanomsg, but which I cannot easily add to nanomsg.
(Again, let’s not get bogged down in *how* I’m building this — I’m
Anyway, I’m opening this for discussion on the API.
not interested in the various eventing frameworks or async I/O engines —I’m doing this using threads.)
written in C (not C++), and retain the MIT license.
Oh, and just to be clear, the new code will still be
respect to adding new protocols, and adding new transports. I look
This rewrite will facilitate a number of changes with
forward to being able to add ZeroTier, UDP, TLS, and Websocket/HTTPS tothis implementation — something that is *extraordinarily*
difficult to do in the existing implementation.nng.h API (nanomsgNG):
Without further ado, here’s the header that defines the
Caution-mailto:garrett@xxxxxxxxxx ;> >
/*
* Copyright 2016 Garrett D'Amore <garrett@xxxxxxxxxx <
*person obtaining a copy
* Permission is hereby granted, free of charge, to any
* of this software and associated documentation files(the "Software"),
* to deal in the Software without restriction, includingwithout limitation
* the rights to use, copy, modify, merge, publish,distribute, sublicense,
* and/or sell copies of the Software, and to permitpersons to whom
* the Software is furnished to do so, subject to thefollowing conditions:
*shall be included
* The above copyright notice and this permission notice
* in all copies or substantial portions of the Software.ANY KIND, EXPRESS OR
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OFMERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL
* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANYCLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OROTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THEUSE OR OTHER DEALINGS
* IN THE SOFTWARE.the SP protocols.
*/
#ifndef NNGH_H
#define NNGH_H
/*
* NNG (nanomsg-ng) is a next generation implementation of
* The APIs have changed, and there is no attempt toprovide API compatibility
* with legacy libnanomsg. This file defines the libraryconsumer-facing
* Public API. Use of definitions or declarations notfound in this header file
* is specfically unsupported and strongly discouraged.projects
*/
#ifdef __cplusplus
extern "C" {
#endif
#include <errno.h>
#include <stddef.h>
#include <stdint.h>
/*
* NNG_DECL is used on declarations to deal with scope.
* For building Windows DLLs, it should be the appropriate
* __declspec(). (We recommend *not* building this library
* as a DLL, but instead linking it statically for your
* to minimize questions about link dependencies later.)topology. Sockets
*/
#ifndef NNG_DECL
#define NNG_DECL extern
#endif
/*
* A socket is the logical handle to the underlying
* are of a specific class (bus, req, etc.) and mode (rawor cooked).
* This defines the behavior, and a given socket must onlyhave one such
* mode. This is analogous to a file descriptor.connections underneath.
*
* They may have any number of underlying physical
*/underneath a socket,
typedef struct nng_socket *nng_socket_t;
/*
* An endpoint (nn_endpt_t) is a the logical entity
* that handles creating individual connections. It canbe either a listener
* (does the bind() and accept() dance, creating manyconnections), or a
* dialer (does connect(), and handles reconnection logic.)underneath the socket.
*/
typedef struct nng_endpt *nng_endpt_t;
/*
* A pipe is logical communications channel used
* Usually (but not always!) this will be an actualconnection, e.g.
* a TCP connection.It returns an
*/
typedef struct nn_pipe *nng_pipe_t;
/*
* nng_socket simply creates a socket of the given class.
* error code on failure, or zero on success. The socketstarts in cooked
* mode.activity and
*/
NNG_DECL int nng_socket_create(nng_socket_t *, int proto);
/*
* nng_socket_close closes the socket, terminating all
* closing any underlying connections and releasing anyassociated
* resources. Memory associated with the socket is freed,so it is an
* error to reference the socket in any way after this iscalled. Likewise,
* it is an error to reference any resources such as endpoints associated
* with the socket.size_t);
*/
NNG_DECL int nng_socket_close(nng_socket_t);
/*
* nng_socket_setopt sets an option for a specific socket.
*/
NNG_DECL int nng_socket_setopt(nng_socket_t, int, void *,
size_t *);
/*
* nng_socket_getopt obtains the option for a socket.
*/
NNG_DECL int nng_socket_getopt(nng_socket_t, int, void *,
callback will be
/*
* nng_notify_register sets a notification callback. The
* called for any of the requested events. The callbackcan be deregistered
* by calling nng_notify_unregister with the same handle.These notification
* callbacks are executed on a separate thread, to avoidpotential lock
* recursion.void *);
*/
typedef struct nng_notify nng_notify_t;
typedef (void *nng_event_cb_t)(nng_socket_t, nng_event_t,
NNG_DECL nng_notify_t nng_notify_register(nng_socket_t,int,
nng_event_cb_t, void *);nng_notify_t);
NNG_DECL int nng_notify_unregister(nng_socket_t,
of events.
/*
* Event types. Sockets can have multiple different kind
* Note that these are edge triggered -- therefore thestatus indicated
* may have changed since the notification occurred.occurred.
*
* NNG_EVENT_RECV - A message is ready for receive.
* NNG_EVENT_SEND - A message can be sent.
* NNG_EVENT_ERROR - An error condition on the socket
* NNG_EVENT_PIPE_ADD - A new pipe (connection) is addedto the socket.
* The argument is an nn_pipe_t.the socket.
* NNG_EVENT_PIPE_RM - A pipe (connection) is removed from
* The argument is an nn_pipe_t.socket.
* NNG_EVENT_ENDPT_ADD - An endpoint is added to the
* The argument is an nn_endpt_t.socket.
* NNG_EVENT_ENDPT_RM - An endpoint is removed from the
* The argument is an nn_endpt_t.information about the event.
*/
#define NNG_EVENT_BIT(x) (1U << (x))
#define NNG_EVENT_RECV NNG_EVENT_BIT(0)
#define NNG_EVENT_SEND NNG_EVENT_BIT(1)
#define NNG_EVENT_ERROR NNG_EVENT_BIT(2)
#define NNG_EVENT_PIPE_ADD NNG_EVENT_BIT(3)
#define NNG_EVENT_PIPE_RM NNG_EVENT_BIT(4)
#define NNG_EVENT_ENDPT_ADD NNG_EVENT_BIT(5)
#define NNG_EVENT_ENDPT_RM NNG_EVENT_BIT(6)
/*
* The following functions return more detailed
* Some of the values will not make sense for some eventtypes, in which case
* the value returned will be NULL.special options,
*/
NNG_DECL int nng_event_type(nng_event_t);
NNG_DECL nng_socket_t nng_event_socket(nng_event_t);
NNG_DECL nng_endpt_t nng_event_endpt(nng_event_t);
NNG_DECL nng_pipe_t nng_event_pipe(nng_event_t);
NNG_DECL const char *nng_event_reason(nng_event_t);
/*
* nng_socket_listen creates a listening endpoint with no
* and starts it listening. It is functionally equivalentto the legacy
* nn_bind(). The underlying endpoint is returned back tothe caller.
*/nng_socket_t, const char *);
NNG_DECL int nng_socket_listen(nng_endpt_t *,
special options,
/*
* nng_socket_dial creates a dialing endpoint, with no
* and starts it dialing. Dialers have at most one activeconnection at a
* time. This is similar to the legacy nn_connect(). Theunderlying endpoint
* is returned back to the caller.const char *);
*/
NNG_DECL int nng_socket_dial(nng_endpt_t *, nng_socket_t,
does not
/*
* nng_socket_endpt creates an endpoint on the socket, but
* start it either dialing or connecting.const char *);
*/
NNG_DECL int nng_socket_endpt(nng_endpt_t *, nng_socket_t,
only possible if
/*
* nng_endpt_dial starts the endpoint dialing. This is
* the endpoint is not already dialing or listening.is only possible if
*/
NNG_DECL int nng_endpt_dial(nng_endpt_t);
/*
* nng_endpt_listen starts the endpoint listening. This
* the endpoint is not already dialing or listening.underlying
*/
NNG_DECL int nng_endpt_listen(nng_endpt_t);
/*
* nng_endpt_close closes the endpt, shutting down all
* connections and releasing all associated resources. Itis an error to
* refer to the endpoint after this is called.endpoint. Note
*/
NNG_DECL int nng_endpt_close(nng_endpt_t);
/*
* nng_endpt_setopt sets an option for a specific
* endpoint options may not be altered on a runningendpoint.
*/size_t);
NNG_DECL int nng_endpt_setopt(nng_endpt_t, int, void *,
size_t *);
/*
* nng_endpt_getopt obtains the option for an endpoint.
*/
NNG_DECL int nng_endpt_getopt(nng_endpt_t, int, void *,
with the error
/*
* nng_strerror returns a human readable string associated
* code supplied.socket. Note that
*/
NNG_DECL const char *nng_strerror(int);
/*
* nng_send sends (or arranges to send) the data on the
* this function may (will!) return before any receiverhas actually
* received the data. The return value will be zero toindicate that the
* socket has accepted the entire data for send, or anerrno to indicate
* failure. The flags may include NNG_FLAG_NONBLOCK.int);
*/
NNG_DECL int nng_send(nng_socket_t, const void *, size_t,
the supplied size.
/*
* nng_recv receives message data into the socket, up to
* The actual size of the message data will be written tothe value pointed
* to by size. The flags may include NNG_FLAG_NONBLOCKand NNG_FLAG_ALLOC.
* If NNG_FLAG_ALLOC is supplied then the library willallocate memory for
* the caller. In that case the pointer to the allocatedwill be stored
* instead of the data itself. The caller is responsiblefor freeing the
* associated memory with free().structure, which
*/
NNG_DECL int nng_recv(nnt_socket_t, void *, size_t *, int);
/*
* nng_sendmsg is like nng_send, but offers up a message
* gives the ability to provide more control over themessage, including
* providing backtrace information. It also can take amessage that was
* obtain via nn_recvmsg, allowing for zero copyforwarding.
*/message structure
NNG_DECL int nng_sendmsg(nng_socket_t, nng_msg_t, int);
/*
* nng_recvmsg is like nng_recv, but is used to obtain a
* as well as the data buffer. This can be used to obtainmore information
* about where the message came from, access raw headers,etc. It also
* can be passed off directly to nng_sendmsg.applications, but
*/
NNG_DECL int nng_recvmsg(nng_socket_t, nng_msg_t *, int);
/*
* Message API.
*/
NNG_DECL int nng_msg_alloc(nng_msg_t *, size_t);
NNG_DECL void nng_msg_free(nng_msg_t);
NNG_DECL const char *nng_msg_data(nng_msg_t);
NNG_DECL int nng_msg_realloc(nng_mst_t, size_t);
NNG_DECL void *nng_msg_header(nng_msg_t, size_t *);
NNG_DECL void *nng_msg_body(nng_msg_t, size_t *);
NNG_DECL int nng_msg_port(nng_msg_t, nng_pipe_t *);
/*
* Pipe API. Generally pipes are only "observable" to
* we do permit an application to close a pipe. This canbe useful, for
* example during a connection notification, to disconnecta pipe that
* is associated with an invalid or untrusted remote peer.size_t *);
*/
NNG_DECL int nng_pipe_getopt(nng_pipe_t, int, void *,
NNG_DECL int nng_pipe_close(nng_pipe_t);nng_socket_create().
/*
* Protocol numbers. These are to be used with
* These values are used on the wire, so must not bechanged. The major
* number of the protocol is shifted left by 4 bits, and asubprotocol is
* assigned in the lower 4 bits.unsupported protocols.
*
* There are gaps in the list, which are obsolete or
* For now we assume that protocol numbers are never morethan 16 bits.
*/(c))
#define NNG_PROTO(major, minor) (((major) * 16) + (minor))
#define NNG_PROTO_PAIR NNG_PROTO(1, 0)
#define NNG_PROTO_PUB NNG_PROTO(2, 0)
#define NNG_PROTO_SUB NNG_PROTO(2, 1)
#define NNG_PROTO_REQ NNG_PROTO(3, 0)
#define NNG_PROTO_REP NNG_PROTO(3, 1)
#define NNG_PROTO_PUSH NNG_PROTO(5, 0)
#define NNG_PROTO_PULL NNG_PROTO(5, 1)
#define NNG_PROTO_SURVEYOR NNG_PROTO(6, 2)
#define NNG_PROTO_RESPONDENT NNG_PROTO(6, 3)
#define NNG_PROTO_BUS NNG_PROTO(7, 0)
#define NNG_PROTO_STAR NNG_PROTO(100, 0)
/*
* Options. We encode option numbers as follows:
*
* <level> - 0: socket, 1: transport
* <type> - zero (socket), or transport (8 bits)
* <code> - specific value (16 bits)
*
*/
#define NNG_OPT_SOCKET(c) (c)
#define NNG_OPT_TRANSPORT(t, c) (0x10000 | ((p) << 16) |
and subject
#define NNG_OPT_RAW NNG_OPT_SOCKET(0)
#define NNG_OPT_LINGER NNG_OPT_SOCKET(1)
#define NNG_OPT_RCVBUF NNG_OPT_SOCKET(2)
#define NNG_OPT_SNDBUF NNG_OPT_SOCKET(3)
#define NNG_OPT_RCVTIMEO NNG_OPT_SOCKET(4)
#define NNG_OPT_SNDTIMEO NNG_OPT_SOCKET(5)
#define NNG_OPT_RECONN_TIME NNG_OPT_SOCKET(6)
#define NNG_OPT_RECONN_MAXTIME NNG_OPT_SOCKET(7)
#define NNG_OPT_RCVMAXSZ NNG_OPT_SOCKET(8)
#define NNG_OPT_MAXTTL NNG_OPT_SOCKET(9)
#define NNG_OPT_PROTOCOL NNG_OPT_SOCKET(10)
#define NNG_OPT_SUBSCRIBE NNG_OPT_SOCKET(11)
#define NNG_OPT_UNSUBSCRIBE NNG_OPT_SOCKET(12)
#define NNG_OPT_SURVEYTIME NNG_OPT_SOCKET(13)
#define NNG_OPT_RESENDTIME NNG_OPT_SOCKET(14)
#define NNG_OPT_TRANSPORT NNG_OPT_SOCKET(15)
#define NNG_OPT_LOCALADDR NNG_OPT_SOCKET(16)
#define NNG_OPT_REMOTEADDR NNG_OPT_SOCKET(17)
#define NNG_OPT_RECVFD NNG_OPT_SOCKET(18)
#define NNG_OPT_SENDFD NNG_OPT_SOCKET(19)
/* XXX: TBD: priorities, socket names, ipv4only */
/*
* Statistics. These are for informational purposes only,
* to change without notice. The API for accessing theseis stable,
* but the individual statistic names, values, andmeanings are all
* subject to change.snapshot
*/
/*
* nng_snapshot_create creates a statistics snapshot. The
* object must be deallocated expressly by the user, andmay persist beyond
* the lifetime of any socket object used to update it.Note that the
* values of the statistics are initially unset.statistic objects
*/
NNG_DECL int nng_snapshot_create(nng_snapshot_t *);
/*
* nng_snapshot_free frees a snapshot object. All
* contained therein are destroyed as well.statistics
*/
NNG_DECL void nng_snapshot_free(nng_snapshot_t);
/*
* nng_snapshot_update updates a snapshot of all the
* relevant to a particular socket. All prior values areoverwritten.
* It is acceptable to use the same snapshot object withdifferent
* sockets.nng_snpshot_t);
*/
NNG_DECL int nng_snapshot_update(nng_socket_t,
individual statistic
/*
* nng_snapshot_iterate is used to iterate over the
* objects inside the snapshot. Note that the statisticobject, and the
* meta-data for the object (name, type, units) is fixed,and does not
* change for the entire life of the snapshot. Only thevalue
* is subject to change, and then only when a snapshot isupdated.
*referenced. Successive
* Iteration begins by providing NULL in the value
* calls will update this value, returning NULL when nomore statistics
* are available in the snapshot.nng_stat_t);
*/
NNG_DECL nng_stat_t nng_snapshot_iterate(nng_snapshot_t,
statistic.
/*
* nng_stat_name is used to determine the name of the
* This is a human readable name. Statistic names, aswell as the presence
* or absence or semantic of any particular statistic arenot part of any
* stable API, and may be changed without notice in futureupdates.
*/statistic.
NNG_DECL const char *nng_stat_name(nng_stat_t);
/*
* nng_stat_type is used to determine the type of the
* At present, only NNG_STAT_TYPE_LEVEL and andNNG_STAT_TYPE_COUNTER
* are defined. Counters generally increment, andtherefore changes in the
* value over time are likely more interesting than theactual level. Level
* values reflect some absolute state however, and shouldbe presented to the
* user as is.the statistic,
*/
NNG_DECL int nng_stat_type(nng_stat_t);
#define NNG_STAT_LEVEL 0
#define NNG_STAT_COUNTER 1
/*
* nng_stat_unit provides information about the unit for
* such as NNG_UNIT_BYTES or NNG_UNIT_BYTES. If nospecific unit is
* applicable, such as a relative priority, thenNN_UNIT_NONE is
* returned.statistic.
*/
NNG_DECL int nng_stat_unit(nng_stat_t);
#define NNG_UNIT_NONE 0
#define NNG_UNIT_BYTES 1
#define NNG_UNIT_MESSAGES 2
#define NNG_UNIT_BOOLEAN 3
#define NNG_UNIT_MILLIS 4
#define NNG_UNIT_EVENTS 5
/*
* nng_stat_value returns returns the actual value of the
* Statistic values reflect their value at the time thatthe corresponding
* snapshot was updated, and are undefined until an updateis performed.
*/together in a device,
NNG_DECL int64_t nng_stat_value(nng_stat_t);
/*
* Device functionality. This connects two sockets
* which means that messages from one side are forwardedto the other.
*/avoid this
NNG_DECL int nng_device(nng_socket_t, nng_socket_t);
/*
* Pollset functionality. TBD. (Note that I'd rather
* altogether, because I believe that the notificationmechanism I've
* created offers a superior way to handle this. I don'tthink many
* direct consumers of nn_poll existed in the wild, exceptvia nn_device().
* I suspect that there not even many nn_device()consumers.)
*/that really should
/*
* Symbol name and visibility. TBD. The only symbols
* be directly exported to runtimes IMO are the optionsymbols. And frankly
* they have enough special logic around them that itmight be best not to
* automate the promotion of them to other APIs. This isan area open
* for discussion.
*/
#ifdef __cplusplus
}
#endif
#endif /* NNG_H */