Oh, I convinced myself that an array is better. What’s even cooler, is
that using an array, I can use a binary search to get filtering operations
back to O(log(n)). Changing subscriptions will still be somewhat
expensive, but I think that won’t be too bad since I can reduce it to the
cost of memcpy(), where the size being copied is basically a pointer and a
size (at most 16 bytes total typically) per subscription. This means if
you have 1000 subscriptions I might have to memcpy 16k when you change the
subscription. But most of the time we’ll be looking at shuffling around a
couple of hundred bytes at most. (Allocations are still an issue, but I
think I can keep that down to minimum — preallocating 16 slots or so to
start, and doubling each we find we need more.)
On Fri, Jan 6, 2017 at 10:19 AM, Karan, Cem F CIV USARMY RDECOM ARL (US) <
cem.f.karan.civ@xxxxxxxx> wrote:
I agree with what you're saying, but I was thinking of somewhat simpler
testing; whether or not an array/list/Patricia tree are better in the real
world. The higher-order stuff is, as you point out, really, really
complicated to test.
Thanks,
Cem Karan
-----Original Message-----On Behalf Of Garrett D'Amore
From: nanomsg-bounce@xxxxxxxxxxxxx [mailto:nanomsg-bounce@xxxxxxxxxxxxx]
Sent: Friday, January 06, 2017 11:30 AMthe identity of the sender, and confirm the authenticity of all links
To: nanomsg@xxxxxxxxxxxxx
Subject: [nanomsg] Re: [Non-DoD Source] status update libnng
All active links contained in this email were disabled. Please verify
contained within the message prior to copying and pasting the address toa Web browser.
generically.
________________________________
Performance statistics are hard to gather somewhat, and harder to use
really you want to modify the wire protocol to carry timestamps.
The difficulty of gathering is that you need to measure entire latency;
Some performance data we can report, such as queue depth, etc.wildly different that what might be perfectly normal for one user
The difficulty of use is that everyone’s system and applications are so
might be tragically bad for another.bug analysis, I can recall precisely one user over he years griping
In actuality, there has been very little demand for performance-related
about inproc performance vs. ZMQ. My response at the time was “I don’tcare”. (Because inproc performance was never terribly
important to me since I don’t think it sees much real world use, and Ialso had bigger fish to fry around trying to fix the MT-safety things in
inproc. libnng is me basically giving up on that. I couldn’t figure outhow to make libnanomsg’s inproc completely thread-safe without
completely trashing the performance. The FSM architecture made passingdata across the different sockets (which had different lock
hierarchies) too damned difficult.cem.f.karan.civ@xxxxxxxx < Caution-
On Fri, Jan 6, 2017 at 7:00 AM, Karan, Cem F CIV USARMY RDECOM ARL (US) <
mailto:cem.f.karan.civ@xxxxxxxx ;> > wrote:gathering code; have you considered adding code that gathers
You were mentioning that you were planning on adding statistics
performance statistics as well? As end users we could send you bugreports that include those statistics, which would let you know if it’s a
problem or not.nanomsg-bounce@xxxxxxxxxxxxx > [Caution-mailto:nanomsg-
Thanks,
Cem Karan
> -----Original Message-----
> From: nanomsg-bounce@xxxxxxxxxxxxx < Caution-mailto:
bounce@xxxxxxxxxxxxx < Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> ]On Behalf Of Garrett D'Amore
> Sent: Thursday, January 05, 2017 1:37 PMverify the identity of the sender, and confirm the authenticity of all
> To: nanomsg@xxxxxxxxxxxxx < Caution-mailto:nanomsg@xxxxxxxxxxxxx
> Subject: [nanomsg] Re: [Non-DoD Source] status update libnng
>
> All active links contained in this email were disabled. Please
linksaddress to a Web browser.
> contained within the message prior to copying and pasting the
>hundred of these things, I’ll switch to an array.
>
> ________________________________
>
>
>
> I could certainly make it an array. :-)
>
> If it seems to be universal that nobody needs more than a few
>about the O(n) vs O(log(n)) thing though. If we know that O(n) is
> I wasn’t so much worried about list traversal times as I was
goodsuperior. One thing about an array, is that if I keep it sorted, then
> enough, then certainly an array is probably going to be
subscribeshuffle on average half the items in the array. I’m assuming
> operations will become more expensive, since I’ll have to
subscriptionlike to know if anyone has an application where changes to
> changes occur less frequently than messages arrive. (I’d really
subscriptions —< Caution-mailto:danielcccc@xxxxxxxxx ;> < Caution-
> subscribe/unsubscribe — is a performance sensitive operation.)
>
> - Garrett
>
>
> On Thu, Jan 5, 2017 at 10:23 AM, Daniel C <danielcccc@xxxxxxxxx
Caution-mailto:danielcccc@xxxxxxxxx ;< Caution-mailto:danielcccc@gmail.com > > > wrote:
>less than 10 or 20 in my case.
>
> I agree with Jason; an array would be faster. N definitely
>j.e.aten@xxxxxxxxx < Caution-mailto:j.e.aten@xxxxxxxxx ;> < Caution-
>
> Dan
>
>
> On Thu, Jan 5, 2017 at 7:58 AM, Jason E. Aten <
Caution-mailto:j.e.aten@xxxxxxxxx ;< Caution-mailto:j.e.aten@xxxxxxxxxmuch faster than a linked list.
> > wrote:
>
>
> For a small N < 10, I would expect an array to be
>garrett@xxxxxxxxxx < Caution-mailto:garrett@xxxxxxxxxx ;> <
> My typical use is for small N, about 3.
>
> On Thu, Jan 5, 2017 at 4:52 AM, Garrett D'Amore <
Caution-Caution-mailto:garrett@xxxxxxxxxx ;< Caution-mailto:garrett@xxxxxxxxxx > > > wrote:
>is such a non-portable nightmare. I almost went down the path of
>
> TCP works now. OMG, close() vs. accept()
> pthread cancellation. Thankfully, that crisis is now averted.I never realized just how broken the use of UNIX file descriptors in
the face ofThank goodness for shutdown(2) and dup2().
> close() and multiple threads really are. But its working now .
>and I’ve started PUB/SUB. Sub is actually done, untested.
> There’s a test suite for TCP and inproc,
>remaining protocols and IPC before the weekend.
> I give 50/50 odds on me finishing up the
>writing the eventing framework (necessary for file descriptor based
> What won’t happen before the weekend is
> notifications for example), or statistics framework. AndWindows will probably happen next week. I’ve designed this such that I
think Iused a simple sorted linked list for subscriptions. My instinct is that
> will be able to crank out Windows stuff super fast.
>
> One question about PUB/SUB. I’ve for now
> for the vast vast majority of folks, this is not onlysufficient, but probably superior to a patricia tree; I think most of us
only
maintaingiven time. That said, I’m keen to hear of actual uses of nanomsg
> around a dozen or so subscriptions on any subscriber at any
where thehundred active subscriptions on a given end node. (Note that
> patricia tree makes a real difference — cases with over several
this is onlythousands — each having dozens of subscriptions — with zero
> on the the end node; a server can service many many clients —
negativefiltering messages is now an O(n) instead of O(log(n)) operation (where n is
> impact in my design.)
>
> To be clear, the linked list means that
> the number of active subscriptions); I suspect that for smallvalues of n, the linked list approach I’ve taken is faster. I have
implementedpatricia tree if there is need. (I didn’t yank Martin’s old code
> this way for now for expediency, but I’m open to pulling in a
because I wantthe new code base — I haven’t had time to be certain of that yet.
> to make sure I thoroughly understand it before I bring it into
>D'Amore <garrett@xxxxxxxxxx < Caution-mailto:garrett@xxxxxxxxxx ;>
>
> On Wed, Jan 4, 2017 at 2:18 AM, Garrett
< Caution-Caution-mailto:garrett@xxxxxxxxxx ;< Caution-mailto:garrett@xxxxxxxxxx > >
> > wrote:swag at TCP. Turns out it was bit more code than anticipated, as I want to
>
>
> I’ve just committed the initial
> support TCPv4 and TCPv6, and properly support differentplatforms that might have different ways to resolve names, or handle
low levelwinsock code later.
> TCP details. This should make it really fast to write the
>details so that transports that want to build on top of TCP (e.g. websocket
> I’ve also tried to abstract the
> or TLS) can do so easily.at the last commit if you want to see what I’m up to. The main missing
>
>
> Totally untested, but you can look
> thing (besides testing that it actually works) is support forsome of the richer TCP options — e.g. KEEPALIVEs etc. I am disabling
Nagle bywanting Nagle turned on. (And I’ve taken care to use writev to
> default, because I think 99% of nanomsg users have no business
avoidis something that sadly isn’t possible using Go….)
> splitting writes up across separate packets when possible. This
>Karan, Cem F CIV USARMY RDECOM ARL (US)
> On Tue, Jan 3, 2017 at 10:54 AM,
> <cem.f.karan.civ@xxxxxxxx < Caution-mailto:cem.f.karan.civ@xxxxxxxx > < Caution-Caution-mailto:cem.f.karan.civ@xxxxxxxx ;<
Caution-mailto:cem.f.karan.civ@xxxxxxxx ;> > > wrote:and forgot that inproc is a thing; for some reason I thought you were
>
>
> Got it, my brain glitched
> building out your own TCP stack, which would have made me...concerned. ;)
>Message-----
> Thanks,
> Cem Karan
>
> > -----Original
> > From:nanomsg-bounce@xxxxxxxxxxxxx < Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx
< Caution-nanomsg-bounce@xxxxxxxxxxxxx > >
Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;< Caution-mailto:
> [Caution-Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;<Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> < Caution-Caution-
mailto:nanomsg-bounce@xxxxxxxxxxxxx ;< Caution-mailto:nanomsg-bounce@freelists.org > > ] On Behalf Of Garrett D'Amore
> > Sent: Tuesday, January03, 2017 1:23 PM
nanomsg@xxxxxxxxxxxxx < Caution-mailto:nanomsg@xxxxxxxxxxxxx ;> <
> > To:
Caution-Caution-
mailto:nanomsg@xxxxxxxxxxxxx ;< Caution-mailto:nanomsg@xxxxxxxxxxxxx ;> >[Non-DoD Source] status update libnng
> > Subject: [nanomsg] Re:
> >contained in this email were disabled. Please verify the identity of the
> > All active links
sender,
> and confirm the authenticity of all linksmessage prior to copying and pasting the address to a Web browser.
> > contained within the
> >________________________________
> >
> >
> >having TCP working. Right now the TCP transport is not written,
> >
> >
> > I’m about 24 hours from
> though I’ve written a fair bit of it locally.working now, as is PAIR, and inproc is working quit well and totally
> >
> > The REQ/REP pattern is
> thread safe.algorithms go — I’m a big believer in mutexes. I need mutexes and condition
> >
> > As far as lockless
> variables to make things work well, and inare used sensibly (with minimal contention), they do not significantly
> > my experience if they
> impact performance.data structures are superior, but you have to have use cases that work
> >
> > For some things lockless
> for them; with mutexes and condvars Iabout this; they Just Work.
> > don’t have to think
> >at alternative designs for that in the future, but I’m only going to do
> > It may be useful to look
> that once I’ve demonstrated a clear need orsomeone else will contribute at that point). Until then, I need to focus on
> > benefit (or maybe
> getting the code working with a sensibleprematurely optimizing things.
> > design and avoid
> >6:45 AM, Karan, Cem F CIV USARMY RDECOM ARL (US)
> > - Garrett
> >
> >
> > On Tue, Jan 3, 2017 at
civ@xxxxxxxx > < Caution-Caution-mailto:cem.f.karan.civ@xxxxxxxx ;<
> <cem.f.karan.civ@xxxxxxxx < Caution-mailto:cem.f.karan.
Caution-mailto:cem.f.karan.civ@xxxxxxxx ;> > < Caution-cem.f.karan.civ@xxxxxxxx < Caution-mailto:cem.f.karan.civ@xxxxxxxx ;> <
> > Caution-Caution-mailto:
Caution-Caution-mailto:cem.f.karan.civ@xxxxxxxx ;< Caution-mailto:cem.f.karan.civ@xxxxxxxx > > > >
> wrote:Message-----
> >
> >
> > > -----Original
> > > From:nanomsg-bounce@xxxxxxxxxxxxx < Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx
<nanomsg-bounce@xxxxxxxxxxxxx > >
Caution-Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;< Caution-mailto:
> < Caution-Caution-Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;<Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> < Caution-
Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;< Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx > > > [Caution-Caution-
> Caution-mailto:nanomsg- ;< Caution-mailto:nanomsg- ;> <Caution-Caution-mailto:nanomsg- ;< Caution-mailto:nanomsg- ;> >
> > bounce@xxxxxxxxxxxxx <Caution-mailto:bounce@xxxxxxxxxxxxx ;> < Caution-Caution-
mailto:bounce@xxxxxxxxxxxxx ;< Caution-mailto:bounce@xxxxxxxxxxxxx ;> >< Caution-Caution-
> Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;< Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx > < Caution-Caution-
mailto:nanomsg-bounce@xxxxxxxxxxxxx ;< Caution-mailto:nanomsg-bounce@freelists.org > > > ] On Behalf Of Garrett D'Amore
> > > Sent: Monday,December 26, 2016 2:46 AM
>nanomsg@xxxxxxxxxxxxx < Caution-mailto:nanomsg@xxxxxxxxxxxxx ;> <
> > > To:
Caution-Caution-
mailto:nanomsg@xxxxxxxxxxxxx ;< Caution-mailto:nanomsg@xxxxxxxxxxxxx ;>freelists.org > < Caution-Caution-
< Caution-Caution-
> Caution-mailto:nanomsg@xxxxxxxxxxxxx ;< Caution-mailto:nanomsg@
mailto:nanomsg@xxxxxxxxxxxxx ;< Caution-mailto:nanomsg@xxxxxxxxxxxxx ;>[Non-DoD Source] [nanomsg] status update libnng
>
> > > Subject:
> > >Christmas (western) update…
> > > Just a brief
> > >managing connections. There’s a lot more to this than you’d think — and a
> > > libnng is now
> lot of details that I’ve written underby New Years I’ll have it in a functional state for TCP for POSIX systems,
> > the hood.
> > >
> > > I expect that
for
> at least the primary patterns /a couple of hours from that point.
> > protocols. TBH,
> > > I’m really only
> > >current code base is pretty much crash-immune with respect to things that
> > > Note that the
> caused grief in libnanomsg — e.g.impact libnng. The whole approach has been from a “correct first”, so I
> > ENOMEM
> > > errors will not
> expect we will have fewer problems onceusing this.
> > we
> > > actually start
> > >moving apace… quite quickly actually.
> > > Things are
> > >inclined to review or give feedback on teh existing code, I’m now at the
> > > If anyone is so
> point to receive it, understanding thatswathes still unimplemented. You can’t use it for TCP for example. But at
> > there are
> > > still large
this
> point I’d rather solicit feedback earlytell me what’s missing, but if you see problems with what’s already there,
> > rather than
> > > late. Don’t
> please *do* let me know.ski trip for the next several days, but will probably work in the evenings
> > >
> > > I’m going on a
> getting the thing to a state where otherwith for actual experimentation.
> > folks can
> > > start playing
> > >looking forward to benchmarking this thing. I think we’re gonna blow the
> > > I’m really
doors
> off libnanomsg, at least for platformspthreads. :-)
> > that have
> > > non-crappy
> > >everyone! :-)
> > > Merry Christmas
> >your hard work!
> > Thank you for all
> >that you can't use it for TCP, what exactly do you mean?
> > BTW, when you say
> >
> > Thanks,
> > Cem Karan
> >
> >
> >
>
>
>
>
>
>
>
