I agree with what you're saying, but I was thinking of somewhat simpler
testing; whether or not an array/list/Patricia tree are better in the real
world. The higher-order stuff is, as you point out, really, really complicated
to test.
Thanks,
Cem Karan
-----Original Message-----
From: nanomsg-bounce@xxxxxxxxxxxxx [mailto:nanomsg-bounce@xxxxxxxxxxxxx] On ;
Behalf Of Garrett D'Amore
Sent: Friday, January 06, 2017 11:30 AM
To: nanomsg@xxxxxxxxxxxxx
Subject: [nanomsg] Re: [Non-DoD Source] status update libnng
All active links contained in this email were disabled. Please verify the
identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address to a
Web browser.
________________________________
Performance statistics are hard to gather somewhat, and harder to use
generically.
The difficulty of gathering is that you need to measure entire latency;
really you want to modify the wire protocol to carry timestamps.
Some performance data we can report, such as queue depth, etc.
The difficulty of use is that everyone’s system and applications are so
wildly different that what might be perfectly normal for one user
might be tragically bad for another.
In actuality, there has been very little demand for performance-related bug
analysis, I can recall precisely one user over he years griping
about inproc performance vs. ZMQ. My response at the time was “I don’t
care”. (Because inproc performance was never terribly
important to me since I don’t think it sees much real world use, and I also
had bigger fish to fry around trying to fix the MT-safety things in
inproc. libnng is me basically giving up on that. I couldn’t figure out how
to make libnanomsg’s inproc completely thread-safe without
completely trashing the performance. The FSM architecture made passing data
across the different sockets (which had different lock
hierarchies) too damned difficult.
On Fri, Jan 6, 2017 at 7:00 AM, Karan, Cem F CIV USARMY RDECOM ARL (US)
<cem.f.karan.civ@xxxxxxxx < Caution-
mailto:cem.f.karan.civ@xxxxxxxx ;> > wrote:
You were mentioning that you were planning on adding statistics
gathering code; have you considered adding code that gathers
performance statistics as well? As end users we could send you bug reports
that include those statistics, which would let you know if it’s a
problem or not.
Thanks,
Cem Karan
> -----Original Message-----
> From: nanomsg-bounce@xxxxxxxxxxxxx <
Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> [Caution-mailto:nanomsg-
bounce@xxxxxxxxxxxxx < Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> ] On
Behalf Of Garrett D'Amore
> Sent: Thursday, January 05, 2017 1:37 PM
> To: nanomsg@xxxxxxxxxxxxx < Caution-mailto:nanomsg@xxxxxxxxxxxxx ;>
> Subject: [nanomsg] Re: [Non-DoD Source] status update libnng
>
> All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the address
to a Web browser.
>
>
> ________________________________
>
>
>
> I could certainly make it an array. :-)
>
> If it seems to be universal that nobody needs more than a few hundred
of these things, I’ll switch to an array.
>
> I wasn’t so much worried about list traversal times as I was about
the O(n) vs O(log(n)) thing though. If we know that O(n) is
good
> enough, then certainly an array is probably going to be superior.
One thing about an array, is that if I keep it sorted, then
subscribe
> operations will become more expensive, since I’ll have to shuffle on
average half the items in the array. I’m assuming
subscription
> changes occur less frequently than messages arrive. (I’d really like
to know if anyone has an application where changes to
subscriptions —
> subscribe/unsubscribe — is a performance sensitive operation.)
>
> - Garrett
>
>
> On Thu, Jan 5, 2017 at 10:23 AM, Daniel C <danielcccc@xxxxxxxxx <
Caution-mailto:danielcccc@xxxxxxxxx ;> < Caution-
Caution-mailto:danielcccc@xxxxxxxxx ;< Caution-mailto:danielcccc@xxxxxxxxx ;>
>wrote:
>
> I agree with Jason; an array would be faster. N definitely less
than 10 or 20 in my case.
>
>
> Dan
>
>
> On Thu, Jan 5, 2017 at 7:58 AM, Jason E. Aten
<j.e.aten@xxxxxxxxx < Caution-mailto:j.e.aten@xxxxxxxxx ;> < Caution-
Caution-mailto:j.e.aten@xxxxxxxxx ;< Caution-mailto:j.e.aten@xxxxxxxxx ;> > >
wrote:
>
>
> For a small N < 10, I would expect an array to be much
faster than a linked list.
>
> My typical use is for small N, about 3.
>
> On Thu, Jan 5, 2017 at 4:52 AM, Garrett D'Amore
<garrett@xxxxxxxxxx < Caution-mailto:garrett@xxxxxxxxxx ;> <
Caution-Caution-mailto:garrett@xxxxxxxxxx ;< Caution-mailto:garrett@xxxxxxxxxx ;
> > wrote:>
>
> TCP works now. OMG, close() vs. accept() is
such a non-portable nightmare. I almost went down the path of
> pthread cancellation. Thankfully, that crisis is now averted. I
never realized just how broken the use of UNIX file descriptors in
the face of
> close() and multiple threads really are. But its working now . Thank
goodness for shutdown(2) and dup2().
>
> There’s a test suite for TCP and inproc, and
I’ve started PUB/SUB. Sub is actually done, untested.
>
> I give 50/50 odds on me finishing up the
remaining protocols and IPC before the weekend.
>
> What won’t happen before the weekend is writing
the eventing framework (necessary for file descriptor based
> notifications for example), or statistics framework. And Windows
will probably happen next week. I’ve designed this such that I
think I
> will be able to crank out Windows stuff super fast.
>
> One question about PUB/SUB. I’ve for now used
a simple sorted linked list for subscriptions. My instinct is that
> for the vast vast majority of folks, this is not only sufficient, but
probably superior to a patricia tree; I think most of us only
maintain
> around a dozen or so subscriptions on any subscriber at any given
time. That said, I’m keen to hear of actual uses of nanomsg
where the
> patricia tree makes a real difference — cases with over several
hundred active subscriptions on a given end node. (Note that
this is only
> on the the end node; a server can service many many clients —
thousands — each having dozens of subscriptions — with zero
negative
> impact in my design.)
>
> To be clear, the linked list means that
filtering messages is now an O(n) instead of O(log(n)) operation (where n is
> the number of active subscriptions); I suspect that for small values
of n, the linked list approach I’ve taken is faster. I have
implemented
> this way for now for expediency, but I’m open to pulling in a
patricia tree if there is need. (I didn’t yank Martin’s old code
because I want
> to make sure I thoroughly understand it before I bring it into the
new code base — I haven’t had time to be certain of that yet.
>
>
> On Wed, Jan 4, 2017 at 2:18 AM, Garrett D'Amore
<garrett@xxxxxxxxxx < Caution-mailto:garrett@xxxxxxxxxx ;>
< Caution-Caution-mailto:garrett@xxxxxxxxxx ;<
Caution-mailto:garrett@xxxxxxxxxx ;> >
> > wrote:
>
>
> I’ve just committed the initial swag at
TCP. Turns out it was bit more code than anticipated, as I want to
> support TCPv4 and TCPv6, and properly support different platforms
that might have different ways to resolve names, or handle
low level
> TCP details. This should make it really fast to write the winsock
code later.
>
> I’ve also tried to abstract the details
so that transports that want to build on top of TCP (e.g. websocket
> or TLS) can do so easily.
>
>
> Totally untested, but you can look at
the last commit if you want to see what I’m up to. The main missing
> thing (besides testing that it actually works) is support for some of
the richer TCP options — e.g. KEEPALIVEs etc. I am disabling
Nagle by
> default, because I think 99% of nanomsg users have no business
wanting Nagle turned on. (And I’ve taken care to use writev to
avoid
> splitting writes up across separate packets when possible. This is
something that sadly isn’t possible using Go….)
>
> On Tue, Jan 3, 2017 at 10:54 AM, Karan,
Cem F CIV USARMY RDECOM ARL (US)
> <cem.f.karan.civ@xxxxxxxx < Caution-mailto:cem.f.karan.civ@xxxxxxxx ;>
< Caution-Caution-mailto:cem.f.karan.civ@xxxxxxxx ;<
Caution-mailto:cem.f.karan.civ@xxxxxxxx ;> > > wrote:
>
>
> Got it, my brain glitched and
forgot that inproc is a thing; for some reason I thought you were
> building out your own TCP stack, which would have made me...
concerned. ;)
>
> Thanks,
> Cem Karan
>
> > -----Original Message-----
> > From:
nanomsg-bounce@xxxxxxxxxxxxx < Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;>
< Caution-
Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;<
Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> >
> [Caution-Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;<
Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> < Caution-Caution-
mailto:nanomsg-bounce@xxxxxxxxxxxxx ;<
Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> > ] On Behalf Of Garrett
D'Amore
> > Sent: Tuesday, January 03,
2017 1:23 PM
> > To: nanomsg@xxxxxxxxxxxxx <
Caution-mailto:nanomsg@xxxxxxxxxxxxx ;> < Caution-Caution-
mailto:nanomsg@xxxxxxxxxxxxx ;< Caution-mailto:nanomsg@xxxxxxxxxxxxx ;> >
> > Subject: [nanomsg] Re:
[Non-DoD Source] status update libnng
> >
> > All active links contained in
this email were disabled. Please verify the identity of the sender,
> and confirm the authenticity of all links
> > contained within the message
prior to copying and pasting the address to a Web browser.
> >
> >
> >
________________________________
> >
> >
> >
> > I’m about 24 hours from
having TCP working. Right now the TCP transport is not written,
> though I’ve written a fair bit of it locally.
> >
> > The REQ/REP pattern is
working now, as is PAIR, and inproc is working quit well and totally
> thread safe.
> >
> > As far as lockless algorithms
go — I’m a big believer in mutexes. I need mutexes and condition
> variables to make things work well, and in
> > my experience if they are
used sensibly (with minimal contention), they do not significantly
> impact performance.
> >
> > For some things lockless data
structures are superior, but you have to have use cases that work
> for them; with mutexes and condvars I
> > don’t have to think about
this; they Just Work.
> >
> > It may be useful to look at
alternative designs for that in the future, but I’m only going to do
> that once I’ve demonstrated a clear need or
> > benefit (or maybe someone
else will contribute at that point). Until then, I need to focus on
> getting the code working with a sensible
> > design and avoid prematurely
optimizing things.
> >
> > - Garrett
> >
> >
> > On Tue, Jan 3, 2017 at 6:45
AM, Karan, Cem F CIV USARMY RDECOM ARL (US)
> <cem.f.karan.civ@xxxxxxxx < Caution-mailto:cem.f.karan.civ@xxxxxxxx ;>
< Caution-Caution-mailto:cem.f.karan.civ@xxxxxxxx ;<
Caution-mailto:cem.f.karan.civ@xxxxxxxx ;> > < Caution-
> >
Caution-Caution-mailto:cem.f.karan.civ@xxxxxxxx ;<
Caution-mailto:cem.f.karan.civ@xxxxxxxx ;> <
Caution-Caution-mailto:cem.f.karan.civ@xxxxxxxx ;<
Caution-mailto:cem.f.karan.civ@xxxxxxxx ;> > > >
> wrote:
> >
> >
> > > -----Original
Message-----
> > > From:
nanomsg-bounce@xxxxxxxxxxxxx < Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;>
<
Caution-Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;<
Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> >
> < Caution-Caution-Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;<
Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> < Caution-
Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;<
Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> > > [Caution-Caution-
> Caution-mailto:nanomsg- ;< Caution-mailto:nanomsg- ;> <
Caution-Caution-mailto:nanomsg- ;< Caution-mailto:nanomsg- ;> >
> > bounce@xxxxxxxxxxxxx <
Caution-mailto:bounce@xxxxxxxxxxxxx ;> < Caution-Caution-
mailto:bounce@xxxxxxxxxxxxx ;< Caution-mailto:bounce@xxxxxxxxxxxxx ;> > <
Caution-Caution-
> Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;<
Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> < Caution-Caution-
mailto:nanomsg-bounce@xxxxxxxxxxxxx ;<
Caution-mailto:nanomsg-bounce@xxxxxxxxxxxxx ;> > > ] On Behalf Of Garrett
D'Amore
> > > Sent: Monday,
December 26, 2016 2:46 AM
>
> > > To:
nanomsg@xxxxxxxxxxxxx < Caution-mailto:nanomsg@xxxxxxxxxxxxx ;> <
Caution-Caution-
mailto:nanomsg@xxxxxxxxxxxxx ;< Caution-mailto:nanomsg@xxxxxxxxxxxxx ;> > <
Caution-Caution-
> Caution-mailto:nanomsg@xxxxxxxxxxxxx ;<
Caution-mailto:nanomsg@xxxxxxxxxxxxx ;> < Caution-Caution-
mailto:nanomsg@xxxxxxxxxxxxx ;< Caution-mailto:nanomsg@xxxxxxxxxxxxx ;> > >
> > > Subject: [Non-DoD
Source] [nanomsg] status update libnng
> > >
> > > Just a brief
Christmas (western) update…
> > >
> > > libnng is now
managing connections. There’s a lot more to this than you’d think — and a
> lot of details that I’ve written under
> > the hood.
> > >
> > > I expect that by New
Years I’ll have it in a functional state for TCP for POSIX systems, for
> at least the primary patterns /
> > protocols. TBH,
> > > I’m really only a
couple of hours from that point.
> > >
> > > Note that the current
code base is pretty much crash-immune with respect to things that
> caused grief in libnanomsg — e.g.
> > ENOMEM
> > > errors will not
impact libnng. The whole approach has been from a “correct first”, so I
> expect we will have fewer problems once
> > we
> > > actually start using
this.
> > >
> > > Things are moving
apace… quite quickly actually.
> > >
> > > If anyone is so
inclined to review or give feedback on teh existing code, I’m now at the
> point to receive it, understanding that
> > there are
> > > still large swathes
still unimplemented. You can’t use it for TCP for example. But at this
> point I’d rather solicit feedback early
> > rather than
> > > late. Don’t tell me
what’s missing, but if you see problems with what’s already there,
> please *do* let me know.
> > >
> > > I’m going on a ski
trip for the next several days, but will probably work in the evenings
> getting the thing to a state where other
> > folks can
> > > start playing with
for actual experimentation.
> > >
> > > I’m really looking
forward to benchmarking this thing. I think we’re gonna blow the doors
> off libnanomsg, at least for platforms
> > that have
> > > non-crappy pthreads.
:-)
> > >
> > > Merry Christmas
everyone! :-)
> >
> > Thank you for all your
hard work!
> >
> > BTW, when you say that
you can't use it for TCP, what exactly do you mean?
> >
> > Thanks,
> > Cem Karan
> >
> >
> >
>
>
>
>
>
>
>
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
