In a surge of creativity and inspiration, I started work last week on a
distributed execution platform that I like to call remap. I'm writing about
this on this mailing list, because I'm pretty excited about how much work
nanomsg took away from me... Matters that take days to test and solve are
now 3-4 lines of code.
http://gtoonstra.github.io/remap/
Remap started as a map/reduce clone, but it's been generalized such that
worker processes can run arbitrary algorithms. Well, you need modules, I/O
classes and apps for that and so far "map" and "reduce" are implemented and
working with more modules on the way.
So what I'm using from nanomsg right now are mostly the pub/sub protocols.
There's a general broker that is used for most of the message passing. At
the moment I'm working on an implementation of a worker module for graph
processing (a la Google Pregel). This requires a 'master' process that
directs all worker processes to execute another step in the algorithm
(called a master step), which is probably going to be done with a survey
protocol.
If it wasn't for nanomsg, I'd probably be still in "message limbo" trying
to figure out which socket failed to communicate or where my task context
went...
So... thanks!
Regards,
Gerard Toonstra
