While working on a generalized backend interface that would let Zoe abstract
away the details of which backend is used to manage the containers I hit again
the problem of how to manage the logs produced by containers.
We discussed this already several times and at Eurecom we tried various
solutions (kibana, graylog, Zoe custom implementation), without finding a good
one. The problem is exacerbated by having backends with different APIs that
could give or give not access to container standard output and standard error.
Moreover we have services, like spark-workers, that produce a (mostly useless)
standard output and save a much more useful log in some location inside the
container (that currently is lost forever).
My idea is simple: let’s shift the burden of deciding what is important to log
and save to the ZApp creator.
Zoe will mount a volume inside each container with a well-known path. It is up
to the creator of the ZApp to decide what to save in this volume during the
lifetime of the container. He could just redirect the standard output or
configure the service to save some log files.
This volume is part of a hierarchy that is exposed by Zoe’s web interfaces (and
APIs), so the user can access the log data.
I have added an initial description of this idea in the wiki, for task 5.2.3.
Distributed Systems Group – Eurecom