Hi Alrik,
thanks for your reply.
The GUI module you are talking about, would essentially be the GUI to
your own save logic. This is because the GUI should have no
functionality by itself and therefore you will need an additional logic
module anyways, that defines your "required" additional parameters and
handles them. This could for example be done as an Interfuse onto the
existing save logic.
Nevertheless, any parameters you define will be quite specific to your
experiment, so the logic will only be valid for you. E.g. you might work
with single diamond sample and single centers while others might work
with different materials or diamond batches and look for regions defined
by position and size, just to take one example. Therefore I would not
define a general parameter framework, that everybody has to use, because
qudi is quite flexible in its use.
Your analysis notebooks we could put into the qudi code e.g. in the
folder notebooks. Your code would then need to be able to at least
analyse data created by the default config generating dummy data. This
way other people can test it and then adapt it for their own purposes.
Also someone (Dan maybe) should please review the code and make sure
that it is understandable, has a good structure and works for them. I
would leave it at a notebook level and not include the analysis scripts
into qudi itself, as this is again very specific to the user.
These are just my thoughts which I discussed a bit with Jan and Niko,
but maybe there are better ideas out there.
Cheers,
Kay
On 04.05.20 19:55, Alrik Durand wrote:
Hi,
We have been using Qudi on most of our setup in our lab in Montpellier (France) for about a year.
We are using additional parameters a lot via notebook scripts, and it works well for us. One problem that I have seen with it comes from changes in the names of the parameters over time. Maybe the typical fields using by people (ex: 'sample') could be fixed by a new GUI that would interact with savelogic.
I personally like that fact Qudi use a centralized saving scheme, it helps when you are trying to explore data from someone else or share codes for analysis.
Concerning the analysis, we also developed some basic tools to load data files in Pandas Dataframes, and some other tools to do some very general things with the pandas objects. Maybe such code could be shared as part of the Qudi project some way, I know for example we have a method that look at the parameters before parsing the files, something nice in the "Load all" strategy.
In the end, all this does not prevent the use of a notebook. We are experimenting with digital notebook in markdown. This has a few advantages and is quite good looking with the editor Typora.
I would love feedback on the two mentioned idea, as we have some free time lately...
Best,
Alrik Durand
PhD student @L2C Montpellier
Le lun. 4 mai 2020 à 19:20, Dr. Kay Jahnke <kay.jahnke@xxxxxxxxxxxxxxxxx <mailto:kay.jahnke@xxxxxxxxxxxxxxxxx>> a écrit :
Hi Dan,
you are correct, the current save logic only saves hierarchical
for dates and then the modules the saved data was created it.
It was designed that way because qudi is not dictating what
modules there are, which parameters they have or which and how the
data is saved. So going by date was the best thing to get any
structure.
There are some ways, that can enhance this structure and give you
a way to access your data more conveniently: The current
save_logic has additional_parameters
(https://github.com/Ulm-IQO/qudi/blob/master/logic/save_logic.py#L633).
These are global and can be set from any module or even from a
notebook script. So here you could save sample, center or any
other parameters you like, into the data files. You should best
not save parameters in file names, as this can lead to massive
problems afterwards.
Some of my colleagues then query the whole data directory
automatically with a script and load ALL the data files. These
data files can then be pushed into for example a "pandas" data set
and this let's you query the specific parameters per experiment.
The disadvantage is that the querying of all the data takes a
while and you might run out of RAM, as the data might get big
(just imagine you want to look at all the data from one PhD). Also
in principle this has nothing really to do with qudi, because at
this point you are writing you analysis scripts and should not
need the qudi core functionality.
Therefore the much more elegant way of solving your problem would
be to write your own save-logic. The current module is just the
default qudi logic module and can be replaced at any time. You
will just need to support the same functions, but you can freely
change what happens in the background.
For your case it would probably be best if you set up a (elastic)
database and write a new save-logic that connects to that database
and safes the data in a clever way. A word of caution: Be sure to
put a lot of though into the design of the database beforehand and
define explicitly what you want to save and how (e.g. which
parameters, which modules, pictures or only data sets,
dimensionality of the data).
I tried to write a save-logic for a database connection once, but
the users could not agree on any standardized parameters and
structure. So the database became maximally flexible and therefore
was extremely complicated to query. It never got used
productively. Therefore, define what you want beforehand, put
thought in it and then keep to your structure.
And finally, if something good comes out, other people might also
want to use something similar. So please than open a Pull Request
and commit back to Upstream. This also has the advantage that
other people might fix bugs for you or enhance the project.
An additional general remark: It is always very useful to keep
some kind of notebook, even if the data is saved automatically. We
had very good experience by going fully digital on the lab
notebooks and using a wiki system for that
(https://www.dokuwiki.org/dokuwiki). The most used plug-in it
turns out is than the one that let's you paste screenshots
directly into the wiki page.
If there are more questions, please don't hesitate to ask.
Cheers,
Kay
Am 04.05.2020 um 17:59 schrieb Dan Yudilevich:
Hi everyone,
We are a new group out of the Weizmann Institute (Israel), and we
are slowly but surely getting to know qudi, with most of the
important features running smoothly. The software is impressive,
so kudos to the developers.
One thing I am struggling with is data management. Although we
are only beginning to acquire data, I already feel it is getting
quite cluttered. The apparent organization hierarchy of
date/modules makes it challenging to trace specific data later
on. I would like, for example, a convenient way to find data
related to a specific sample (or defect); data from specific
types of pulse sequences, etc.
By managing a lab notebook I can, of course, refer to the
specific dates and files, but I feel it somewhat defeats the
purpose.
So, I wanted to ask if someone has any recommendation –
Does anyone have a particularly elegant way of organizing the
information?
Am I missing something in the save logic, so that I’m
under-utilizing this feature?
Thank you all, and stay healthy,
Dan Yudilevich
Finkler Group | Dept. of Chemical and Biological Physics
Weizmann Institute of Science
-- Dr. Kay Daniel Jahnke
Küfergasse 1
89073 Ulm
[T] +49 176 444 346 51
[@]kay.jahnke@xxxxxxxxxxxxxxxxx <mailto:kay.jahnke@xxxxxxxxxxxxxxxxx>
