[mirtoolbox] Re: Out of memory when running a flowchart

  • From: Olivier Lartillot <olartillot@xxxxxxxxx>
  • To: mirtoolbox@xxxxxxxxxxxxx
  • Date: Mon, 28 Sep 2009 20:22:39 +0300

Hi,

I spent some time working on the new mechanisms for complex flowchart evaluation, as I mentioned in my previous mail. I have now a version of the toolbox (called 1.2.2.1) that solves a part of Ewald's problem, but another part still needs to be considered. I planned to work on that soon, because memory management is one of the core topic I will develop in my poster for the "Late-Breaking Demo" Session at ISMIR 2009.

So in the new version 1.2.2.1, as I mentioned in my previous mail, the chunk decomposition is performed for each tmp variable separately, and the dependent features are computed for each successive chunk, in order to avoid storing the whole tmp variable in the memory.

Now in Ewald's flowchart, the ideal would be to have also the possibility of computing a chain of dependent variables that would all be stored in the output of the evaluation, without recomputing the chain for each of those results. This is what I plan to work these next days.

Regards,

Olivier

Olivier Lartillot kirjoitti 23.9.2009 kello 9.51:

In my previous mail, I said:

> Basically, each output of your complex flowchart should be based on at > most one 'tmp' variable, and this 'tmp' variable should correspond to a > variable that, once evaluated, will have its chunks recombined entirely.

Actually, I have just reworked the internal architecture of the toolbox, and in my current version of this experimental attempt, 'tmp' variable can be computed in each chunk separately.

Does that mean I can only store one tmp variable at a time?

No, you can have several tmp variables.

Unfortunately my flow chart is indeed more complex, see Code below. Note that quite a few features depend on the spectrogram which I therefore would like to store in the tmp field. Can this flow chart also be "reduced" to take advantage of the internal memory management?

Maybe I can do something along the lines of

myflow.tmp.mspe = mirspectrum('Design', 'Frame', 512, 'sp', 1);
[calc all mspe dependent features]
[start a new flowchart and calculate the rest]

so that I at least do not have the spectrum calculated again and again?

Yes, absolutely, you can do that. In the current version released (1.2.2), the 'tmp' variable myflow.tmp.mspe would have been computed across all chunks and stored in memory for the computation of the dependent features. In the new version I am preparing, such 'tmp' variable is computed in each chunk separately, and once the dependent features are computed, it is erased from memory.

I will send you the new version once I have finished cleaning and checking the code. Anyone else interested to receive this beta- version can send me an email (olartillot@xxxxxxxxx).

Regards,

Olivier

Olivier Lartillot
Academy of Finland Research Fellow
Music Mining Plant
Finnish Centre of Excellence in Interdisciplinary Music Research
Department of Music
PO Box 35(M)
40014 University of Jyväskylä
Finland
http://users.jyu.fi/~lartillo/
Tel +358 14 260 1346





Olivier Lartillot
Academy of Finland Research Fellow
Music Mining Plant
Finnish Centre of Excellence in Interdisciplinary Music Research
Department of Music
PO Box 35(M)
40014 University of Jyväskylä
Finland
http://users.jyu.fi/~lartillo/
Tel +358 14 260 1346




Other related posts: