[argyllcms] Re: Determining proper error value for -r

Ben Goren wrote:
Would that be time and resources that us non-C-hackers could help with? If it means, for example, some grunt work of creating and measuring charts or hacking together Perl programs or spreadsheets to analyze them, I'd be more than happy to volunteer.
As it looks like you are obviously pretty interested in finding the best -r value, probably you could do something for this purpose.

Graeme already briefly mentioned the algorithm for LOO (leave-one-out) cross validation in a previous message. Cross validation is one of several methods which attempt to estimate how well the fitted model might perform on future data, which are not part of the training set. Since for LOO CV only one data point is left out in each training sub-set being fitted to the model, it is a pretty unbiased estimator. However for a large number of data points, a brute force implementation is not practical, since the model needs to be fitted N times (where N is the number of patches in the .ti3 file). For linear models, it's even possible to calculate the LOO CV analytically, without fitting the model N times, but there are some complications in practice either if the model is too complex, as it then requires the inversion of a huge matrix (e.g. a 35937x35937 matrix for a 3-dimensional 33-grid-point linear spline model), which is not feasible.

Besides LOO CV there exist also several other cross validation methods (for a brief overview see e.g. http://www.faqs.org/faqs/ai-faq/neural-nets/part3/section-12.html), for instance k-fold cross validation, which is maybe a little bit more biased, but requires the model only being fitted k times, still giving reasonable estimates (or maybe even better ones, according to the document I referred above). With k=10 and "profile -qm -B ..." this is still computationally expensive, but can be IMO computed in reasonable time on today's PCs, if one isn't too impatient. So if you like, you could for instance write a perl script which estimates the optimal -r<avgdev> via k-fold cross validation:

Pseudocode:

   parse command line arguments:
   - get measurements (.ti3 file)
   - get options which are passed through to the profile command
   - ...

   k = 10 (just an example, 10-fold is IMO a good starting point)

   Split the measurements into k subsets S[1]...S[k], with
   approx. the same number of elements in each subset

   (You may also want to put them into a random order before
   splitting into subsets, to avoid that e.g. all samples from
   a gray ramp or color ramp in the measurement set fall into the
   same subset; or alternatively you could interleave them in a
   round robin manner, i.e. 1st data point -> S[1], 2nd data point
   -> S[2], ..., k+1-th data point goes again into S[1], etc.)

   best_avgdev = -1;
   best_sse = -1

   for avgdev in all "-r" numbers wou want to try {

       sse = 0

       for i in 1..k {

           training_set = empty

           for j = 1..k {
               if (j != i) {
                   add all elements of S[j] to training_set
               }
           }

           create .ti3 file from the training_set

           create a profile from this .ti3 file, using the
             desired profile options, augmented with
             -r<avgdev> and -B, but exclude -s, -S, -t, -T,
             -c, -d, -g options since you don't need to
             waste time for creating B2A tables and gamut
             mapping, which are irrelevant for this
             consideration.

           create a .ti3 file from the subset S[i]

           run profcheck to verify the profile against the subset S[i],
           extract the RMS dE from the profcheck output

           sse += rms_dE * rms_dE
      }

      printf "avgdev=%g, SSE=%g, RMSE=%g", avgdev, sse, sqrt(sse/k)

      if (best_sse < 0 || sse <= best_sse) {
          best_sse = sse
          best_avgdev = avgdev
      }
   }

   Finally create the actual profile, from the original .ti3 file,
   using -r<best_avgdev>


Other related posts: