On 5/7/12, Graeme Gill <graeme@xxxxxxxxxxxxx> wrote: > Elle Stone wrote: >> Does anyone have a clue what really is loading this "almost linear >> calibration but with rounding errors" vcgt information upon computer >> startup? > > Hmm. It's not a 16 bit scaling error introduced somewhere in X11 or > the video driver by using the wrong "convert to 16 bits by shifting by 8 > bits" > in an attempt at setting a linear VideoLUT is it ? > > wrong: val16 = val8 << 8; > > If you do this you get: > > 0.000000, 0.000000 > 0.003922, 0.003906 > 0.007843, 0.007813 > 0.011765, 0.011719 > 0.015686, 0.015625 > 0.019608, 0.019532 > 0.023529, 0.023438 > 0.027451, 0.027344 > 0.031373, 0.031250 > 0.035294, 0.035157 > . > . > 0.976471, 0.972671 > 0.980392, 0.976577 > 0.984314, 0.980484 > 0.988235, 0.984390 > 0.992157, 0.988296 > 0.996078, 0.992203 > 1.000000, 0.996109 > > If so, this is a coding error, not a stray calibration. > (Note that 16 bit values are used internally in X11). > > The correct way to scale 8 bits to 16 bits is to multiply by 257, > or shift by 8 bits and add: > > val16 = val8 * 257; > or > val16 = (val8 << 8 | val8) > > > Graeme Gill. > > Graeme, those values you just gave are exactly the values that I'm getting after using "dispwin -s" to check the vcgt information after restarting my computer, allowing for rounding as dispwin only printing five decimal places for the higher values ( eg 0.98830 instead of 0.988296). BEGIN_DATA 0.0000 0.0000 0.0000 0.0000 3.9216e-03 3.9063e-03 3.9063e-03 3.9063e-03 7.8431e-03 7.8126e-03 7.8126e-03 7.8126e-03 0.011765 0.011719 0.011719 0.011719 0.015686 0.015625 0.015625 0.015625 0.019608 0.019532 0.019532 0.019532 0.023529 0.023438 0.023438 0.023438 0.027451 0.027344 0.027344 0.027344 0.031373 0.031250 0.031250 0.031250 0.035294 0.035157 0.035157 0.035157 . . . 0.97647 0.97267 0.97267 0.97267 0.98039 0.97658 0.97658 0.97658 0.98431 0.98048 0.98048 0.98048 0.98824 0.98439 0.98439 0.98439 0.99216 0.98830 0.98830 0.98830 0.99608 0.99220 0.99220 0.99220 1.0000 0.99611 0.99611 0.99611 END_DATA -- Elle Stone http://ninedegreesbelow.com