SGP-generated fits files from canon DSLRs

it looks to me like SGP rescales the 14-bit (or 12-bit) canon DSLR data to 16 bits when writing out fits files. would it be possible to add a toggle to carry the 14-bit data straight across to the low 14 bits of the uint16 rather than scaling it?

thanks

rob

I guess that depends. Why is this helpful?

well, it makes SGP fits files incompatible with CR2 files for calibration purposes. and it kinda goes against the unwritten rule of not touching the sensor data in an image acquisition program. plus, the scaling seems kind of odd; saturated stars appear to have ADUs in the 57000s, not 65535 as i’d expect (but i suppose the camera itself could be clipping at < 16383, hard to know)

Fits and cr2 will almost always be incompatible unless you happen to get really lucky. And even if they are compatible from the capture application you have no guarantee that the processing application will treat them the same.

Scaling provides no ill effects. We’re just representing the data in the native format of the fits. I don’t believe it is possible to set the fits format to 14bpp.

Thanks
Jared

the fits files from SGP seem to be mono CFA so aside from fits reader direction there’s at least some hope of them being compatible. i do realize though that every non-canon package has a little different idea about the dimensions of a particular camera’s output, but if you are using DCRAW then you’ll at least be compatible with other DCRAW-based applications as far as the dimensions go. DCRAW can def not scale the data if you ask it not to.

the saturated values concern me; i’m pretty sure that if you open a saturated CR2 in DCRAW non-scaling mode you’ll get 16383 in all channels, but i will check again. if the scaling in SGP isnt exactly a multiplication by 4 then the data is being changed, which is not good. so far i have N=1 which is not a very good sample size, but it’s clear in that file that saturated pixels != 65535.

just to be clear i’m not saying that the fits format needs to be i14 (which as you say, does not exist), what i am asking is that you put the 14-bit values into i16 right aligned, that is, the top two bits are always 2’b00. that avoids any manipulation of the data by the capture program.

by the way, i did some experiments. first, i am making the assumption that you are calling dcraw as follows:

dcraw -4 -o 0 -r 1 1 1 1 -d -t 0 -c

little d being the key here, which causes the scaling (vs -D, which does not)

if this is not the case, then the following analysis is pure BS.

i read dcraw.c and while it’s a little hard to follow with all the globals, it’s clear that the scaling is not simply expanding 0…16383 to 0…65535 by multiplying by 4. instead it’s finding out the value of the “blackest” pixel and the value of the most saturated pixel, and then expanding the data to fill that range. i think the “black” pixel must be coming from an overscan region because in light frames i tested the lowest scaled value in the actual image is still much greater than 0.

i tested 139 CR2 files from my canon 50D (taken over 3 consecutive nights), and found that the “black” pixel value was not constant. here is a sorted list showing how many frames had identical scaling:

1 Scaling with darkness 1001, saturation 15763, and 1 Scaling with darkness 1003, saturation 15763, and 1 Scaling with darkness 1005, saturation 15763, and 1 Scaling with darkness 1008, saturation 15763, and 1 Scaling with darkness 1018, saturation 15763, and 1 Scaling with darkness 911, saturation 15763, and 1 Scaling with darkness 946, saturation 15763, and 1 Scaling with darkness 954, saturation 15763, and 1 Scaling with darkness 965, saturation 15763, and 1 Scaling with darkness 971, saturation 15763, and 1 Scaling with darkness 982, saturation 15763, and 1 Scaling with darkness 988, saturation 15763, and 1 Scaling with darkness 992, saturation 15763, and 1 Scaling with darkness 998, saturation 15763, and 2 Scaling with darkness 1006, saturation 15763, and 2 Scaling with darkness 1007, saturation 15763, and 2 Scaling with darkness 1015, saturation 15763, and 3 Scaling with darkness 1021, saturation 15763, and 4 Scaling with darkness 1019, saturation 15763, and 6 Scaling with darkness 1009, saturation 15763, and 7 Scaling with darkness 1011, saturation 15763, and 7 Scaling with darkness 1014, saturation 15763, and 10 Scaling with darkness 1010, saturation 15763, and 10 Scaling with darkness 1020, saturation 15763, and 11 Scaling with darkness 1012, saturation 15763, and 12 Scaling with darkness 1013, saturation 15763, and 13 Scaling with darkness 1022, saturation 15763, and 16 Scaling with darkness 1016, saturation 15763, and 20 Scaling with darkness 1023, saturation 15763, and

granted these are not huge differences, but the spread here is pretty significant (112 counts).

so anyway, my point is that the scaling is not a constant function, but is somehow dependent on the data in the frame.

secondly, DCRAW is outputting 16-bit integers… which means that all this floating-point division and multiplication is happening, and then the decimal places end up truncated (or maybe rounded, i don’t know what happens when a ushort is assigned from a RHS containing floats). either way, this means that multiple ADU values in the 14-bit space map to a single ADU value in the 16-bit space.

both taken together it’s quite a bit of modification of the raw sensor data, and the changes are not reversible.

anyway in practice does any of this matter? i don’t know. i just feel that a capture program should avoid touching the camera data as much as possible. assuming this analysis is correct, -d touches the data in somewhat unpredictable ways.

rob

Hi Rob,

from the log file (Version 2.5.1.14):

[19.05.2016 23:59:09] [DEBUG] [Sequence Thread] Canon Camera: Saving FITS file…
[19.05.2016 23:59:09] [DEBUG] [Sequence Thread] Created full file name (file does not exist): C:\Users\HCO\Pictures\Astro-SGP\M51\Light\600sec\2016-05-19 L_0001.fit
[19.05.2016 23:59:09] [DEBUG] [Sequence Thread] Extracting RAW data…
[19.05.2016 23:59:09] [DEBUG] [Sequence Thread] RAW arg data: -D -T -4 -W -v -t 0 “C:\Users\HCO\AppData\Local\SequenceGenerator\Temp\cannonRaw-0.tmp”

The “d” parameter would have been extremely bad for the calibration process.

Regards,
Horia

thanks for the logfile… i was too lazy to hook up my 50d to SGP (i can’t find the battery anymore) so my analysis is garbage.

given that it uses -D, SGP itself must be scaling the data, which if it’s repeatable and not dependent on the image data the way dcraw’s scaling it, should be just fine.

rob

I’ll look in to adding an option to the dslrs to disable scaling. I’m not going to do this across the board though as it would destroy everyone’s calibration data.

Thanks
Jared

yes to be honest because SGP has had scaling from the beginning the change could be disruptive/confusing. thanks for considering it.

IMO there’s no need for SGP to be directly compatible with any particular processing package, but since so many people are using PI these days, if you called DCRAW the same way it does at least there would be some chance of the fits files being compatible with the way it handles CR2.

To fix lag in your camera , hard reset your camera to default settings by following steps. I hope this method will resolve your issue. If still there is lag in camera then fix it using Canon 50d Manual.

  • Turn ON the printer.
  • Press and Hold the “Reset button” on the printer.
  • While pressing the “Reset button“, press and hold the “Color Start” button.
  • Wait for 5 seconds.
  • Release the “Color Start” button, and then release the “Reset Button“.
  • Wait until the printer starts to configure itself. This should take approximately around 20 seconds.
  • The no. “1” should be on the display, press and hold the “Tool” button and the letter “A” should now be on the display.
  • While pressing on the “Tool” button, press and hold the “Reset button“.
  • Wait util there are no more blinking lights on the printer.
  • Release the “Tool” button, and then release the “Reset Button“.