Question about downloading and dithering

Sequence Generator
22

I have to disagree with this. If the noise is truly random, i.e. it follows a Poisson distribution, dithering will not help at all. Truly random noise would vary from shot-to-shot in exactly the same way that a random dither would. Random is random. There would be no reduction, or increase, in random noise nor a change in SNR due to dithering.

Dithering serves a single purpose - it moves fixed-pattern noise around the field so that outlier pixels can be corrected during stacking with sigma-clipping rejection. Fixed-pattern noise is primarily warm and hot pixels but can also include column defects and some (but not all) banding from readout. The biggest benefit that a lot of people don’t recognize from dithering is that it can, in some cases, negate the need for darks. Darks largely serve the same purpose as dithering in that they are intended to provide a means to subtract fixed-pattern noise from the subs before stacking. However, unlike sigma-clipping rejection, which adds no noise to the final image, darks subtraction does add noise. In this case, by “noise”, I mean random variation in pixel brightness that follows a Poisson distribution. All of the noise in the dark images adds to the noise in the lights in quadrature. So, calibrating with darks always adds noise to the final image. That’s the reason that we collect and stack many darks. The stacking process tends to even out the random variation in pixel brightness by averaging them, making the amount of noise added to the final image less.

Since dithering adds no noise to the final image and is (typically) effective at preparing the image stack for fixed-pattern noise removal using sigma-clipping, it is possible, and even preferable to use dithering in place of dark subtraction, which is what I do. Of course, for this to work, your camera needs to add no large-scale fixed-pattern noise (such as amp glow) to the party since only pixel-level fixed-pattern noise can be removed using dithering and sigma-clipping.

Dithering can also be extremely helpful for DSLR users. Warm and hot pixel brightness varies with sensor temperature - the warmer the sensor, the more quickly warm and hot pixels get brighter. Ambient temperatures may change by tens of degrees over the course of an imaging session and as a result, hot pixels in some images may be several times brighter than hot pixels in others. Dark frame subtraction will subtract a fixed hot pixel brightness from each frame, and for those frames with differing hot pixel brightness, the subtraction may be too much, or too little. Dithering adds a secondary method of correcting remaining hot (or cold) pixels by again moving them around the frame to allow sigma-clipping to detect them.

Sorry
 I went on longer than I intended to


Tim

2 Likes
23

Nice detailed explanation @spokeshave. Would you suggest for me with my DSLR that I dither and not use darks? My Canon 6D is cooled to 46 degrees F below ambient and during the winter runs at 0 F all night long under thermostatic control. I have darks taken at the same temp for a range of exposures. Would I get better results dithering? Additionally, my frames drift more than 1 pixel, usually several, which is also dithering. The noise level in my DSLR images are quite low.

Plus an additional factor is I use Cosmetic Correction in PI to remove all the hot pixels. Does this not accomplish the same thing as dithering?

@Jared I am really impressed that your polar alignment is so good that your frames drift less than 1 pixel. How to you accomplish this? I spent many nights last fall doing polar alignment in my new obs and ended up with what I thought was a big improvment: Az error of 12 arcsec and alt error of 48 arcsec. I used PHD2 drift alignment for this which worked quite nicely. This apparently is not nearly as good as I should be getting. And that took a lot of sensitive tweaking of az and alt knobs. What accuracy is normally achievable? My mount is a Losmandy Titan. I guide with at 400mm scope using a ZWO ASI120MM camera.

1 Like
24

Well my polar alignment is not THAT good but PHD2 keeps things on target and my PA is good enough that field rotation is near minimal. Also helps to only have 780mm of FL (although this would actually make the field rotation more apparent)

That is actually quite good alignment. I’ve often heard that you can stop under 2 arcminutes but that likely depends on a lot of other factors. I have no clue how accurate my alignment is
“good enough” I suppose, likely not as good as yours.

This is likely the reason that you’re seeing a couple of pixels drift in your main scope. I image with a 780mm refractor and guide through an OAG. My imaging and guide scale are very close and my guide drift is almost always around 0.25 pixels. Mount is a G11 with some upgrades and PEC programmed via PemPro (which is probably about 4 years old at this point). My mount tracking error is around ±2 arcseconds
I call it my “poor mans Mach1”.

It sounds like your guide scale and imaging scale are very far apart? Also at the imaging scale you’re using it seems like some amount of drift is just a fact of life.

Jared

25

I have avoided OAG because of the tiny FOV which apparently requires field rotation. I have no rotators and would like to avoid that complication. Would I see a big improvement if I just swapped out my 400mm guide scope for a 700-800mm?
I know flexure between guide scope and image scope can be an issue. I don’t think mine is significant, but that certainly needs to be checked. MetaGuide has a feature to measure this.

The amount of drift I am seeing is quite a mystery to me since I am guiding with PHD2 quite successfully. Typical rms error is .8 to 1.0 arcsec, as low as .6 with good seeing. Why would there be a sizeable drift if PHD2 is keeping me on the target? My tiny stars are definitely pinpoint which would not be the case if it was drifting 5-10 pixels between frames. Maybe @Andy could chime in here.

26

It is worth a try. You have nothing to lose. Do an imaging run with dithering and try stacking with and without darks (you still need bias and flat frames). See if you notice a difference.

Tim

27

Again, my overall FOV is larger than yours but I’ve never rotated to find a guide star. There’s always something that is acceptable in the guider FOV. I suspect that if I were to use an SCT this might not be the case. But the lodestar pulls out some pretty faint stars at 2 seconds per frame.

Jared

28

That sounds like it could be differential flexure. The way you can measure this is:

  • find a good star field and start guiding.
  • take a series of short exposures in your imaging camera, say, 15-30 seconds.
  • stack the frames without aligning them.

Any star elongation in the stack indicates differential flexure or mirror shift. (It could also be caused by field rotation but not in your case since you have excellent polar alignment.)

Andy

29

I have a lot of new info on the drift issue and PI Cosmetic Correction to remove hot pixels with no dithering.

@Andy you suggested my drift could be from differential flexure, and that is now appearing to be the likely cause. The hardware change I just made changed my guiding from a 400mm FL guide scope to OAG using the 1944 mm RC12 with a Lodestar X2 guide camera. I have attached a few cropped images of NGC3718 I just took, 24 x 360 second exposures on a new QHY10 color camera. The first 22 images had a total drift of .7 pixel, and the last 2 were 1 pixel drift each (that somewhat of a mystery). So essentially all the images are stacked on top of each other. Clearly a major improvement using OAG.

I have always imaged without dithering since I run 2 or 3 scopes on the same mount, often imaging 2 or 3 at the same time. Since my drift had been large that was giving me the same benefit as dithering, plus I always use the PI Cosmetic Correction feature of the BPP. The result is perfect removal of all the hot pixels. Now with the new OAG and almost 0 drift, I have a great test of the value of Cosmetic Correction by itself. The results are close to perfect, but not quite there yet. Hopefully someone can point me to a PI feature that can take care of the remaining imperfection.

The result of using just PIs Cosmetic Correction is it removes ~99% of all the hot pixels perfectly. Only 4 hot pixels (they are white pixels) in the entire image are not removed, and they appear to be the hottest ones. And each of them is surrounded by a ring of hot blue pixels. This looks to me like it could be a bug in PI. @pfile, you might know what’s going on here. All the other removed hot pixels cover the full range of colors, at least as displayed in PI.

Here is the set of cropped images:

All the images are the same center crop. In the bottom right corner is one of the four hot pixel groups that does not get removed.

The three NGC3718 images are:
(1) Windsorized Sigma Clipping WITH cosmetic correction. Just 1 of the 4 hot pixel groups show, bottom right corner.
(2) Windsorized Sigma Clipping with NO cosmetic correction. A hundred hot pixels show.
(3) Frame #9 for comparison with the raw frames.

There is one image from a series of M51:
(1) Windsorized Sigma Clipping WITH cosmetic correction. The same 1 of the 4 hot pixel groups shows. All 4 hot pixel groups are in exactly the same locations.

The DARK master I used for the Cosmetic Correction. Shows hot pixels in all the same places they show up on the non-corrected image.

Observations:
(1) all 4 of the hot pixel groups have one feature in common: the master dark shows all four of them to have exactly 2 hot pixels separated by exactly 2 pixels in both x and y.
(2) all the other hot pixels I looked at have more than a 2 pixel separation from any other hot pixels.

My conclusion:
The PI CC routine is not sophisticated enough (or it has a bug) to deal optimally with 2 hot pixels close together. It presumably is replacing each hot pixel with a value taken from some nearby pixels, and that value is erroneously inflated by another close hot pixel, such that many more nearby pixels end up getting hot pixel values instead of correct non-hot pixel average values.

It would be wonderful if someone knows an easy way to deal with this, other than using Clonestamp on the four locations.
@buzz this might be useful info for your chapter on cosmetic correction vs dithering.

Things I have tried that do not help:
(1) all the different “Integration Rejection Algorithms” besides the Windsorized Sigma Clipping.
(2) Dark Master: a wide range of “Hot Pixels Threshold” values. Qty = 2000 leaves lots of hot pixels in the image. Qty >= 8000 removes all the hot pixels except the 4 groups of close paired hot pixels. I tried up to Qty = 80000, no change.
(3) Dark Master: added “Use Auto Detect” with Sigma = 1. No change.

30

a couple of thoughts, first is that did you try changing the high sigma slider (to a lower value) during ImageIntegration with WSC? this will cause II to reject more pixels.

second is that if there’s a hot pixel left in the calibrated image, the registration algorithm will sometimes cause ringing around the hot pixels. that might be what’s going on in the blue channel. is one of the images you posted post CC but unregistered? it would be interesting to see what those pixels look like after CC but before registration. there are 2 things that can be done during registration to mitigate this; one is to mess around with the clamping threshold for the default registration algorithm (which is Lanczos-3), or switch to a registration method that’s less prone to ringing, like Bicubic Spline, and possibly mess with the clamping threshold there as well. you need to lower the threshold to increase the deringing effect. these controls are found at the bottom of the StarAlignment process under “Interpolation”.

if ringing around the hot pixels is happening, then that may make it harder to reject the hot pixels since now they have grown and may overlap in successive subs such that they no longer look like outliers to the rejection algorithm.

third is that CosmeticCorrection has a real-time preview mode. i loaded your NGC3718_frame9 image and with auto-detect and sigma = 4.6, the dimmer of the two hot pixels in the lower right is eliminated (the brighter one is eliminated even with hot sigma = 50, so it’s a real outlier. so i’m not sure why CC is not eliminating that pixel for you - this is puzzling.

finally CosmeticCorrection has a checkbox that will display the map of corrected pixels instead of the corrected image. that could be helpful while debugging.

screenshot of CC

rob

31

Thanks for all the suggestions. None of them seem to make any difference. The surrounding blue pixels are slightly different with each combination of parameters, but is basically unchanged. Seems to be a condition that the CC can’t handle.

A real plus for me is your suggestions have made me aware of a lot of useful features in the CC process.

My solution fixes them perfectly in the 2 targets of this example. I created a ProcessContainer that just has the CloneStamp process that fixes the 4 locations by overlaying the hot pixel area with the adjacent 5 pixel area. For some images the adjacent area might have a small star in it which would not be optimum but the correction would likely not be visible in the image.

Hopefully PI will fix this so it works completely. It is so close now. Does a fabulous job of removing all the hot pixels except for this rare condition.

32

hi jerry - like a lot of processes and scripts in PI, CC was written by a user (Nikolay - NKV in the PI forum.) before it was part of the official release, NKV released the source code. i am not sure if what’s linked here is the latest code but you can take a look at it to try to figure out what it’s doing. otherwise you can post your case in this thread and maybe NKV will look at it.

33

I posted this on the PI forum and got a quick response. This is a known issue with the CC script and an easy workaround is to list the pairs of hot pixels in the “user defect list” (be sure to check the ‘Limit’ box). That fixed them perfectly and now the CC routine fixes 100% of the hot pixels. This makes non-dithering (and therefore multi-camera/scope support) very operational.

34

cool, yeah i saw that. i’ve only ever had to correct column defects and never noticed the “limit” checkbox!