SGP 5.2.1 no dither after first frame

Did a sequence last night that contained three targets with multiple frames per target. Dither set to extreme, latest build of PhD2.

What I noticed is that there was no dither shown either in SGP or PhD between frame one and frame two on the first target of the night. A blink also confirms no dither happened. It was set to dither every frame and extreme dither. All other frames showed a dither in both PhD and SGP and blink confirmed they did happen. This was even true of all frames (even the first) for the second and third targets of the night.

The bottom line is that SGP was not dithering after the first frame of the night but did for all other frames.

Anyone else seeing this? If you could have a look at some of your frames and let us know it would be helpful.

FYI, every other aspect of the imaging session was totally perfect!

I suppose that one could insert a very short “to be thrown away” exposure to start the night but that seems like a waste.

Yes, I can confirm this is happening with my setup as well. No dither between first frame and second frame. All of the other frames dithered. I did not report it thinking maybe I didn’t have something setup correctly.

mahaffm

Good to hear I am not crazy (well, not based on this anyway :wink:). I have been using SGP for several years very successfully so I was pretty sure it was not my setup either.

I cannot reproduce this behavior. Please send your SGF and logs.

Disregard… I found a way to reproduce. Seems to be an issue with older profiles or sequences and the new “dither every N frames” functionality.

I kinda wondered if the cause might be related to that new feature - it makes sense. I did use an older profile (but converted to 2.5 using the converter) but the plan itself and targets were all fresh and put together within the beta. For now I will just insert a 30 second dummy frame at the start of the sequence and delete it as soon as it downloads.

Does this mean if I make a totally new profile and sequence using the latest 2.5 that this would not happen?

UPDATE: I tried this, making a totally new profile (typed in the parameters) and totally new sequence in 2.5.2.1 and it still failed to dither after the first frame but dithered properly after that. Guess I will have to use a dummy first frame for a while.

1 Like

The math works out fine for skipping frames for dithering, within reason, and it certainly would do no harm to not dither one frame. The reason for this is because most sigma clipping algorithms are median sigma clipping, not mean sigma clipping.

The intent of dithering is to move hot pixels around such that they fall on enough different pixels that when stacked, the value of the hot pixel will fall outside of the sigma clipping threshold. The sigma clipping threshold is calculated by starting with the median for the stack, calculating the standard deviation (sigma) for the stack and then adding the median to some multiple (kappa) of the sigma.

Consider a simple example of 20 subs that are dithered every frame, and for a given pixel in the stack, the median is 450ADU, the mean(with no hot pixels in the pixel stack) is 500 ADU and in one frame of the stack, a 10,000ADU hot pixel appears. The mean used to calculate the sigma-clipping threshold would be (19500 + 110,000)/20 = 975. Sigma is simply the square root of the mean, or about 31. If we choose the commonly-used kappa of 3, the rejection threshold for that pixel is 450+3*31 = 543. So, clearly our 10,000 ADU pixel would be rejected and replaced with a median pixel value of 450. In fact, any pixel with an ADU greater than 543 would be rejected. This example would therefore reject any pixels 21% or more brighter than the median.

Now let’s look at the same example, but dithered every 4 frames. For each set of 4 frames that are not dithered, the hot pixel will be in the same place, but for the remaining 16 frames, it will be in a different place. So for the final stack, for each pixel in which the hot pixel has appeared, the mean will be (500x16 + 10,000x4)/2 = 2400. However, for median sigma clipping, it is the median that is most important, and it is far less affected by additional outliers than the mean. In the first example, most of the pixels were scattered about the mean with one outlier. The median is the middle ADU number going from the lowest to highest. Since 19 of the 20 pixel values in the first example were near the mean, the median was also close to the mean. In the second example, we added three more large outliers, which changed the mean dramatically. But since 16 0f the 21 pixels in the stack are still scattered about the mean, the middle or median pixel value will remain statistically unchanged. However, the mean is changed, which means that the sigma value will be different. In this example, the sigma is about 49, so a 3-sigma clipping threshold for this stack would be 450+3*49 = 597. So, the 10,000 ADU pixel clearly still gets clipped. However, it is important to note that the threshold is higher, so, any pixels less than 597ADU would not get clipped. We lose the ability to clip the much dimmer bright pixels since our threshold rises to 33% greater than the median. This is the downside to not dithering every frame.

But dithering every few frames has the most value when a lot of subs are taken because the time overhead for dithering can become significant. The more frames that are taken, the less the non-dithered frames affect the mean in the stacking calculation. Using the same example as above, but this time, stacking 40 frames instead of 20, the stack mean for a pixel with a hot pixel in 4 frames of the stack would be(36500 + 410,000)/40 = 1450. The sigma would be 38, and 3kappa-sigma would be 564. So our clipping threshold now drops to 25% greater than the median. It might seem that this is still marginally worse than dithering every frame – and it is – but it is important to understand that there is a fair amount of uncertainty associated with the median. Since the outliers don’t really affect the calculation of the median (within reason), the actual median selected for any given pixel stack will be subject to Poisson noise and is essentially randomly selected with a standard deviation roughly equal to the square root of the median. So, while we have been using 450 as the median in all of our examples, in reality, it could be 450 +/- 20 or so. As I hope you can see, this uncertainty completely swamps the small difference in rejection threshold. The bottom line is that for a sufficiently large stack, there is essentially no difference in the performance of sigma-clipping between frames dithered every frame and skipped dithered frames, provided that the number of skipped dithered frames is reasonable.

But that’s not the end of the story. If a reasonable dithering frequency is selected (like four used in this example) the time saved can significantly increase the number of subs collected, particularly if the subs are short and the dithering overhead is large in comparison. In this case (which is the base case for the request to be able to dither every “X” frames) we have the potential to dramatically increase SNR in the same imaging time period with absolutely no other penalty. Additionally, the subs are likely to be of higher quality since for most of them, there will be no settling after a dither – a time period when guiding can often be poorer even after the settling threshold has been reached.

Of course, there are limitations. If one is doing long-exposure narrowband imaging where only a handful of subs are collected in a session, dithering every frame is probably essential. But in that case, the total dithering time overhead is insignificant compared to imaging time. However, for short exposures sessions where a lot of subs are collected, skipping dithered frames can result in substantially more, and better quality subs collected with no penalty.

So, I challenge your assertion that dithering every frame is somehow the “right” way and that the math doesn’t work out for skipping dithered frames. It simply isn’t true.

Sorry for the lack a paragraphs above - I wrote it in Word and pasted and clicked reply before I noticed that the paragraphs didn’t come over. Best I can tell, one cannot edit a post…

Tim

Pencil Icon at the bottom of the post.

The beauty of having an option is that people can choose not to utilize it. You are certainly free to dither every frame if you think that is best. However, I thought I made a compelling argument, including mathematical examples, that skipping dithering can improve image quality if done properly. Unless you have some sound mathematical argument to the contrary, of have found a flaw in my reasoning, I stand by my assertion. I find the argument that you’ve been “doing it that way for 20 years, therefore it is right” to be lacking.

Tim

That’s odd. I don’t see a pencil icon, but when I hover near the bottom of the post, I can see the options.

Weird…

Tim

I hope you noted that I never said experience was not important. It is probably the single most important part of astrophotography. What I tried to convey is that experience alone is not a basis for a mathematical argument.

I’ll bow out now. I have enough “experience” to know that when someone else resorts to putting words in my mouth, there is no point in discussing further.

Tim

I have withdrawn all argumentative posts. It is a waste of time. Folks will do what they please anyway. My results speak for themselves.