Banding on bottom of image

UPDATE:
It does work well on some images, and very poorly on others. It works best when all the images are on the same side of the meridian so that all of the banding is isolated to either the top or bottom of the image, and the image does not have lots of nebulosity in the banding area, such as M42. In those cases it works poorly by totally fixing it on the bottom banding area, but screwing up other bands higher up in the image that never had any noticeable problem. Changing the intensity to less than 1 moderates both effects, but often I just end up cropping the image since it did not improve things enough to use.

well the canon banding script is supposed to remove a particular sensor oddity in some canon cameras where there is banding throughout the frame, not just at the top and the bottom. you might mess around with the “highlights protection” to try to prevent bogus band fixing.

depending on the f/ ratio of your optics, sometimes the mirror casts a shadow on the sensor. however, that should be in the flats and should be removed by flattening, so not sure exactly what’s going on here.

It is, indeed, very strange what is going on here. I have played with the “highlights protection” and the “sigma”, and I can’t get either one to make any difference. The only one that has a strong effect is the strength parameter. And it is also quite a mystery why the flats do not correct this, which for me they have no effect. Could it be that if I took sky flats they would help a lot better? I always take my flats with a light panel about 2 feet in from of my 12" rc.

in general how well do your flats flatten the lights? in the images linked above by Dean, the CR2 has a dark band in the subexposure and a bright band in the integrated image. to me, that’s a symptom of bad flattening, which is either due to low SNR in the flats, or bad calibration of the flats leading to overcorrection during flattening. sometimes the effect in a single subexposure is pretty subtle and the overcorrection is only seen clearly in the integrated image.

DSLRs are of course pretty hard to calibrate unless they are cooled, so that may lead to bad calibration of the flat subs and subsequent overcorrection of the lights. there are some examples of “fixing” flats that overcorrect on the PI forum but i don’t have the links offhand.

i have used both sky flats and panel flats and for mono CCDs the results have been similar for me. when using a panel, i generally have the panel just a few inches from the aperture. when i used a DSLR i also used a CLS filter which gives a very strong blue cast to the flats. not necessarily a problem but it meant that the red channel was relatively underexposed and had bad SNR. so i eventually started using a slightly pink t-shirt when making sky flats.

Interesting points and makes me wonder if it might help to run the fixer script against the master flat before using it. Have you tried that?
Here is a link to a full set of masters that I used to process IC4603: bias, dark, flat, resulting light using the dark master, resulting light not using the dark master.

This is probably the worst case I have tried the fixer on since the images come from both sides of the meridian, and there was a major rotation between different sets which produced an angular band across the top and level band on the bottom. I assume it was rotation, although I have no idea how it could have happened. I have not even been in the obs for 2 months so none of the hardware has been touched. I took the 32 lights on 2 different nights, all at 240 seconds, iso 1600, and cooled to 0 degrees F, since my cooler gives me 46 F below ambient and the ambient was below 46F when I took these on 4/6 and 4/9 between 3am and 5am.

My darks match the light exactly, and I even included the run without using the darks, which gives the same result. All masters were taken with the cooler running since the cooler and camera use the same power switch. Some of the masters might not have been taken at 0 F since I did not keep close track of the temp when taking them, but would have been at least around 20 F or better.

pfile, hope you can make some sense of this. Thanks for the help.

by fixer script do you mean the canonbandingreduction script? the flat fixing i was referring to is essentially a flat normalization step performed after creation of the flat master.

i can see in your flat master linked above that the red channel is indeed kind of weak, but i don’t know if it is related to this problem. can you post the CR2 for one subexposure?

the flat is interesting - when debayered the mirror shadow is quite noticeable (the sensor is read out upside-down so the shadow appears at the bottom instead of the top where it belongs). also in the R and G channels there is a circular wave-like pattern which does not appear in the B channel. was this master flat made from sky flat subs or panel flat subs?

the canonbandingreduction script is really only good for banding that’s exactly parallel to the long axis of the frame… it’s meant to be run on subexposures. since the predominant banding here seems to be due to the rotation + possibly the mirror shadow problem, you’ll get weird results as the banding is not parallel to the edge of the frame. having said that the banding in the flat is probably reasonably parallel, but my feeling is that removing the mirror shadow from the flat is the wrong thing to do.

oh - one more edit. how long are the flat subexposures? given the filenames here it seems like everything was created with BatchPreprocessing, which is going to try to scale the dark when calibrating the flat subs. i have pretty much always only calibrated my flat subs with a master bias, because my flats are usually pretty short (1-10s). this has to be done by hand, however, because BPP always tries to use the dark on the flats. so if you have some uncalibrated flat CR2s that would be helpful as well.

Wow pfile, I am really impressed with your expertise on this subject. How is it you know so much about this?
So to answer you questions,

Yes, by fixer script I am referring to the canonbandingreduction script, and yes I always use the BPP script for my preprocessing. I also always have ‘Optimize dark frames’ checked but that is not in play if I don’t use the dark master, and the light I sent you which did not use the dark master seems to be basically identical to the one with darks, so I would think that is not a factor in this, at least for the light itself. Now on reading your last paragraph more carefully, I see that you are referring to the scaling of the master flat with the darks. Yes when I made the initial run to create the master flat it scaled the dark on the flats.

I expect this is caused by my using a Hutech HEUIBII filter for all the 6D images. If you are not familiar with this filter, at passes all the visible bands except the red where it passes a very wide version of an Ha filter, so it is cutting off all the red that is not somewhat close to Ha. This effectively produces RGB with enhanced Ha, and it makes sense that the red channel would be comparatively weak. It seems to have produced wonderful images for me that appear to have enhanced Ha much like actually adding an Ha filter to the processing.

All my flats are made from a flat panel 2 ft in front of the ota. The panel is mounted on the side of the obs so I can at any time turn it on and slew the scope to point to it and take flats. I am rarely at the obs so this works well for me. This flat master is 80 subs, 2/10 (.2) second exposure, iso 500.

Yes I knew that and expected it would have trouble with the angled one on top, but I wanted to show a really hard test example. Actually I got a worse result on one of my M42 sets with the same hardware where the band was only on the bottom. It removed the bottom band perfectly but also reduced the intensity of a much wider band smack halfway down the image to very dark. Many images it does a perfect job on, but some it also does its thing on 1 or more other bands that don’t have a banding problem. Come to think of it, the ones I thought were perfect maybe It did a lot of processing on other bands that I never noticed because they were basically black, like open clusters. I will try some more.

It would be great if it had an additional slider to restrict what portion of the image to work on. Then it would be totally awesome.

Here is one light sub and one flat sub in FIT format. I only get FIT images out of SGP.
https://www.dropbox.com/s/bi94mc7nfw612f1/Canon6D-FLAT-LIGHT.zip?dl=0

And finally do I have this correct: You recommend I run the canonbandingreduction script individually on each subexposure and not on the final integrated image. That is clearly a lot more work. Will it really make a big difference? And not run it on bias, darks, or flats?

well… i have messed around with the data you posted and i pretty much have failed to get a calibration where the shadow is not overcorrected. i have a few ideas why you might be having trouble though.

1st - the flat was taken at iso500 and the lights at iso1600. because of this i’m reasonably sure that the data is incompatible with BPP. the readout noise is different at the two isos, and so the bias frames will be different; the flats really need to be calibrated with an iso500 bias, and the lights with iso1600 darks and bias. BPP is not set up to handle multiple ISOs… i understand that it’s necessary to lower the iso for the flats or else the shutter speed is way too high; if you’re going to stick with this method then you’ll have to resort to a manual calibration flow - calibrate the flat subs with an iso500 master bias, calibrate your dark subs with an iso1600 master bias, then load the master bias, the calibrated master dark and the calibrated master flat into imageintegration when calibrating the lights. i am guessing that attempting to calibrate these flats with (what i assume is) an iso1600 bias is what’s messing up the flat scaling.

given that you’re using that type of filter, i think it’s probably also necessary to use a slightly pink light source to try to get the master flat back to being grey. you could actually kill two birds with one stone by using some kind of pink material to both compensate for the filter and attenuate the flat; if you attenuate it enough you may be able to use iso1600 for the flats and then you’d be able to use BPP again.

god bless SGP, but i’m personally wary of software that touches the output from the camera before writing it to disk. i’ve never used SGP with a DSLR; as you say there may be no choice. the reason i bring this up is that almost all canon cameras have 14-bit D/A converters, and when you use DCRAW (or pixinsight) to convert the CR2 to fits or tiff, the 14 bit data is put into the lower 14 bits of 16-bit integers. this means that a completely saturated (debayered) flat would read out as 0.25 on the pixinsight histogram display, and so you want to aim for flats that have their histogram humps centered around 0.125. if i debayer your flat, i see the green channel all the way up at 0.45, which i assume must mean that SGP has scaled the data. not that this is a showstopper, but it would be interesting to know what a fully saturated flat looks like coming out of SGP so that you can judge the flat exposure. if SGP gives you 1.0,1.0,1.0 for a saturated flat, then in fact you want to set your exposures so that the histogram humps are around 0.5 rather than 0.125.

as for the CBR script, what i’ve done in the past was to segregate my lights into groups which have similar position angles, thus mostly guaranteeing that the stacks made from each group has minimal rotation and thus the banding is still mostly horizontal. after fixing each stack with CBR, i’d then register and integrate the stacks. this is suboptimal since there have been multiple interpolations on the data, but indeed it’s easier to run CBR 3-4 times rather than on each sub. having said that you can probably run CBR on an ImageContainer full of subexposures, but i can’t remember if that actually works.

with respect to the script messing up parts of the image that don’t need help, i think you can apply a mask to the image that only exposes the banded areas before applying the script. that should work.

i don’t think that CBR should be run on any calibration frames. the problem that it’s supposed to solve is a random, unrepeatable banding in all frames from some canon cameras - 40D in particular if i remember right. IMO it was intended to be used on calibrated lights or stacks that showed banding.

anyway, i think that in theory this problem should go away with better calibration, and the CBR script should not be necessary.

rob

Great info, Rob. Thanks.
Based on your earlier help I started an all night sequence of 320 bias frames and as many 240 second darks as I can take over the course of tonight’s snowy weather, on both my 6D and my SBIG8300M, which of course does not have any banding problem but I needed to do anyway. All to the purpose of completely redoing all my masters on both cameras. I could just redo all the masters using all the subs I used previously, but I have wanted to redo the masters for some time with a lot more subs, particularly for the bias masters, so this is a good opportunity to do so.

To do this I think I can do everything you have suggested within BPP. Why does it not work to run BPP 4 different times, each time using the correct mix of masters to calibrate the 1 group of subs that needs calibrating?

First run creates the iso500 Bias Master, other subs don’t matter.
Second run creates the iso1600 Bias Master for which I will by tomorrow have 320 subs
Third run uses the iso1600 Bias Master to calibrate the iso1600 Darks
Fourth run uses the iso500 Bias Master to calibrate the Flats

Now I have correct masters for Bias, Flats, and Darks
All runs integrating lights use this set of masters. I only have to do this not very difficult process once, after which it is clear sailing.

This seems like a reasonable solution to fixing the master flat balance, but I think it could be very tricky in practice to get a good balance. Why can’t I just use pixel math or even Color_Calibration on my master flat to balance it? Don’t those two processes just scale all pixels proportionally? That’s what we do with the integrated light.

I will get you one tomorrow, and take a whole new set of flats.

The angled banding on the top of this image is a real mystery to me. It is the only example of this among roughly 20 targets for which I have processed Canon6D images. The cameras on all three of my scopes have a fixed angle of very close to 90 degrees, and none of them have been touched for 10 weeks since I have not been to the obs during that period. Since none of the other images have this issue, I think I will chalk it up to an astro space gremlin doing mischief.
Except of course where subs include both sides of the meridian so there is a horizontal band both top and bottom. So this is an excellent suggestion for solving those images. Coupled with the great suggestion to mask just the area where there is a band should do a fine job of making the CBR work for me.
Thanks Rob.

that’s true, you can do that. i don’t use BPP so i don’t know off the top of my head, but i thought its standard flow would be to create uncalibrated masters, and then calibrate them at the time of light calibration (essentially by ticking the “calibrate” boxes in the master dark and master flat section of ImageIntegration, under script control of course. if you feed it pre-calibrated masters, my worry is that it would attempt to calibrate the masters a 2nd time, which would be a disaster. in the worst case you can take your BPP-created masters and just use them in ImageCalibration to calibrate your lights.

the problem is that those tools are designed to work with 1-plane or 3-plane RGB images. the calibration frames from your 6D are always used by pixinsight while still in bayered format. so in order to do those things, you’d either have to use SplitCFA on your images to get the individual images from the bayered image, do the surgery, and then somehow get the images back into mono CFA format or RGB CFA format. not easy. i think if you’re going to go down that route you might as well just use SplitCFA and then treat your OSC like it was a mono camera with 4 filters. at that point you wouldn’t have to worry about the flat scaling since you’ve got 4 mono images. but SNR is SNR and just boosting the level of the red channel won’t actually improve the SNR.

it’s not terribly too hard; at the time i was using a DSLR i used t-shirt flats and i was able to find one that was just pink enough to balance the flat. it doesn’t have to be perfect; you’re just trying to get the histograms to mostly line up.

anyway if this all works out hopefully you won’t need to worry about the CBR script.

I am pretty sure the risk you refer to here is not an issue with BPP. My understanding is when BPP creates any master, it is a fully calibrated master. When the calibrated masters are fed to the script, via checking the master box for bias, dark, or flat, it stars every image loaded on the respective tab as a master and never alters that file and never tries to calibrate it again. ie. masters are assumed to be calibrated by PI. I sure hope I am correct in this assumption.

I have always used BPP this way and get fine results. I usually have a stock project icon for BPP that has them all pre-loaded and I load several Dark masters at the same ISO as the lights but with different exposures so I can run the same script with an arbitrary light exposure set, since BPP will (I assume) pick the dark master exposure closest to the exposure of my lights for processing in the script.

Use SplitCFA, what a great idea! Then I will have a work flow identical to my CCD workflow from that point forward. I have to do this anyway if I want to add Ha images from my CCD into the RGB images from the 6D. Since I have three telescopes/cameras on the same mount, all possibly taking images, I can easily do this. And I swap the CCD with the 6D at times to get the FOV+camera combination that works best for a series of targets.

In the simple case of 1 telescope and 1 camera this makes a lot of sense. In my case this does not work because I use the same light panel for all three telescope+camera units on my mount, and usually controlled remotely from far away, so I can’t manipulate any hardware in the observatory.

This week when I am re-configuring the whole shebang, I will remove the Hutech HEUIBII filter from the 6D and see if that improves these issues. It will definitely add a lot of red to the RGB image and should yield a good color balance. I maybe should never have messed with the HEUEIBII filter.

And on the whole multi-step process we have outlined for BPP to accommodate the mix of iso500 with iso1600: I am redoing all the lights anyway. I should just redo them at t=1/16 second and iso1600 which will give the same exposure as the existing t=1/5 second and iso500 to totally simplify the whole thing. I haven’t touched any of the hardware for 10 weeks so lights I take now should work about as well as the ones I took 10 weeks ago for all the images I took during that period. They should be better for recent images and maybe slightly worse for the oldest images if any foreign spots have accumulated on my objective lens during that period. Basically an even trade-off.

A final question on the SplitCFA approach. Not sure what difference this makes to this color balance issue, or even why it changes anything. I am assuming by SplitCFA you are referring to the PI toolbar button labeled “Split RGB Channels”. This produces separate R, G, and B images, which I have used before, particularly if I want to add some Ha from my CCD. I then immediately run the RGB_Combination tool on these three component images, and that gives me another RGB image, presumably and visually identical to the one I started with. I am clearly missing some important step here.

i just read the source code and if i understand it right, flat masters are calibrated, but dark masters are not. so this means that what you are trying to do should work. this may have been designed exactly for this case, i would have to ask juan.

understood, yeah, if you’ve got a remote observatory and multiple OTAs then you’re kinda stuck with your flat panel the way it is.

SplitCFA is actually a separate process. it’s purpose is to take CFA images either in “mono CFA” or “3-plane CFA” and extract the 4 images from the bayer matrix. it’s a little like superpixel debayering, but you get 4 images since there are 2 green pixels in each bayer quad. (superpixel debayering averages the two green pixels together while making the green channel of the RGB image.) the idea would be that you run SplitCFA on all your masters and your lights, and then treat the calibration flow as though you had 4 sets of mono images (bias, dark and lights for R, G, G, B). the “split RGB channels” button in PI is intended for 3-plane RGB images, such as what you’d get after debayering an OSC image. it doesn’t understand CFA images.

the color balance issue with the flats stems from the fact that all calibration of OSC images is done on bayered (CFA) images. as far as i can tell, there is a single scaling factor that’s computed for the entire flat, so if you have one channel that’s significantly weaker than the others the globally-computed scaling factor is not correct for the weak channel. if you’ve used SplitCFA then you’re calibrating 4 mono images and each flat is composed only of pixels from the particular bayer site it belongs to, so there’s no “color” issues anymore.

i’ll ask in the PI forum about these issues. it’s been a long time since i used any OSC camera so the answers to these questions stopped being important to me before i really understood everything about calibration of CCD images…

Nice explanation. Thanks.
I will forge ahead by removing the HUEIBII filter so I should get a good color balance automatically. Should have tried it long ago. And I will take new flats at iso1600 so the whole process should be very straightforward using only BPP.
The complete dialog is very important to me because I will probably need to use the more complicated process for older image sets where I also have mixed the isos in the masters.
I will keep you posted.

thanks - i asked in the PI forum about these random issues. we’ll see if juan answers.

you know what, i just happened upon a subtlety that i had not realized before. your fits files are mono CFA files and as such have one channel containing the raw sensor data. this is distinct from the bayer RGB format where there are 3 planes, each containing a “sparse” matrix containing only the pixels for that particular color. when representing OSC flats as bayer RGB, pixinsight computes a scaling factor for each channel independently, but when using mono CFA there is only one scaling factor. BPP seems to also choose to read in CR2 files as mono CFA as well, so even if you had CR2 files the behavior would be the same. i think this is an argument for trying to make your flats look grey when using BPP or risk improper scaling.

Well that explains why all my masters are greyscale and not RGB. I was going to ask if that made any sense for RGB lights, but now I know. They are a monochrome storage format, but the RGB data is embedded within the single channel.
But I still don’t know how to tell if they are grey or not. They display as greyscale regardless.

so i did an experiment using some old flats i had taken thru an astronomik Ha filter and a modified 50D - this gives a really strong color cast. the upshot is that no matter the format, mono cfa or bayer RGB, the flattening is perfect. in retrospect this probably should have been expected, but it’s still a bit counter-intuitive to me that a single scaling factor applied across all 3 channels yields the same result as scaling the channels individually. after all, if the scaling is wrong, my assumption is that the flat will over- or under- correct the light. maybe that’s a wrong assumption.

anyway, your results with the matched bias files will be interesting. the above experiment seems to indicate that the only thing that would cause a flat to overcorrect a light would be improper calibration of the flat. low SNR is bad in a flat with a color cast, but it apparently won’t cause mis-flattening of a light.

by this do you mean you don’t know how to tell if the channels are well-balanced? all you need to do is debayer the raw flat file (there’s a process in PI called… Debayer) and check the histogram of the debayered version of the file.

rob

well… it’s just a huge day of discovery over here :slight_smile: - another SGP user is having trouble with DSLR flats… i did the experiment of loading one of my CR2 files into SGP and writing it out as a fits file… and it’s clear that SGP rescales the 14-bit data to 16-bits. this means that if you make a master bias out of CR2 files and then try to calibrate an SGP-captured DSLR image with that master bias, the calibration is going to be bad. i don’t think that’s the situation you are in, but it does explain why your flats had values > 0.25 in them. i guess the lesson is (as always) don’t mix frames from different image acquisition packages.