No Chris - it’s how measurements are made in scientific work - with associated error bars. If anything, it is THE way to do it.
The fwhm measurement is the pertinent ground truth value you are optimizing. A Bhatinov mask is indirect. A temperature measurement could be used to estimate focus - but it is also indirect. A measurement of a V curve far from focus is indirect.
But a curve through focus is a direct measurement of what you are optimizing. And how well it fits the curve translates into uncertainty in the final answer - when error propagation is applied based on the model.
You need to make sure the curve is repeatable - but with a well behaved focuser it will be.
If you want to get a better sense of confidence in the process you can repeat and measure scatter in the results as a form of confirmation - and they should be consistent with the estimated uncertainty. But there is no need for a separate measurement of something else - because the fwhm is the very thing that is being minimized - and the measurement of it is by definition the quality of the image.
If you don’t like HFR or fwhm as the best indication of image quality - feel free to use something else and its associated model. But the key is to have a quantified figure of merit that you value. If you don’t have such a thing you can quantify - then there is no way to compare method performance or talk about in-focus vs. out of focus.
The same data used to make a measurment cannot also be used to demonstrate that the measurement is correct.
It doesn’t matter what methods are used to generate the data, the same data cannot be used to make a measurment and demonstrate that it is correct.
This is true no matter how many red herrings you introduce how yu anipulate the data you can’t use it twice, both to make a measurment and demonstrate that it is correct.
It doesn’t matter how many people you convince by this sort of pseudo-scientific mumbo jumbo it is still wrong to use the same data to make a measurment and demonstrate that it is correct.
You are clearly confident in what you are saying - but I have no idea where it comes from.
We have a measure of image quality - and we are finding the value that optimizes it based on a model and controlled changes in one experimental variable. There is nothing else involved.
You seem to think the value found isn’t the “real” focus - and we need to confirm that it “really” is. But by construction the minimum in the curve is the real focus - because it is optimal based on the figure of merit that concerns us. So this amounts to standard experimental model fitting. We could be finding a mass or a temperature or anything - based on measurements and a model.
How on earth would a bhatinov mask fit in this? What “other” measurements are you proposing to verify we are “really” “in-focus”?
I think I know the source of the confusion and perhaps I can clear it up.
If you train a model based on data and then show how well it works with that data - that is bad and misleading. It will imply your model is much better than it is - and you should not do that. It is indeed bad.
But if you take all the measurements and determine parameters of the model - and their uncertainties - and then state those parameters with error bars - that is perfectly fine. Go ahead and use all the data and get everything you can from it to nail down the parameters.
In the case of a parabolic fit you get the polynomial terms - and their uncertainties - and that is all perfectly fine. But what you really want is the location of the minimum - and its uncertainty - and that is where error propagation is involved.
But this is all not only fine - it is THE way to do it. Take data - fit the model - and find uncertainties in the parameters and derived quantities based on those parameters. Show the results as values with error bars.
At no point in this are we applying the model to the original data and saying how great the model is.
I think the point is that you have not demonstrated that the best focus chosen by the quadratic fit is better than the one chosen by SGP. It is a legitimate question. You are applying a quadratic fit to a data set that isn’t quadratic. Nor is it hyperbolic. The physics of the light cone is that it is linear until it isn’t. Near focus it becomes non-linear, but certainly not quadratic. It is much higher order and if the CFZ theory is to be believed, the non-linear section nearer focus should have curved shoulders and a flat, linear center where FWHM does not change with focus position (the definition of CFZ).
It may very well be that modeling the focus curve as quadratic - even though it isn’t - may indeed lead to better focus than the SGP process, which models the regions on either side of focus as linear - which they are. But it is not obvious that you have proven that. Just saying your model is better isn’t sufficient. Having said that, it does appear that when the dataset is of good quality, both your model and the SGP model agree well.
The question to me is how well your model copes with poor data. Your model does appear to find a minimum in what appears to be about the right place when the SGP model utterly fails by selecting a focus that is clearly nowhere near actual focus. So, it that respect, your model clearly does better. What we don’t know is whether your model actually predicts good focus, or simply better than SGP but still out of focus.
I have no idea if the minimum of the curve you get is the ‘real’ focus and neither do you. All you know is that it is the minimum of the curve.
If you want to claim that it is the best focus - and in particular better than the SGP derived focus then you need to collect more data. This means measure the star size at the arious candidate focus positions and demonstrate that your claimed better focus position really is better using the new data.
This quote:
Shows that @jmacon believes that the original data can be used to generate a better focus position but gives no additional data to support this.
All the curve fitting shows is that a curve can be fitted to the data. it can’t say that the minimum of that curve is a better focus position than some3 other position without additional data. In some cases it loos as if it should be but @jmacon is going far beyond that - with no evidence.
Why mention a Bahtinov mask? I didn’t. Nor did I mention FWHM or HFD. Trying to invalidate what someone is saying by attributing things to them that they didn’t say isn’t very nice.
In this whole discussion I get the feeling we are talking about different things. I just have no idea what you are really trying to say in this whole post. This process is so simple. We have a collection of data points. They usually show some form of V curve with a low point near the center. We (the entire world of astonomers) have been using these V curves to come up with what that ONE V curve implies is the best focus. Your entire post is saying that inferring anything from this curve is a waste of time.
What do you propose we do with this curve? Never use them? Always use some other focusing aid?
In no way whatsoever am I doing this. I am just mathematically trying to determine what is the best estimate of the low point of the focus curve that a few distinct points imply.
With all the examples I have given, just by visually inspecting the plots, it is obvious that my routine has chosen a much better guess at the center than SGP has. In other words, I would much prefer moving the focuser to my quadratic routines choice than the SGP choice.
This is not rocket science. It is always going to be a best guess.
You and I are clearly not talking about the same thing.
Another thought has occurred to me about our differences here. You seem to think that I am saying that my routine PROVES that the best focus it comes up with is the best possible focus that can be derived from a simple plot of lets say 9 data points. I am NOT saying that. I am saying that visually, I think my routine’s choice of best focus is at least as good as SGP has chosen, and is enormously better in the many cases where SGP just gives ridiculous values, or gives values that are not VISUALLY optimum. There is no one best choice for most less than perfect curves.
My opening statement says it all. Please examine all the examples I gave and tell me if you disagree with this statement for the examples I provided. If so, please give details.
I claim it is the best focus because by definition it is best focus. I have a figure of merit and I turn a knob to the point where the knob is at the minimum of the curve. There is no ultimate truth to be found here because at step one we need to define what best focus means - and for me it is the point where measured fwhm is minimum.
I do a form of validation - which is to estimate the uncertainty in final focus AND final fwhm - and measure it after returning to the new focus. The final point should be near the minimum of the curve and have about the right fwhm - and indeed it does except for seeing variations in short exposures.
There is no need to collect more data because the model fit and appropriate error propagation give a statistically valid estimate of the uncertainty in the final result. You seem to think this is magic or cheating - but it is standard and rigorous. I am not using the model to say how well it fits the data. I am using the scatter in the data - and uncertainties in the measurements - to estimate corresponding uncertainties in the model parameters. If you search on experimental curve fitting, model parameters, and covariance you will find examples.
But in fact I do agree with you in one way - and that is the use of R^2 as a criterion. That IS a case of saying - look - this is a really good fit to the curve so the answer is good. All it does is give an overall score for the fit - but it doesn’t tell you how far off the answer might be - and it doesn’t let you compare one result with R^2=0.5 to another with R^2=0.9.
What I am talking about is different - and it is the model fit with statistical estimates of uncertainty in the derived quantities.
As for hyperbola vs. parabola - that is a non-issue. If the curve is wide enough that the hyperbolic shape is evident then it should be fit to hyperbola - but if it is close to focus then parabola will give a better result - in terms of uncertainty in the derived quantities.
In order to demonstrate that “thing 1” is better than “thing 2” you have to produce a data set with “thing 1” and then produce a data set with “thing 2” where both use the same source. Then the two data sets are processed by a third, uninterested party, to see which data set has the better results.
So a star field is picked and an auto focus run is done (or many) with the original SGP algorithm and the same star field with the new algorithm. Then something like PixInsight is used to evaluate the images for quality of focus. To be accurate, you probably need to do a couple of dozen auto focus runs with each algorithm and produce some kind focus quality average of each set.
The parabola fit looks better and especially more robust but this is subjective, and I agree with Chris that it is not technically proven to be better because you do not have FWHM or HFD at the best focus position (only the model-predicted value).
One way to settle this is to run the current SGP script and the parabola script on the same AF data points, then take an image at the focuser position predicted by each model.
Comparing the HFD values at predicted focus position will be a direct comparison of the focus quality.
This may be tricky because this has to be done in the field, and temperature drift cause be an issue.
Another way is to take AF data with very small step (I’m talking 100’s of points) then pick a few evenly spaced points (e.g. 9) to run both scripts.
Comparison can be made a posteriori since HFD data will be available near the predicted best focus possible of each method. This method can also give information on the precision and robustness of each method by selecting various subsets of points from the large dataset.
I’m not sure I follow. At first, you say that the figure of merit is the minimum of the curve. I presume this means the calculated minimum of the fitted parabola curve. Then you imply that the figure of merit is where the measured FWHM is minimum. It can’t be both.
All of these models - yours, jmacon’s and SGP’s models - apply a mathematical fit of some kind to interpolate a minimum. Both you and jmacon claim (hypothesize) that your models find a more accurate minimum than the SGP method. In order to prove that hypothesis, there must be some kind of rigorous validation of focus for both your models and SGP’s model before you can substantiate that hypothesis.
That’s good, and this is the first mention of any validation of the accuracy of your model that I recall seeing. However, it is not particularly rigorous to say that the resulting focus position has “about the right FWHM” particularly when you are making the claim that your method is more accurate than another method. Rigorous treatment would include a statistically significant number of focus runs that compares the predicted FWHM to the measured FWHM. There are numerous statistical techniques that will tell you how well your interpolated value correlates with the measured value. That’s one part that is missing.
The other part that is missing (if your hypothesis is that your model is more accurate then the SGP model) is a comparison between the accuracy of your model and the SGP model. I haven’t seen that.
I have a figure of merit that is a function of one parameter and I want to find the value of the parameter that minimizes that figure of merit. That is what I mean by ‘in focus.’ Nothing else is better or worse to me because it is the actual thing that I care about.
I assume my focusing system is well behaved so that with backlash compensation the mechanical location is repeatable and the x-axis values are effectively deterministic. This is a standard assumption in a least squares fit.
I assume my measurements in the y axis are very noisy - but I also have a meaningful estimate of that noise that I can use to weight the values. That is the standard deviation of the fwhm measurements in each frame.
I assume the measurements are close enough to the minimum that it is a shallow bowl and only needs a quadratic fit. This again is standard and amounts to a Taylor series expansion. Higher order terms would not help here since close to focus the noise in y only allows finding the quadratic term with any confidence - but that is sufficient to find the minimum and its uncertainties.
I also assume the location of best focus is slowly drifting - so I don’t have infinite time to make many measurements. I need to do a number of exposures quickly so that the drift of focus is small on the scale of the measurement uncertainty.
After I make the measurements I do a weighted least squares fit to find the terms of the polynomial and the covariance matrix. Those terms and the fit to the curve are precursor information to what I really need, which is the x value where the curve is minimized. So I can calculate that x value and its associated uncertainty by error propagation.
At this point what I have done is calculate a fit to the model and its associated error bars. I am not aware of anyone else finding focus and estimating the error in the result - but without that the measurement has no means for comparison. If people think I am somehow cheating or something - by doing error propagation and estimating uncertainty in a derived quantity with the covariance matrix - there are many sources and texts to learn how that is done.
If someone has an approach they feel is superior but it has no corresponding estimate of the uncertainty in its result - then I have no idea why they would think it is a good approach since they have no idea how far off the optimum it is.
HFR or FWHM should both work well as long as you are close to focus. There are other metrics that would work also - as long as they show a clear minimum.
I prefer fwhm in general because for defocused or aberrated stars, HFR becomes sensitive to how you determine the background level - if there is a lot of flux spread thinly away from the center. All these things are also implementation dependent. But close to focus everything is better behaved.
The HFR implementation in SGP seems fine near focus. And they shouldn’t need to deal with donut fits because there is no reason to be so far from focus - unless you are starting out far from focus. I just recommend also using the standard deviation of measurements as a weight in the fit - and to calculate uncertainties in the final result.
If you click on your avatar image in the upper right of this forum, you will see a little gray envelope icon. Click there and you will see the direct messages (DMs) I have sent you.
I think you’re making a good point. Implement the new focus routine and give us the option of picking which focus method we want to use. At that point, the users can run both routines as many times as they like and collect comparison data to determine which one produces the best, repeatable focus based on final HFR or FWHM (or whatever is determined to be the best method - maybe more than one) measurements taken after the routine(s) run.
Seems like we’re at the validation step of this process - push it out and let us run with it. May be a bit time consuming, but there are folks out there that are happy to participate in helping.