QUADRATIC best fit Focus routine TESTING - SEND LOGS

The parabola fit looks better and especially more robust but this is subjective, and I agree with Chris that it is not technically proven to be better because you do not have FWHM or HFD at the best focus position (only the model-predicted value).

One way to settle this is to run the current SGP script and the parabola script on the same AF data points, then take an image at the focuser position predicted by each model.
Comparing the HFD values at predicted focus position will be a direct comparison of the focus quality.
This may be tricky because this has to be done in the field, and temperature drift cause be an issue.

Another way is to take AF data with very small step (I’m talking 100’s of points) then pick a few evenly spaced points (e.g. 9) to run both scripts.
Comparison can be made a posteriori since HFD data will be available near the predicted best focus possible of each method. This method can also give information on the precision and robustness of each method by selecting various subsets of points from the large dataset.

I’m not sure I follow. At first, you say that the figure of merit is the minimum of the curve. I presume this means the calculated minimum of the fitted parabola curve. Then you imply that the figure of merit is where the measured FWHM is minimum. It can’t be both.

All of these models - yours, jmacon’s and SGP’s models - apply a mathematical fit of some kind to interpolate a minimum. Both you and jmacon claim (hypothesize) that your models find a more accurate minimum than the SGP method. In order to prove that hypothesis, there must be some kind of rigorous validation of focus for both your models and SGP’s model before you can substantiate that hypothesis.

That’s good, and this is the first mention of any validation of the accuracy of your model that I recall seeing. However, it is not particularly rigorous to say that the resulting focus position has “about the right FWHM” particularly when you are making the claim that your method is more accurate than another method. Rigorous treatment would include a statistically significant number of focus runs that compares the predicted FWHM to the measured FWHM. There are numerous statistical techniques that will tell you how well your interpolated value correlates with the measured value. That’s one part that is missing.

The other part that is missing (if your hypothesis is that your model is more accurate then the SGP model) is a comparison between the accuracy of your model and the SGP model. I haven’t seen that.

Tim

@jmacon I sent you my email address in a DM

I have a figure of merit that is a function of one parameter and I want to find the value of the parameter that minimizes that figure of merit. That is what I mean by ‘in focus.’ Nothing else is better or worse to me because it is the actual thing that I care about.

I assume my focusing system is well behaved so that with backlash compensation the mechanical location is repeatable and the x-axis values are effectively deterministic. This is a standard assumption in a least squares fit.

I assume my measurements in the y axis are very noisy - but I also have a meaningful estimate of that noise that I can use to weight the values. That is the standard deviation of the fwhm measurements in each frame.

I assume the measurements are close enough to the minimum that it is a shallow bowl and only needs a quadratic fit. This again is standard and amounts to a Taylor series expansion. Higher order terms would not help here since close to focus the noise in y only allows finding the quadratic term with any confidence - but that is sufficient to find the minimum and its uncertainties.

I also assume the location of best focus is slowly drifting - so I don’t have infinite time to make many measurements. I need to do a number of exposures quickly so that the drift of focus is small on the scale of the measurement uncertainty.

After I make the measurements I do a weighted least squares fit to find the terms of the polynomial and the covariance matrix. Those terms and the fit to the curve are precursor information to what I really need, which is the x value where the curve is minimized. So I can calculate that x value and its associated uncertainty by error propagation.

At this point what I have done is calculate a fit to the model and its associated error bars. I am not aware of anyone else finding focus and estimating the error in the result - but without that the measurement has no means for comparison. If people think I am somehow cheating or something - by doing error propagation and estimating uncertainty in a derived quantity with the covariance matrix - there are many sources and texts to learn how that is done.

If someone has an approach they feel is superior but it has no corresponding estimate of the uncertainty in its result - then I have no idea why they would think it is a good approach since they have no idea how far off the optimum it is.

Frank

Okay I’ll bite… what is your theory for FWHM being the correct measurement for an unfocussed star, as opposed to HFR (Half-flux radius)?

-Ray

HFR or FWHM should both work well as long as you are close to focus. There are other metrics that would work also - as long as they show a clear minimum.

I prefer fwhm in general because for defocused or aberrated stars, HFR becomes sensitive to how you determine the background level - if there is a lot of flux spread thinly away from the center. All these things are also implementation dependent. But close to focus everything is better behaved.

The HFR implementation in SGP seems fine near focus. And they shouldn’t need to deal with donut fits because there is no reason to be so far from focus - unless you are starting out far from focus. I just recommend also using the standard deviation of measurements as a weight in the fit - and to calculate uncertainties in the final result.

Frank

@jmacon

If you click on your avatar image in the upper right of this forum, you will see a little gray envelope icon. Click there and you will see the direct messages (DMs) I have sent you.

I think you’re making a good point. Implement the new focus routine and give us the option of picking which focus method we want to use. At that point, the users can run both routines as many times as they like and collect comparison data to determine which one produces the best, repeatable focus based on final HFR or FWHM (or whatever is determined to be the best method - maybe more than one) measurements taken after the routine(s) run.

Seems like we’re at the validation step of this process - push it out and let us run with it. May be a bit time consuming, but there are folks out there that are happy to participate in helping.

3 Likes

I would love to be able to help test a new focus routine

Send me all the logs you have saved and I will send you some results that you can validate your runs against.

Finally, after 6 weeks of intense effort by all involved, we have released our Quadratic Best Fit Focus routine. Ken is now evaluating it.

Those working on this project include @bulrichl, @mikaelA, @jmacon, and an early recruit Nicolàs de Hilster.

We have developed a system that allows us to compare our new routine with the current routine in SGP. I should be clear about what it does and does not do. It has nothing to do with star detection. It is the routine that evaluates the focus data comprising the set of Focuser Positions vs average HFR for each image to decide what the new best focus position should be.
To demonstrate the improvement our routine provides we have plotted graphs of Temperature versus Focuser Position, comparing what SGP produces versus our new Quadratic routine. To visually observe how the routine is operating, we have produced graphs of all 3233 focus runs that our log contributors have provided. An example:
FORUM1 FORUM2

This example illustrates one of the kinds of focus curves that SGP has difficulty with. The quadratic curve models this curve nicely. The red X shows a data point that the routine decided was bad data and ignored. The purple down arrow centered on the quadratic curve is the best focus determined by the quadratic routine.

The following charts display the results of a single night of taking continuous auto focus runs with my Televue NP127is refractor, a total of 154 focus runs. The first graph plots the SGP focus positions vs Temperature, the second plots the Quadratic Routine focus position vs Temperature. Quadratic clearly produces superior results.


The routine has evolved through many iterations over the past 6 weeks and has been thoroughly tested on a wide range of hardware, focus settings, observing conditions, and focus curve quality.

2 Likes

In Proposal of a "Quadratic Fit" Auto Focus evaluation method - Auto Focus - Main Sequence Software I suggested Quadratic Fit (QF) as an alternative method for the evaluation of best focus from Auto focus data. This is the base of Jerry’s application.

For the comparison of best focus results obtained from different evaluation methods an objective measure is needed to judge which one is superior. In the thread cited above, a simple method for this judgement was presented as well:
For both of the methods best focus vs. temperature (T) charts were plotted, a linear regression was made, and the temperature coefficient c (= negative slope of the fitting line) and the coefficient of determination, R^2, were calculated. The better method for evaluating best focus is the one which results in a better correlation of best focus with temperature, i.e. which results in the larger R^2.

In the course of our cooperation I developed a spreadsheet that is capable of processing the results of hundreds of AF runs at one blow. It calculates, displays and exports as graphic the quadratic fitting curve from the AF data of each AF run. Furthermore it displays the best focus vs. T plot for the whole data set, both for SGP’s and QF’s best focus results. The comparison of the best focus vs. T plots and the comparison of the calculated R^2 immediately shows which method is superior. The graphics Jerry presented are generated with this spreadsheet.

The underlying data were extracted from the SGP logfiles by Mikael’s (mikaelA) SGP AutoFocus LogViewer - Auto Focus - Main Sequence Software .

Actually the improvement of the best focus values that Jerry demonstrated above is such that a previously unvisible feature of his used instrument (Televue NP 127is) is revealed: the temperature coefficient is temperature dependent. At higher temperature, the temperature coefficient is larger than at lower temperature. This dependence is better described by a polynome of grade 2. Therefore I append one additional chart [3] that complements the charts presented by Jerry above:

[1] SGP: best focus vs. T, linear regression, R^2 = 0.9104, c = -5.2
[2] Quadratic: best focus vs. T, linear regression, R^2 = 0.9865, c = -5.2
[3] Quadratic: best focus vs. T, polynomial regression grade 2, R^2 = 0.9933, c: temperature dependent, see below

For case [3] the polynome is calculated as:
f(T) = 0.0905 T^2 - 4.90 T + 25365.2
The first derivative which gives the temperature coefficient then is:
f’(T) = 0.181 T - 4.90
(Important: T is specified in degrees F!)

This function must not be applied beyond the upper and lower bounds of the measured data, so the function is valid only in the temperature range from 48 to 64 degrees F. The change of the temperature coefficient is like this:

T [°F]	c [steps/°F]
  48       -3.8
  50       -4.2
  52       -4.5
  54       -4.9
  56       -5.2
  58       -5.6
  60       -6.0
  62       -6.3
  64       -6.7

best_focus_vs_T_JM_20190917_Quadratic_polynomial
I guess this is a convincing example for the superiority of QF to SGP’s current method.

Bernd

1 Like

The improvement on the temperature versus focus plot is very impressive and gives credence to the quadratic fit method.

It is mentioned that this has been tested over a large range of hardware. Is the improvement on the temperature plot significant over all cases, or it is marginal for some configurations or settings?

Another issue in general with autofocus is the imaging time wasted. I think there would be value in minimizing the number of AF points needed. What’s your thoughts on this? Does the benefit of QF comes at the expense of more points needed?

On the other hand the QF AF scan range can be smaller since there is no need to go in the linear region of the V-curve, so the time the focuser is moving is reduced.

Anyway, I am looking forward to having the QF focus as an option in SGP.

@MARA25 you make some good points. The benefit does extend to all the configurations we had logs for. However, the benefit is proportional to the quality of your curves. If your curves are really good, just about any method gives good results, the bottom of a sharp V curve is easy to find. The more problematic curves you have, perhaps with a little irregularity around the bottom, that is where the quadratic improves the results dramatically. Choppy bottoms have little effect on the center point of a fitted quadratic curve. You will be able to get good results with fewer data points.

Guys. I’m really impressed. When I next have my RCT on the PMX, I will store some logs. I know from the past that I get asymmetrical curves when it’s slightly out of collimating.
Is it best to give images or use the focus log extractor that was linked to an earlier post of bulrichl?

Hi Chris,

images or the extract generated by SGP AF LogViewer will not be sufficient, we would need the SGP logfiles. If you are interested to have your data evaluated, please specify a link to the corresponding SGP logfiles.

For the next 14 days Jerry is travelling in Europe, so you can send the link to me via personal message. Then I will send back a report containing the results (text file and charts).

Bernd

Thanks Bernd - no rush - I have to finish the squid neb first before I swap over to the RCT. Most online images have 20h +, just for OIII.

Hi Jerry, only just now saw your post, thanks for including me, much appreciated!

For those not directly involved: I have only suggested to implement a robust detection of outliers based on the Mean Absolute Deviation (MAD) algorithm, which seems to do its job quite well on the focus runs. In addition I supplied a number of logfiles, verified some of the calculations done, and shared my thoughts.

So most of the real work has been done by the rest of the team, but I am glad my input is appreciated.

Nicolàs

PS: today we had confirmation that Jared is working on the implementation, which he hopes to test later this evening.

My first try with the new autofocus routine!
image

The autofocus log viewer does unfortunately not handle the new routine without modifications

Just wanted to thank Jerry M, Jared and Ken for adding this focusing routine…I meant to write this several months ago when I did the upgrade with this new focus feature. For me, it works flawlessly and seems quicker. A few years ago, I saw this video that helped me with my settings which translated to this knew routine seamlessly. It is a two part youtube video, Sequence generator Pro - Part 1 of 2 Setting up auto focus - YouTube