Last night my run aborted at 00:32, even though I have Recovery set for every 5 minutes for 3 hours.
Here is what happened:
PHD2 lost the guide star
SGP went into recovery mode
Centering process failed after 3 Plate solves were not within my 30 pixel requirement.
[08/17/19 00:32:56.226][DEBUG] [Center Scope Thread] Unable to achieve results below allowable error (30 px) in 3 attempts!
SGP then terminates the recovery process. So I lost the rest of the night’s imaging.
This is not welcome behavior. SGP should continue to try recovery for 3 hours.
Recovery was aborted as you tracked through the meridian. This is one area where we just shutdown as to not continue tracking too far and damaging equipment:
[08/17/19 00:38:07.682][DEBUG] [Recovery Sequence Thread] Something bad has happened... attempting to recover the sequence (attempt 3)...
[08/17/19 00:38:07.682][DEBUG] [Recovery Sequence Thread] Sequence recovery aborted because the telescope has reached zenith before a recovery could occur (this can also occur of the telescope disconnects), aborting for safety...
[08/17/19 00:38:07.683][DEBUG] [Recovery Sequence Thread] Closing recovery dialog...
[08/17/19 00:38:07.939][DEBUG] [Sequence Thread] Sequence recovery failed (CenteringAndGuiding)!
[08/17/19 00:38:07.940][DEBUG] [Main Thread] Adding sequence level notification: Sequence recovery failed (CenteringAndGuiding)!
[08/17/19 00:38:07.997][DEBUG] [Sequence Thread] Sending Notification: Error - Sequence recovery failed (CenteringAndGuiding)!
[08/17/19 00:38:07.997][DEBUG] [Sequence Thread] Guide star lost! Aborting sequence.
Jared
It was 2 hours away from meridian flip. What’s different from PHD2 can’t yet find a guide star and we wait another 5 minutes, then try again? This looks like a special case for shutting down that is not valid.
Transit time for my target was 12:43, plus I track past meridian 70 minutes. I think your meridian logic must be flawed.
It would be really nice to implement something you mentioned in another thread, perform a meridian flip then continue recovery.
The mount had done a meridian flip. There is a slew at about 00:30 which does a flip. It’s possible to tell because the subsequent successful plate solve shows the image rotated by 180 degrees.
But SGP didn’t do the slew as a meridian flip and doesn’t know this.
The centering failed because it ran out of attempts because the first attempt failed when the solve failed before the flip. The two remaining attempts were’t enough.
What I’d do is increase the number of centering attempts. If the centering succeeds it takes no longer and if there are problems it gives more of a chance for the system to recover.
I’d also be curious about why both PHD2 lost the stars and so did the initial plate solve attempt. It looks as if the whole imaging system was unable to see any stars until after a meridian flip.
This could be an inconvieniently timed cloud or could it be something else? Are there any images from around the time of the failure?
For example if a dome were to change its azimuth assuming the mount had flipped when it hadn’t then the scopes would be looking at the inside of the dome. Th
Chris, you are right on here. I think what actually happened is the following:
normally the mount would flip at 1:53, which is 12:43+70 minutes.
however, guide star lost, on reattempt SGP forces a plate solve with does a slew to the coordinates
the fresh goto causes the mount to do a flip because it is in the range where it can flip and continue
SGP does not know anything about my dome, so it does not know that my dome has not yet reached the final destination
this causes the initial plate solve to fail, and 2 remaining tries are not enough.
This is a regular occurrence for me under normal operations, and is never a problem for SGP. It tries again in 1 minute and then always works.
SGP won’t continue the recovery
So we still have the problem that SGP won’t honor my request to try recovery for 3 hours, even though my mount is pointing to the zenith. It didn’t even try recovery for more than 10 minutes. SGP knows where my mount is pointing so it should not be concerned about injuring hardware. It also knows that I expect this target to be tracked and imaged until 5:15 which would certainly be safe for my hardware.
So you do have a dome, and SGP knows nothing about it.
How is your dome azimuth controlled? It needs to know which side of the pier the scope is on. What software controls the dome Azimuth?
If the dome control software believes that the mount pointing state has changed from West to East it will move in aimuth so the scope can see out when it has changed the pointing state. If the way the pointing state is determined sometimes gives the wrong answer then the dome can be moved so the scope cannot see out.
The mount has not changed pointing state so when this happens the telescope can no longer see out. First the guide star is lost because the guide camera can’t see the stars because the dome is in the way then the plate solve fails for the same reason. This is before any scope slew is done.
The dome may have moved independently of the scope.
It’s only after the slew after the first solve failure that the scope and dome are aligned and there’s then no problem, but there aren’t enough align attempts available.
How is the dome controlled?
Where does it get the side of pier status of the mount?
I don’t think you can expect SGP to manage this because it knows nothing about the dome.
What’s needed is that the dome controller correctly reflects the actual mount pier side so the mount can see out.
This is not an issue because of how it is controlled. The dome is a HomeDome controlled by the Digital Dome Works Ascom driver.
SGP controls the mount (Paramount MEII)
Digital Dome Works follows the mount
So SGP has no idea I have a dome, it just controls my mount.
Wherever the mount goes, my dome tracks it perfectly. I have written a master control program which monitors the weather and controls the operation of the dome. At the start of the evening, if the weather is clear, it opens the dome and optionally starts SGP. Usually I start SGP manually and do some initial focus runs. Whenever the weather becomes overcast or rain starts, the master control program closes up the dome. When the weather clears, it opens the dome again. Come dawn it closes the dome, shuts down and closes all programs, shuts off all 8 power outlets.
I have chosen to not interface the dome to SGP because I like having separate and complete control over the dome. Not that I don’t trust SGP, but…
The only tiny issue as I mentioned above, is SGP may start a plate solve before the dome has rotated to the final position. I can increase the settable time delay, but that time delay will apply for every slew, including the very small ones, so I don’t want it very large.
Or just increase the number of tries. In normal operation if 3 tries is not enough, the 1 minute delay before another attempt is not significant. So your idea is to set the tries to more than 3. I have a funny little quirk with my mount and/or SGP control of it: If 2 or 3 tries does not get there, many more won’t do any better. Letting it quit and start over 1 minute later always works with 1 slew to better than 5 pixels.
That’s why I use 3.
This whole setup has worked flawlessly for the past 3 years, night after night of perfect imaging. Unless the clouds come in, and SGP won’t really do recovery for 3 hours (or even 20 minutes).
I guess that what you do is read the Ra and Dec from the mount and use that to determine the dome aimuth.
For a GEM the dome azimuth required will also depend on the side of the pier the mount is on, how do you determine the pier side? Do you read the SideOfPier property from the mount?
The reason I’m wondering about this is because if you are using hour angle then the dome will think the mount has changed pier side when the mount tracks past the meridian and this will move the dome so the scopes can’t see out so giving the initial lost guide star and failed to solve errors.
You should be able to tell if you have logs that show the dome aimuth, you will see a relatively large move of the dome at about the time the guide star was lost. Or set up the scope and dome so the mount is pointing a little to the east, with the scope on the West and see what happes as it tracks across the meridian. The dome should continue to be in the correct position for the scope to see out.
BTW the mount and SGP will have different ideas of the Ra and hour angle to the scope because the SGP sync is done in SGP using the offset method. They are using different Ra values. Minor differences in time or location will also affect the Ha calculation.
As for using 4 or 5 tries the initial centering shows errors of 2036, 162.5, 69.5, then 3.1 so successful.
The failed centering shows a failed solve, then 4062, 61.5 and a failure at 48 pixels. The impression I get is that another attempt could have worked.
The scope centering delay could be avoided by using a scope/dome hub which provides the scope driver for SGP. Scope slew commands are intercepted and used to start the dome slewing to where the scope will be. The scope IsSlewing property only returns true when both the scope and dome have finished moving.
I think it’s worth getting to the root of the reason the initial guiding and solve failed. Or was it really a cloud that coincided with the scope reaching the meridian.
Excellent feedback Chris, I really appreciate it. It would like to get this resolved.
This is all controlled by the Digital Dome Works Ascom driver. I don’t have any control over it. DDW follows the mount by reading this file created by the mount software: “Telescope Position.txt”.
Which at the moment contains:
RA=20h 11m 54.42s DEC=-00° 33’ 35.91" AZ=277° 34’ 34.06" ALT=-11° 01’ 21.6"
I don’t see any side of pier info here, so you may be right. Should I select the “Use Ascom Telescope” choice?
However, many times over the last three years I have been watching the operation for several hours, sometimes a flip occurs, and the dome has always rotated 180 degrees. It does not know where the mount is going to end up. It just follows it as it does the slew, and always ends up with the 180 rotate. Regardless of whether or not the dome knows anything about side of pier, it has always ended up where it should be, at least that is true every night that I image for the initial slew, and any other slews I initiate. And many nights with up to 4 different targets and a couple of flips, it has worked flawlessly. I have never seen it in the wrong position.
I think to really resolve this I should record the entire night’s video from my camera on top of the upper scope. This camera lets me monitor exactly where the dome is. That’s how I know it has been going where it should when I am watching. I will do this.
This time it did work. Sometimes it does and other times not. It can just as easily stay around 60 pixels indefinitely. I used to have it set at 6 assuming that would always work, but it didn’t. I have the settle delay set to 30 seconds so that is an extra 90 seconds, plus the time the routine takes for each try, it comes out to several extra minutes. It is a trade off. Until I resolve this I have set it to 4.
My cloud monitor showed clear the entire night, but that does not guarantee a small cloud did not appear in my FOV. The cloud monitor only sees part of the sky.
If I implemented this, would I still be able to control the dome with my Master Control Program?
That looks as if there is no side of pier information so the dome probably uses the hour angle to infer the side of pier. This will be incorrect when the mount tracks across the meridian.
I don’t know if the DDW driver will cope if it uses the ASCOM driver or if the SB ASCOM driver reports the pier side correctly. You will need to test this. It should be pretty easy to test without imain and even in daylight by pointing the mount at a position 15 to 30 miuntes before the meridian and leaving the mount to track through the meridian.You should not see a major change in the dome position as it is tracking and the scope should continue to be able to see out of the dome slit.
Ok, I just finished running your recommended procedure Chris. I slewed within 10 minutes short of Meridian, scope on West side of pier, flip set for 10 minutes past Meridian. SGP running taking images, connected to mount, no knowledge of dome. Dome running under Digital Dome Works, tracking perfectly with shutter lined up perfectly on scope.
20 minutes later, SGP initiates flip, mount slews. Dome immediate starts following the mount, target AZ constantly changing because mount is still moving. 1.5 minutes later mount settles on other side of pier, dome has followed it closely and now stops at perfect centering on scope, about 100 degrees from original AZ. Altitude was about 45 degrees so it did not need a full rotation.
DDW dome control software definitely knows what side of the pier the mount is on.
Here is what may well have happened, assuming no small cloud happened in my FOV. My target was Sh2-114 Flying Dragon Nebula, crosses meridian at 12:43 at an altitude of 88 degrees. Smack overhead. This is the most challenging for my dome/scopes because the dome must rotate 180 degrees. The real issue is the shutter only opens past the meridian about 12 inches which provides a nominal opening along the RA axis of 24", which is a lot less than the 32" shutter width. My rig with 2 scopes, a 12" newt and 5" refractor, has an image circle of 20", not a lot of leeway.
Wonder where it’s getting the pier side info from, it isn’t in the file you mentioned.
The dome movement when tracking an object through the zenith isn’t intuitive. Initially, when the mount is significantly before the meridian, the line of sight of the scope will be looking to the East but then, before the scope reaches the meridian the line of sight will pass close to the zenith and go to the West. At that point I would expect that the dome will rapidly slew from East to West. The mount hasn’t changed pier side. Later the mount will reach the zenith and cross the meridian. Then a slew will change the pier side. Initially if the point is close to the meridian the line of sight of the scope will go through the dome to the East. If the dome is following this it will then move back from West to East. The mount continues tracing and once again the sight point on the dome will pass near the zenith from East to west and the dome will slew fromEast to West.
When the scope is following an object tracking through the zenith then with a GEM and the scope on the west side of the mount initially looking East the point on the dome that the scope is looking at will move to the West before the mount reaches the meridian. The dome should move so this is centered so there shoud be a large dome movement from East to West.
So the dome will:
Start looking East
Move to looking West before the pier flip.
Slew from West to East with the pier flip.
Move from east to West after the pier flip.
End up looking west.
But this may not matter too much because your point that the shutter opening past the zenith is small may mean that when the mount is looking close to the zenith there could be a pathological case where the scope is obstructed by the small shutter opening past the zenith. Tiny errors in the scope-dome geometry could have a big effect here.
I think I may have misled us here a bit. The DDW software is used to set some of the dome parameters, including the one that points to the Telescope Position.txt file. But I actually run their ASCOM driver for controlling the dome during an imaging session. The ASCOM driver is presumably getting the side of pier info.
I spent a couple of hours doing a couple of flips this afternoon. The dome motion tracked just like you said it should. It kept the shutter positioned where it should through both flips and motion on either side. Looks like its working ok for me. I imaged Sh2-132 Lion Emission Nebula last night from 11:00pm to 5:15am and it did just fine. It transits at 70 degrees. The 12" Newton had a slight loss of star count around the meridian, but the 5" refractor on top got nothing for 1.5 hours on each side of the meridian. Looks like I need to avoid imaging around the meridian for high targets if I want both scopes to give me images. Oh well, if isn’t one problem its another in this hobby.
I don’t know the DDW software at all but if it’s responding as I described then it must be getting the mount pier side from somewhere, possibly the scope’s ASCOM driver.
However looking at the DDW manual the only method mentioned is a LX200 control and there is a screenshot of the setup which has a scope flips at meridian option. This will be wrong for any significant amount of tracking past the meridian because the DDW software will assume that a flip has happened when it hasn’t. Delaying the flip for 70 minutes would show this.
However the DDW manual also has mention of ASCOM so it may be that their software is more up to date than the manual. That woud imply that the DDW software connects to TheSky using an ASCOM driver as well as SGP connecting to DDW through the DDW ASCOM driver and to TheSky through a second instance of the TheSky ASCOM driver.
This all looks like things that only you can sort out with the assistance of the DDW people. SGP isn’t involved.
I only run the ASCOM driver during imaging sessions, so DDW is not involved. I only use DDW for, presumably, initial setup. Yesterday I changed the DDW setting to use ASCOM instead of the TXT file. I really doubt this makes any difference since I am running the ASCOM driver directly. The setup page it includes has all the critical specifications for the dome. I suspect nothing I do in DDW has any effect on my operation when I control the dome with ASCOM.
