PHD2 disconnecting from SQP or failing recovery

Sequence Generator
1

I have been plagued with PHD2 failures with my latest hardware configuration. The newest hardware has been in place for about 4 weeks:
TeleVue NP127is, Starlight XPress AO+OAG(Lodestar X2), ZWO filter wheel, ZWO ASI 1600mm camera, fixed installation in my obs. Running SGP 2.6.0.24 and PHD2 2.6.4dev3.
Almost everything is working perfectly, autofocus perfect, mount and meridian flips perfect, guiding excellent, except the following:
I can only image 1 to 3 hours before PHD2 develops some problem and no longer responds to SGP commands. The AO aspect seems to be working nicely. My polar alignment is very good and allows the guide star to stay in the center third of the box generally for a 7 minute exposure. Corrections from box edge back to center work well on both sides of the meridian.
The attached SGP/PHD2 logs from Oct 12 show a failure after 9:08, then after 2:04, since I restarted PHD2 manually at 2:00.

The attached logs from Oct 11 show a failure after 10:28.

Any help will be greatly appreciated.

2

I took a look a the PHD2 debug log from 10/11.

PHD2 logs commands that come in from SGP and there was a gap between 22:21:16 and 06:50:59 when it looks like SGP closed the connection to PHD2 then immediately re-connected (and repeated the connect/disconnect 5 times in about 5 milliseconds.) The actions at 06:50:59 probably don’t matter as the issue manifested at 22:21:16 or shortly after.

I’m not sure what this means but maybe Ken or Jared will have some ideas?

22:21:10.448 00.526 5388 evsrv: cli 04CDBA30 request: {"method":"get_app_state","id":1001}
22:21:10.903 00.454 5388 evsrv: cli 04CDBA30 request: {"method":"get_app_state","id":1001}
22:21:12.014 00.540 5388 evsrv: cli 04CDBA30 request: {"method":"get_app_state","id":1001}
22:21:13.078 00.074 5388 evsrv: cli 04CDBA30 request: {"method":"get_app_state","id":1001}
22:21:14.193 01.115 5388 evsrv: cli 04CDBA30 request: {"method":"get_app_state","id":1001}
22:21:15.308 00.730 5388 evsrv: cli 04CDBA30 request: {"method":"get_app_state","id":1001}
22:21:15.408 00.100 5388 evsrv: cli 04CDBA30 request: {"method":"get_app_state","id":1001}
22:21:15.510 00.102 5388 evsrv: cli 04CDBA30 request: {"method":"dither","params":[0.5,false,{"pixels":2,"time":0,"timeout":600}],"id":1002}
22:21:15.561 00.049 5388 evsrv: cli 04CDBA30 request: {"method":"get_app_state","id":1001}
22:21:15.662 00.101 5388 evsrv: cli 04CDBA30 request: {"method":"guide","params":[{"pixels":2,"time":0,"timeout":600},false],"id":1003}
22:21:15.713 00.051 5388 evsrv: cli 04CDBA30 request: {"method":"get_app_state","id":1001}
22:21:16.124 00.000 5388 evsrv: {"Event":"SettleDone","Timestamp":1507782076.124,"Host":"DARKSTAROBS","Inst":1,"Status":0,"TotalFrames":1,"DroppedFrames":0}
06:50:59.390 00.000 5388 evsrv: cli 04CDBA30 request: {"method":"get_app_state","id":1001}
06:50:59.390 00.000 5388 evsrv: cli 04CDBA30 short write 0/48 Input / Output error
06:50:59.390 00.000 5388 evsrv: cli 04CDBA30 disconnect
3

Thanks Andy, for taking a look at this. It appears that on both nights, the gap is caused by the camera staying in image capture mode for the entire span of the gap.

22:27:16.662 00.000 5388 GuideStep: -0.1 px 0 ms EAST, 0.4 px 1 ms SOUTH
22:27:18.177 01.515 3988 Exposure complete
22:27:18.201 00.024 3988 worker thread done servicing request
06:50:59.351 30221.150 5388 OnExposeComplete: enter
06:50:59.351 00.000 5388 UpdateGuideState(): m_state=6
06:50:59.351 00.000 5388 Star::Find(15, 272, 118, 0, (0,0,0,0), 0.0, 0) frame 5337
06:50:59.351 00.000 5388 Star::Find returns 1 (0), X=272.31, Y=118.06, Mass=50282, SNR=156.2, Peak=4074 HFD=4.3

21:09:57.899 00.000 12236 Handling exposure in thread, d=1000 o=3 r=(257,106,31,31)
21:09:59.438 01.539 12236 Exposure complete
21:09:59.462 00.024 12236 worker thread done servicing request
01:58:42.859 17323.397 8576 OnExposeComplete: enter
01:58:42.859 00.000 8576 UpdateGuideState(): m_state=6
01:58:42.859 00.000 8576 Star::Find(15, 272, 120, 0, (0,0,0,0), 0.0, 0) frame 2636
01:58:42.859 00.000 8576 Star::Find returns 1 (0), X=271.94, Y=121.34, Mass=50700, SNR=157.5, Peak=4652 HFD=3.9

On the 11th, it was in expose mode for 30221.150 seconds.
Ob the 12th, it was in expose mode for 17323.397 seconds.
These times correspond exactly with the length of the gaps.
Am I correct in assuming that PHD2 will not respond to queries from SGP when it is in expose mode?

4

I made one change that appears to have resolved this issue. I changed the Titan communication protocol from USB to Ethernet. I have imaged successfully for the last 3 nights with no sign of this problem. There may be some error trapping somewhere that would allow this condition to exit gracefully,
Thanks for taking a look a this.

5

Unfortunately, the issue has not gone away. Last night after 2 hours of good imaging, PHD2 just froze again, apparently while taking an image, and was unresponsive to SGP which went into recovery mode. 4 hours later when I looked at it, I disconnected the reconnected PHD2. This caused PHD2 log entries to continue from that point and a restart of SGP allowed the session to continue to normal termination.
Logs:

6

I think I have gotten to the bottom of this.
I noticed that when PHD2 gets stuck taking an image, that no entries appear in the PHD2 debug log until I log back on to my obs computer, usually several hours later. This same thing occurs to the log I produce with my obs control program which normally is creating log entries every minute. However, there are continuous log entries for the entire period in the SGP log. I have discovered that SGP does not write any log entries to the disk until the program is closed.

This seemed to imply that my computer was going to sleep during the imaging session and the gaps in the PHD2 and Obs control logs are being caused by the computer going into sleep mode. Checking this, the computer was set to NEVER go to sleep. However, the hard drive power setting was power down after 20 minutes. Both the PHD2 log and my Obs log write to a disk file. PHD2 does it when it loses the guide star image so it writes an image file. A quick test today after 1.5 hours of being logged off the obs computer then logging back on, finds SGP and PHD2 still running. After tonight’s session I will know for sure.

A mystery how this power parameter got changed. Never used to be any issue.

7

I’ll be interested to hear your results @jmacon. I have experienced a similar problem but in my case PHD indicates that the camera has been disconnected. Do you receive a message like that?

8

I’ve experienced a few problems like this in the past even though all settings were set to prevent sleep etc.

I now use the MouseMove app which artificially moves the mouse cursor a little bit every now and then if there is no user input. Works very well.

9

I had a similar issue with my X2 lodestar - but it was occurring outside of SGP and so I never posted here. I switched drivers (from the latest on Starlight’s website to PHD2’s own) and it has never recurred. It seemed to hang during image download and froze everything up - but thankfully for me, I noticed it when I was looking at the PHD2 window

10

@joelshort the camera stays connected. My configuration change did not solve the problem. It quit at 1:09am with exact same symptoms: both PHD2 and my Obs log had gaps from 1:09am until 6:45am when I logged on to the obs pc. Then both logs continued and the Obs program proceeded to close the obs since it was light out then shut down. SGP continued posting log entries to its internal storage for the entire night. At 6:45 PHD2 looked to be running as usual, fully connected.

The config change I made was to change the Power parameter “Turn off hard disk after Setting (minutes):” from 20 to 0. Maybe I should have set it to 1200, which I have now done.

@Mike your suggestion sounds like the real solution. I have installed AutoMouseMove and will run with it on tonight. If that does not work @buzz I will try your suggestion of PHD2’s driver. Thanks guys, for all the help.

11

SUCCESS!. Last night’s run was perfect. Ran for 10.5 hours with 5 targets including meridian flips. Could not have been better.
I changed the hard drive turn off parameter from 0 to 1200 minutes. I added AutoMouseMover to move the mouse, and added a mouse move command to my Obs control program. I’m sure it only needs 1 of these, but I went for overkill.