SGP crashed last tonight whil running unattended, it seems the camera driver started throwing an error first (System.OutOfMemoryException: Insufficient memory to continue the execution of the program) and then over 1 hour later SGP finally crashed while basically being idle.
SGP is mostly likely not to blame for the camera error, but i think SGP should have a way to handle a problem like this, reconnect camera and if that doesn’t work simply park the observatory.
I will watch next the memory usage to see if i memory is the actual cause for the driver problem, maybe a memory leak in the driver?
Link to Logs
Useful Info
OS: Microsoft Windows 10 Pro Ver: 3.1.0.446 .NET: 4.8
SGPro never lost connection with the camera, but every time it asked for the image data the driver indicated it was out of memory.
When recovery is done, end of sequence actions will be run.
This camera exception is misleading (not QHYs fault… it comes from the OS). This typically means that there does not exist sufficient contiguous space on the hard drive to store an image. Of course, this is typically caused by near full hard drives, faulty hardware or, in the case of older spinning drives, severe fragmentation.
Crashes like the one here (where no last message was left) are not preventable by SGPro because they have occurred too far outside of the .NET run time. That said, we can see what the issue was because these crashes do leave an entry in the Windows Event Viewer. You can start that up and then go to the time of the crash to see if SGPro has an entry there. If you can post that here, we may be able to prevent doing a thing when certain conditions exist.
SSD has lots of space so it’s definitely not a problem.
I belive it’s a memory leak either in the driver or somewhere else, might not have anything to do with SGP.
I did actually get an error message from windows about high memory usage 30 min ago and memory usage was around 60% (of 16gb) and regular memory usage is around 15-20%.
I will continue monitor the memory usage in the future.
Here’s the 2 messages:
Application: Sequence Generator.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: exception code e06d7363, exception address 766535D2
Programnavn med feil: Sequence Generator.exe, versjon: 3.1.0.446, tidsangivelse: 0x5e2257db
Modulnavn med feil: KERNELBASE.dll, versjon: 6.2.18362.535, tidsangivelse: 0x5bd9df62
Unntakskode: 0xe06d7363
Feilforskyvning: 0x001135d2
Feil prosess-ID: 0x%9
Feil starttid for program: 0x%10
Feil programbane: %11
Feil modulbane: %12
Rapport-ID: %13
Fullstendig navn på feilpakke: %14
Relativ program-ID for feilpakke: %15
The only real way around something like this that i can see is a small server application connecting directly to the equipment that would handle some basic safety functionality in case of SGP crash.
Maybe… kernel level crashes are very difficult to recover from. This is a thing we have had on the to-do list for a while… a sort of “watchdog” that is so simple, the likelihood of software defects or crashes remains low. BUT… this little watchdog will have the same limitations as SGPro. It, at some point, needs to invoke code from drivers and if those drivers crash in the kernel, the watchdog will also just stop working.
