Memory problem with ASI6200

Description

After using the ASI6200 without problems for weeks it suddely lost 50% of images due to camera error last night.

This is the error message
[02/24/20 00:09:45.457][DEBUG][Camera Thread][SQ;CC;] ASCOM Camera Error : CheckDotNetExceptions ASCOM.ASICamera2.Camera ImageArray Get System.OutOfMemoryException: Insufficient memory to continue the execution of the program. (See Inner Exception for details) (System.OutOfMemoryException: Insufficient memory to continue the execution of the program.)

The pc has 16GB ram and ram usage was low.
I think SGP should throw an error about this onscreen, but continue the sequence like it did, i ran fine all night, but a lot of images was of course lost.
An automatic disconnect/reconnect camera driver might be enough to get rid of the problem.

I will of course also report the problems to ZWO, just hoping for some better error handling in SGP that would tell the user about the problem and possibly a quick fix with automatic reconnect.

Link to Logs

Useful Info

OS: Microsoft Windows 10 Pro
Ver: 3.1.0.454
.NET: 4.8

It looks like this might actually a problem with SGP, it’s a 32bit software so memory usage is limited…
Just testing capturing some images i can see memory usage jumps by 1GB+ just capturing an image.

Memory usage right after opening SGP: 56MB
After connecting the camera: 180MB
After taking 1 frame & focus image, which didn’t even download, it usually doesn’t download the first time: 420MB
Capturing the 2nd F&F image (which actually downloaded): 1535MB
3rd F&F: 1780MB
4th F&F: jumps to 2370MB then lowers a little.
Disconnect of camera, 2142MB before and 1442MB after.
Closing the F&F image: goes to 1029MB

All through this only 1 image were open.
Since memory usage is still at 1029MB after both disconnecting the camera and closing the open image i wonder if there is a memory leak?
The high memory use is also true with a brand new profile with just the camera added to equipment.

Opening fits files from the same camera i see that they are using like 150-200MB each, this means SGP can fill the 4GB memory limit pretty easily.

To help finding a solution it’s possible to allow remote control of the pc/gear to the SGP devs for testing.

For this kind of error, it is unlikely. This issue is misleading… it does not really have anything to do with RAM. It is that Windows is unable to find a contiguous 32-bit addressable spot in memory to place the image.

On screen errors are not of much use in an application designed for unattended operation. This is why we emit error notifications. While the message is not currently specific, it did note that there was an error while capturing an image each time.

No automatic disconnect/reconnect will of course not help at all since SGP’s memory handling seems to be the problem.
I will probably be forced to switch to a different software with better memory handling if a solution isn’t found :frowning:

64-bit support is mostly done.

1 Like

64-bit support is nice, but it’s a problem that many drivers doesn’t support 64-bit

That may be true, but that is not specifically an SGPro issue. Btw… I’m pretty sure the ASI drivers ARE 64-bit compatible. This should clear the issue you are seeing.

So, the issue isn’t really SGP but the camera drivers, correct?
As a person who is planning to get the ASI6200MM I’m quite interested in this as well. We need to petition ZWO to write 64 bit drivers, or will that not solve this problem? IDK, just asking.

Does ASCOM support 64bit?

Doing a little testing with competing 64-bit software i found that all drivers with support was able to connect, ZWO, 10 micron, Optec Feathertouch. So they must be 32/64bit universal :grinning:

Some gear is still usupported in that software so i couldn’t test them.
Dome, flat panel, safety driver and environment driver.

There is definitely some weird things going on with memory, memory use varies a lot in SGP like described in my 2nd post here.
I would love if one of you guys could take look at it for the system to see if anything could be fixed easily.

Yes, that’s pretty typical of C# applications which utilize garbage collection to reclaim space. You’ll certainly see the memory footprint jump up after an image or two and the OS may not reclaim this memory for a bit. It’s not an issue and is almost certainly a red herring to whatever behavior you’re seeing. Things don’t get to be an issue until you see an out of memory exception…if you haven’t seen this, then it’s unlikely to be a memory allocation problem.

Thanks,
Jared

1 Like

Taking just 1 image memory usage jumps to 1-1.5gb.
As you can see in my first post i did actually have out of memory exception.
I lost 50% of a series of 4 filters with 30x120s exposures.
Which images got lost was random, i only got 10/30 green.

My bad, there are a couple of different levels of exceptions. That particular one was thrown from the ASCOM driver and is not specific to the allocation within SGP. Basically the ASCOM driver ran out of memory, something we do not control. Depending on how the ascom driver is written you likely have a separate process for it in your Process Explorer or it may run under the SGP host process (you’ll see multiple SGP processes but one is SGP and the other is the ASCOM driver). Most drivers run under their own process. So you should see a ASCOM.ASICamera2 process in process explorer and watch its allocation.

But yes, we could potentially handle this better.

Right, and is probably released a little thereafter. This is pretty typical C# Garbage Collection behavior. SGP has a base size of memory that it will typically use. Taking and displaying an image will require some amount of memory. Taking and displaying another image will take some additional amount of memory. At which point you may see the memory allocation for both of those images even though SGP has released the memory for the first, but the runtime hasn’t freed it. Continue taking images and at some point the runtime will come along and say “Welp, I need to clean up some unused memory…oh here’s some that SGP isn’t using” and it will reclaim it. Basically you’ll be doing nothing for a while and the memory allocation will just drop. Or you’ll need to allocate more and the runtime will clean up as it’s allocating additional memory. It’s not like C++ where you have to meticulously manage every bit of memory you use.

That is correct. Using the same 32bit ASCOM driver on basically any platform would likely end up in the same results. As for SGP memory allocation and access we have people very successfully using QHY600s that aren’t having this issue (just illustrating similar sized chips and raw image size). The main advantage that a 64bit SGP would have is that it would then allow you to access the 64bit ASI driver … which also probably has the same memory issue but it would at least take longer for you to notice it. But SGP doesn’t have a need to be 64bit.

Jared

I can’t seem to find any process for the driver itself, but i did try something here.

I upgraded the native driver to get ASI6200MM support and by running continouus frame & focus i see a lot lower memory use with the native driver compared to the Ascom driver.
I did throw some errors in the logfile so it’s probably not working as it should, i also got it to freeze up (image never downloaded, stuck in aborting image)

Native drivers doesn’t seem to go much above 1000MB, Ascom driver goes up to at least 2700MB memory usage.

I believe the easiest way to fix this problem right now is to update and support the native drivers for the time being.
I don’t like this approach because it’s not what the Ascom standard is about, but it’s either ZWO fixing the Ascom driver or SGP supporting the native driver and i’m not sure how likely it is to see ZWO fixing the Ascom driver :frowning:

I would prefer to devote that time to completing the 64bit implementation rather than adding to the native implementation. I’m unsure if 64bit will resolve the issue or if it will just push it off until the driver consumes even more memory. The real fix can only be implemented in the ASCOM driver which is what is leaking. But 64bit may provide enough headroom for it to go on for a while.

So the next question is “So when will 64bit be complete?” unfortunately I don’t have a great answer here but we should be able to start pushing out something within the next couple of weeks to adventurous testers.

Jared

I’m a little late to this conversation, but this appears to be the same problem I was having on my QHY600 when I first got it (see this thread: 64-bit Version of SGP? - #16 by flywaldo - Feature Requests - Main Sequence Software). QHY discovered a memory leak and released an update to the SDK. Once that was released, the out of memory errors no longer showed up. I ran sequences last week with my 600 capturing 154 images per night with no issue. I’m not sure it’s the same root cause here, but I wanted to throw this out there in case it is helpful.

Thanks for your feedback!
I made a post about the memory problems on CN, i made one on the ZWO forum too.

There’s also a problem with the filter wheel so it doesn’t repeat the position all the time, might be a hardware design problem :frowning: