Flats take too long. We really need access to a DSLR's memory card!

Sequence Generator Feature Requests
1

Hi guys

First off I want to say that I am really loving SGPro! I’m still in the trial period and am yet to actually try it out under the stars, but testing has gone well so I’ve no reason (so far!) to assume that it won’t work. Once I’ve had a single successful night I will be purchasing the full package for sure.

Something’s bothering me though. Flats! With my current setup, and the long waits (~35 secs or so) between each flat due to the download time, it takes about 17.5 mins to capture 30 Flats. Each flat is only 1 sec long, so that’s 17.5 mins to capture 30 secs of data! Correct me if I’m wrong here, but isn’t that insane? I presume other DSLR users are in a similar boat, so how come this is not more of an issue?

I’ve obviously had to find a workaround, as an additional 16 mins at the end of each session is just too much in my book (I could be mostly packed up in that time, or captured another couple of lights). I’ve done some testing and now know the exact % brightness to set my Ipad to, such that a 1 sec exposure leads to a Mean ADU of just under 8,000, which I understand is what one needs to aim for with any 14bit camera, such as my Nikon D5300. I was able to confirm this with the Flats Calibration Wizard. I chose this method as I wanted to be able to use a standard exposure time, such as 1 sec, as opposed to something non-standard like say 0.28, that way I wouldn’t be tied to using SGPro for capturing them. So now what I plan to do is, at the end of capturing my lights (usually 8 min exposures) I will add an event to do a single 1 sec exposure to reset the camera from Bulb to 1” so I won’t need to scroll through them myself manually on the camera, thus risking nudging the optical train too much. Then shut down SGPro, disconnect the camera’s USB cable and shut the camera down. Then insert the memory card, power up the camera, and then use the Nikon’s built-in intervalometer setting in the menu to take the 30 1sec exposures that I need (with an interval of 1 sec between each one). This way I can knock them out in about a minute, instead of 17.5! The only downside that I can see, is that I need to touch the camera for the power cycle and memory card insertion, but I don’t see that as being too big of a deal.

And all of this just because we can’t access the memory card from within SGPro. Surely something can be done about this, right? I can think of at least one other Astro program that gives access to a DSLR’s memory card, so why can’t SGPro?

Ps – Is my workaround really necessary, or am I missing something obvious here? Also, I presume there will be no difference (consistency wise) between the Raw flat files I create on the memory card and the Raw Lights that SGPro creates?

1 Like
2

Yup. The weird thing is other programs seem to pull it off. I use qDSLRDASHBOARD when I’m in the field and it writes to the memory card then downloads it to the iOS/computer.

3

It would be great to be able to save the images to the memory card instead of the computer. That would eliminate download time. When I use the intervalometer on my canon I can do subs back to back thus really using almost all of my imaging time for collecting photons. BYEOS does that but it obviously is a much simpler program than SGP. perhaps there should be a DSLR mode where the images are downloaded only do focus, plate solve etc but not during the subs acquisition.

1 Like
4

35 sec seems like a long time, with my 6D download takes around 7 seconds.
Have you checked the speed between the camera and the pc with a short USB cable?

5

SGP does a lot of automation and to do it well we need raw data. Not the JPG data from the camera. If we used JPGs for display in SGP this would mean that you would lose out on accurate data for things like:

  • Image History Metrics
  • Auto focus
  • Plate solving
  • Pretty much all automation that requires images from the camera.

Some of the things would likely still work but things would certainly be sketchy. For every minute you saved for download time you’d likely pay for it with 15 minutes of headache, chasing demons. It’s just not worth it.

Thanks,
Jared

6

JPG has that much of an effect on the quality of auto focus and plate solving?

I had no idea it was changing the images that much.

7

Yeah I’m surprised to hear that too. Other people (some with very expensive setups) seem to be able to image with other software that uses JPG for focus, plate solving etc, and their images don’t seem to suffer.

I do take your point about not wanting to feed people with false data, but is it possible there’s a slight bit of OCD going on here, and if so, would it not be ok to relax some of these restrictions, seeing as we are only talking about DSLR users here anyway, and by their very nature DSLR’s are never going to be exactly pushing the boundaries of what one can achieve in AP. Also, let us not forget that there are no doubt many more DSLR users out there than CCD users.

To get back on topic though :stuck_out_tongue: Xplode 7 secs is seriously fast mate, I can only wish for such speed. Do you have a fast setup by chance? (SSD, and/or USB3). Your 6D is about 20MP while my D5300 is 24MP, but even so that doesn’t even come close to explaining the time difference. I am using an old Netbook which has 3 USB2 slots and a 5400rpm HDD. Yes it’s slow, but when I use other Astro software it can take Raw exposures almost back to back while it saves to both the SD card and the computer, so I still can’t understand how it can manage it and SGPro has such long download times by comparison. I’ve tried a few different cables and all the USB slots, there’s no difference either way.

The only thing I can surmise is that there must still be a lot more headroom for SGPro when it comes to data transfer rates?

8

Absolutely. JPG is 24 bit but that’s only 8 bit per color. Where fits is 16bit and grayscale and also linear, that’s a significant difference. Also let’s not forget that the JPG data is also compressed. But the real issue is that the JPG data is NOT linear coming out of the camera. Things get automatically stretched and white balance gets applied. This causes issues mostly with plate solving but to a lesser amount with focusing.

I’m not being OCD here. We initially used JPG for plate solve and auto focus and had significant issues with it. So we went back to raw and those problems started going away. We don’t just use RAWs for fun, we would much prefer JPG as they’re smaller and easier to deal with than having to convert the RAW data. Some plate solvers deal with JPG better than others…

It might not be the download time. If you have “Image History” turned on SGP will also do some analytics on your image. These can take some time if you’re using a slow machine. May want to turn that off and see if your time improves.

Raw download time on my 6D is about 7 seconds as well. If I have “Image History” turned on that can add a couple more seconds using my i7 with SSD.

Thanks,
Jared

9

Thanks Jared.

I have Image History turned off, so unfortunately it’s not that
By the sounds of it, SGPro doesn’t like the fact that my system is pretty old. Can’t deny I’m a little bit gutted at this, lol, as one of the few saving graces of DSO imaging is that you have never needed a powerful computer to do it (Planetary is a different matter altogether obviously). Due to being a city dweller, I have to do all my imaging (which isn’t much these days) at a dark site, which means being portable. My old netbook has great battery life so that’s always been one less thing to have to worry about, and it’s never held me back before when it comes to image capture. I certainly can’t stretch to buying a new laptop just for imaging (funds are tight enough as it is), and I can’t imagine I’m the only one who’s not using a fast computer for DSO imaging.

Jared, I know it’s a strange question, but did you guys do much testing on older hardware? Because it still seems strange to me that other Astro packages can seem to save down Raw exposures much more quickly, even on my old netbook. I’ve tested a couple of other programs and if I had to hazard a guess I’d say they too take in the region of about 7 secs, so I’m wondering what else is going on here? The image analysis part doesn’t take long at all, a second or two at most, it appears to be mostly the downloading that is taking up the all the time.

And finally, do you think there will ever be a possibility of keeping the SD card in the camera and using it for some things, such as Flats? Or is it 100% not ever going to happen? Just want to manage my expectations on this one.

10

One thing to check is to see if you are downloading FIT + RAW if you set to just one it should be faster as well. Check your settings in the drop box for the camera.

I noticed the CPU spike and delay for this download occurs during the invocation of DCRAW. Not sure if there is any room to enhance that particular step.

11

Nice idea Entilza but I’m already saving as Raw only. If i choose to save both the download takes nearly a minute!

12

Yes, my obs machine is a 8 year old core 2 duo. Not sure about the download times. I’d have to check with a Nikon and unfortunately the USB connection on our test Nikon seems to have died a couple months back and I haven’t gotten around to checking it. My 6D and 60D both seem to download just fine even on my Core 2 Duo, so I doubt this is related to hardware.

It’s unlikely, SGP is not a dedicated DSLR imaging platform, rather it is an automation platform that supports some aspects of controlling a DSLR. There are other dedicated DSLR options out there but they lack the automation of SGP. I guess it comes down to if you want better automation and a single learning curve if/when you get a CCD or better integration with your DSLR.

I will look into the Nikon download issue but it may be a while as we’re sans-Nkon at the moment. If you know anyone that wants to buy a D5100 with a non-functional USB port let me know :-/

Thanks,
Jared

13

Hi Jared

The issue doesn’t appear to be just Nikon-related, like yourself I too own a Canon 60D and I’m seeing the same behaviour with it.

I did a bit more testing last night, all done with the 60D and no memory card.
First I tested the speed of each of my Netbook’s USB slots. I then compared the transfer speeds to those I was getting on my powerful desktop computer, and saw no difference. So the Netbook’s USB slots are definitely operating at full USB2.0 speeds.
I then fired off some Raw exposures with the other Astro software I’ve used in the past, and timed how long it took for each one to appear in the download folder. Each 19MB CR2 file was taking about 7 to 8 secs to appear, which sounds similar to the speeds you are seeing on your machine.
I then fired off some similar Raw exposures using SGP. I also analysed the download folder and saw that on average, each 19MB CR2 file was taking about 25 secs to appear, followed by the Image Analysis time which tended to fluctuate from anywhere between 5 to 10 secs.

I’m comfortable with the Image Analysis time, but how come the pure transfer time itself is so much longer? (7s Vs 25s). This is the source of my main gripe (not the necessity of using Raw files for Focusing, PlateSolving etc) as at the moment it means I am wasting a lot of time between frames. If we can get it sorted then I would be happy to use SGP for my Flats and avoid the need to use the memory card at all. Plus it would have the obvious benefit of enabling me to capture more Lights in the same timeframe.

Cheers,

Ciarán.

14

The main issue is not how long it takes, but in the case of sky flats, I constantly chase the wright exposure time, as Histogram drifts during capture and download, this morning I started my capture run at 5 secs for OIII and ended up with 1 sec at the end of my 18 flats …

There should be a better way …

/Yves

1 Like
15

Sorry to bump this again, but can nothing be done about the download time? I’m having a hard time understanding how other Astro software can manage Raw downloads so much faster than SGP. I really, really want to invest in this software (note, not for automation, but simply for target framing and acquisition) but as it stands the long gaps in between frames is really hurting bad.

16

I know one major delay is the invoking of DCRAW for each image download and each image load, how necessary is it to invoke this? Is it to convert it to a FIT for the preview? If this step was not done, would the image also not appear bayered? This would be nice in the preview window. What would the repercussions be?

17

DCRAW is used to read the data from the CR2. For frame and focus images you can use the “JPG” option in the frame and focus module to speed things up. That will pull the JPG for display and it will be in color (debayering technically happens via the camera as JPG is stored in the CR2 file).

I’ll move this to the feature requests for us to look into the transfer speeds a little more. I can’t promise if/when that will happen. Only that we’ll look into it and see if we can speed things up.

Thanks,
Jared

3 Likes
18

That would be really helpful. Thanks Jared.

19

Thanks also Jared!

1 Like
20

I upgraded to the latest beta (2.5.2.5 i believe) in case it would solve my slow download times, but unfortunately it has not.
So i’ve included a link to my logfile below. In it, i booted up SGP afresh, connected the D5300 (chose no telescope or autoguider just to keep things simple) and started a basic sequence of 3 images of 5 seconds each. Each one took 25 or so seconds to download.
Maybe you guys can use it to help work out what is going on with my system.

https://1drv.ms/t/s!AhhWC3D3zU7BmgonLctRONamBfnU

ps - I should also point out that each time i install a new version, i have to manually copy over the Type 11 files with the ones i downloaded as part of the Nikon SDK, otherwise SGP can’t connect to the camera. I don’t think my issue is related to this (as my Canon 60D has similarly slow download times) but it is kind of annoying that each new version of SGP doesn’t seem to have the updated files needed to connect automatically.