Repeating Light Frame Problem

I had a strange issue last night that I have not encountered before. I am working on a 4 panel mosaic over the course of several nights. Last night I started the sequence and the first panel needed to complete 2 more frames in an event (frames 11 and 12 out of 12). Frame number 12 completed, but the event counter did not update and still showed 11/12 complete. So the sequence continued to take “frame #12” two more times:

So I ended up with 3 “#12” frames:

The LOG FILE shows the correct Frame 12 completed at 10:16:28, followed by the information:
“Found event 1 with remaining frames”
“Frame count 11/12”
Both of these would be false according to how I had the event set up.

Two more “Frame 12” images were completed at 10:36:39 and 10:56:49. It was at this point that I stopped the sequence and manually ticked the event counter DOWN to 11 frames. After some wierd Auto Recovery false starts I started the sequence again and it ran fine all night.

Joel,

Do you, by chance have the SGF file, in a similar state to starting the sequence that led to this behavior? Also… Did you:

  • Ever use the mark image as bad feature during this sequence?
  • Ever modify the number of completed frames by double clicking the event fraction display?
  • Did you re-order events?
  • You obviously re-ordered targets. When did you do this?

Here is the SGF file. Unfortunately after last night I have altered the frame count so it now reads 12/12 complete.

  • I have not used the mark image as bad feature.
  • The only time I modifed the number of frames completed was after the problem happened and not until the next time I opened SGP in the morning after the sequence completed.
  • I did not re-order events for the problem target (Veil-2). However I did re-order events in targets Veil-3 and Veil-4.
  • I don’t exactly remember when I reordered targets, but I have re-orderd them on more than one occassion. I’m sure I re-ordered them right before this sequence run (last evening).

Joel,

First… this is a real bug… I’m not saying otherwise. I am saying that I cannot reproduce it and I have spent hours staring at code and trying to analyze paths that could lead to this behavior. I even went so far as to recreate your sequence in its entirety so it looked exactly like yours did when you started it. Everything worked as expected.

So… I’m hoping this is a pretty rare state (however you got there). If you (or anyone else following this) sees this behavior and you want to participate in helping to track the issue, your first action should be to go to “Save As” and capture the sequence in as close to the error state as possible.

Please let me know if you are able to produce this behavior again.

No worries Ken. I’ll save the sequence if it happens again, though I have my doubts it will show up again.

Well Ken, the repeating image showed up again. And what’s disturbing is that this time it happend on a different target (same sgf) and a different filter event. Veil-3 event 1 completed 14/14 images, but SGP stayed on 13/14 and started #14 over again.

I aborted the sequence and immediately saved the sgf file with a new name.

LOG FILE

I should also note that the sequence file has been highly altered from the previous state, as I duplicated all the targets.

if its not too much to ask, can you also zip up the contents of the data created from this sequence (folder structure included)? There is a chance the issue lies in the list of files already taken and i want to produce your environment as closely as possible.

Sure thing, I’ll get that to you tomorrow. It’s going to be a huge file though.
One other tidbit, I observed that the total time remaing in the sequence went from 6:04 left to 6:24 at the start of the repeated 20min light frame.

Understood. Instead of this, you can also send screenshots of your lights folder. I just need to create file names with the same name… they dont really need to be FITS files.

@joelshort

I’m having a really hard time with this issue… the kind of time where a thing is happening that simply does not seem possible. I am missing something small and cryptic. Here is the basis of the problem… The correct event and frame start:

[8/20/2015 10:22:05 PM] [DEBUG] [Sequence Thread] ------------- Starting capture frame for event[0] -------------

Note the event 0. Somehow, between starting this event frame and ending it, the event number is mysteriously incremented by 1.

[8/20/2015 10:42:21 PM] [DEBUG] [Sequence Thread] Incremented numComplete (14) for event[1]...

This should still be reading event 0, but the sequence has moved to event 1 for some reason…

So… I have placed a debug build on dropbox. Since you seem to be fairly proficient at reproducing this issue, I am really hoping (pleading) that you can do it with this build. I have injected near 25 log points where the current event number is reported and will really help me narrow down the specific area that is acting up.

It is not an installer. You will need to replace the exe in your Program Files (x86) directory:

https://dl.dropboxusercontent.com/u/24325878/SequenceGeneratorDebug.zip

I decided to go ahead and give you the full files/folders. I hope you have a fast internet connection (1.2GB). Veil.zip
I should note that after I aborted the sequence, updated the frame count and restarted everything worked fine all night and the sequence completed.
Let me know if you have any problems downloading. It’s too big for dropbox so this is coming from my personal website and I’m not sure how that’s going to work.

I should be able to test the debug build tonight.

Joel,

I have the zip file. You can remove it from your site at your leisure. Thanks for getting that together.

Well shoot, Ken. I ran the sequence last night with the debug version and it worked flawlessly. No repeating frames. I’ll keep an eye out but I’m probably done with this particular sequence.