Date: Mon, 19 Dec 2005 01:22:33 -0800
From: Constance Rockosi
Subject: Re: 2.5m obslog 20051218 (MJD 53723)
SOP is *never* supposed to be talking to the spectrograph cameras when
they are reading out.  A look at the murmur log shows that SOP checked
up on the cameras only 14 seconds after it started them reading out --
SOP is supposed to know to wait 60 seconds.
On strange thing that shows up in the log is that right after
the DA is enabled (iccExec -riskyOn data -async sd1 readoutBegin) the
messages about the VCI missing interrupts start coming:
dsc_W_missedVCIInt VCI 1 has fallen behind. Generating a fake readout message for 0xa6000000.
dsc_W_missedVCIInt VCI 0 has fallen behind. Generating a fake readout message for 0xa2000000.
...etc
but this happends before SOP has a chance to tell the cameras to start
sending pixels (with FREAD).  All the other missed interrupt messages
for all the frames in tonight's murmur log come several seconds after
the FREAD is issued.  How can the VCI think it has fallen behind if
it hasn't gotten any lines yet?
So maybe SOP went on to start to talk to the camera because the readout
for frame 35543 failed somehow as soon as the DA readoutBegin was
enabled?  There isn't anything recorded in the murmur log about that,
and the "beginData finished" message is there in the murmur log,
so I can't say for sure.  SOP has always known not to talk to the
spectrograph cameras for 60 seconds after telling them to read, so
I would guess something else went wrong to confuse SOP.
--connie
> There was one strange occurrence while taking the 25 biases.  It stopped after
> 19 frames and the frame it stopped on (35543) looked like each camera's frame
> was made from sections of the others.  See the main log for sop's complaints.
> Subsequent exposures were fine.
>
> My sop returned prematurely from the loop of 25 biases with the following
> messages.
>
>
> sp1 cameraNag {} t-g-sdss-2.apo.nmsu.edu 2200
> sp2 cameraNag {} t-g-sdss-2.apo.nmsu.edu 2200
> 2005-12-19 01:39:43Z: Camera sp1 not idle: "READING"
> 2005-12-19 01:39:43Z: Camera sp2 not idle: "READING"
> Error: Aborting: Cameras not idle: "READING"
> sp1> 2005-12-19 01:39:43Z: registered (addr12) listener
> sdsshost2.apo.nmsu.edu:observer:26360
> 2005-12-19 01:39:43Z: Connection to sdsshost2.apo.nmsu.edu:observer:26360 has
> timed out; last packet 2005-12-19 00:57:37Z
> IOP running as Backup Relay Server
> requestListen: listening                                    7
> 2005-12-19 01:39:51Z 11 sp1 camcheck
> 2005-12-19 01:39:51Z 11 sp1 ln2stat
> 2005-12-19 01:39:52Z 11 sp2 camcheck
> 2005-12-19 01:39:54Z 11 sp2 ln2stat
> 2005-12-19 01:39:56Z: registered (addr13) listener sdsshost2.apo.nmsu.edu:observ
>
> I was then returned to the sop prompt with 6 more frames to go in the loop.
>
> Files for frame 35543 are messed up.  It looks like parts of different cameras
> were pasted into one camera's file.  The next bias was fine.  Could the relay
> server have gotten into the loop at a bad time and caused some sort of
> collision?
>