From: Stephen Bailey
Subject: Re: [sdss3-25mlog 1076] Re: [doravetz@apo.nmsu.edu: 2.5m obslog 20130214 (MJD 56338)]
Date: Sat, 16 Feb 2013 15:18:49 -0800
Stephen
On Feb 16, 2013, at 12:06, "John K. Parejko"  wrote:
> With tips from Benjamin Weaver, I've copied the *583 data over, and the di=
rectory will be resynced shortly.
>=20
> Kaike: you can sos-redo MJD 56338 if you like. Do you think its necessary a=
t this point?
>=20
> David: Craig said he had some notes about this he was going to send me. Ap=
parently it wasn't as fixed as previously thought (plus, I think it might be=
 a case of "only be tested when it occurs").
>=20
> One question is how to get *582 marked as "don't use", since it's a duplic=
ate of *583 for sp2. Should I email boss-pipeline, or does someone know the b=
est way to do that?
Are all exposures of *582 duplicates, or just the sp1 exps?
Jonathan Richards: if it is all exps as duplicates, you can flag these as ba=
d using apofix as normal.   If it is just the sp1 exps, then you can hand ed=
it the sdHdrfix file to to flag just r1 and b1 as bad instead of the wildcar=
d *.  Coordinate with John Parejko for exactly which exposures of which came=
ras are good/bad. =20
Stephen
>=20
> Cheers,
> John
>=20
> On 15Feb 2013, at 23:31, David Schlegel wrote:
>=20
>> John
>>=20
>> Excellent. It looks like we get one of these exposures back, which is goo=
d since the weather was better-than-perfect.=20
>>=20
>> 581 is lost. 582 and 583 are the same thing, and we should probably just c=
opy those back into the data directory with whichever exposure number matche=
s the header.=20
>>=20
>> Maybe the recovery should go to a more obvious directory. I had also thou=
ght this problem was fixed...?
>>=20
>> D
>>=20
>> Sent from my iPhone
>>=20
>> On Feb 15, 2013, at 20:21, "John K. Parejko"  wrot=
e:
>>=20
>>> I do see exposure 156583 (all cameras) in:
>>>=20
>>> /home/sdss3/products/Linux64/boss/v2_5_13/python/boss
>>>=20
>>> and from the timeline, it looks like that is the one that should have co=
me out. 156581 was "cleared", 156582 was the second failure, and was "recove=
red". The recovery code uses the *next* available exposure number, so the re=
covered exposure number should be 156583.
>>>=20
>>> Is the data in 156583 what you would have expected? Looks like a 900s da=
ta exposure, and the b2 and r2 file values are identical with those in 15658=
2.
>>>=20
>>> John
>>>=20
>>> On 15Feb 2013, at 22:06, Daniel Oravetz wrote:
>>>=20
>>>> Hi John,
>>>>=20
>>>> The last time I experienced this problem (56031 with calibration
>>>> exposures) the 'boss reloadControllers' and subsequent 'boss exposure
>>>> recover' did recover the data with both sp1 and sp2 data being written t=
o
>>>> disk. The recovered data were not written to the expected
>>>> /data/spectro/$MJD directory, but instead Craig found them in
>>>> $BOSS_DIR/python/boss, so he moved them to /data/spectro/56031 and adde=
d
>>>> them to the sha1 file. We could not find last night's recovered sp1 dat=
a
>>>> even though the indication was that the re-read resulting from the 'bos=
s
>>>> exposure recover' completed normally. Any idea as to whether the sp1 da=
ta
>>>> were properly recovered and where they may have been written? If not, w=
hy
>>>> would this instance have not recovered the data?
>>>>=20
>>>> Dan
>>>>=20
>>>>=20
>>>>> It looks like this is the same "netburner problem" as happened in spri=
ng
>>>>> of 2010, 2011 and 2012. See logs for MJDs e.g., 55244, 55264, 56007, a=
nd
>>>>> probably the closed tickets #475 and #895.
>>>>>=20
>>>>> On 15Feb 2013, at 12:28, Constance Rockosi wrote:
>>>>>=20
>>>>>> John or someone else who can,
>>>>>>=20
>>>>>> If you'd pull the comm logs, I can help parse through what happened
>>>>>> with the sp1 communications.
>>>>>=20
>>>>> Ok, I think I've found what you would be looking for. The log chunk is=
>>>>> from /data/logs/actors/boss/2013-02-14T07:12:28.log
>>>>>=20
>>>>> Does this help?
>>>>> John
>>>>>=20
>>>>> First error (~05:57:45):
>>>>>=20
>>>>> 2013-02-15 05:57:44.540Z root             20 CamProtocol.py:29 sp1cam >>>> REDPFWD     ok > ok'
>>>>> 2013-02-15 05:57:44.541Z cmds             20 CommandLink.py:117 > 2 51=
07 d
>>>>> text=3D"Completed message from sp1cam"
>>>>> 2013-02-15 05:57:44.541Z cmds             10 CommandLink.py:106 flushi=
ng
>>>>> queue to all outputs...
>>>>> 2013-02-15 05:57:44.542Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5107 d text=3D"Completed message from sp1cam"
>>>>> 2013-02-15 05:57:44.542Z root             20 BaseProtocol.py:117 sp1ca=
m >
>>>>> 'FREAD'
>>>>> 2013-02-15 05:57:44.543Z cmds             20 CommandLink.py:117 > 2 51=
07 d
>>>>> text=3D"Sending FREAD to sp1cam"
>>>>> 2013-02-15 05:57:44.544Z cmds             10 CommandLink.py:106 flushi=
ng
>>>>> queue to all outputs...
>>>>> 2013-02-15 05:57:44.544Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5107 d text=3D"Sending FREAD to sp1cam"
>>>>> 2013-02-15 05:57:44.636Z root             20 DAQInterface.py:863 sp1da=
q:
>>>>> Last Response: LINE=3D0:PIXEL=3D0:PIXCNT=3D2176
>>>>> 2013-02-15 05:57:44.658Z root             20 DAQInterface.py:980 sp1da=
q:
>>>>> Pixel 0 for amp0: 2204 (89c)
>>>>> 2013-02-15 05:57:44.658Z root             20 DAQInterface.py:981 sp1da=
q:
>>>>> Pixel 0 for amp1: 2328 (918)
>>>>> 2013-02-15 05:57:44.659Z root             20 DAQInterface.py:982 sp1da=
q:
>>>>> Pixel 0 for amp2: 2190 (88e)
>>>>> 2013-02-15 05:57:44.659Z root             20 DAQInterface.py:983 sp1da=
q:
>>>>> Pixel 0 for amp3: 2245 (8c5)
>>>>> 2013-02-15 05:57:44.659Z root             20 DAQInterface.py:984 sp1da=
q:
>>>>> Pixel 0 for amp4: 2295 (8f7)
>>>>> 2013-02-15 05:57:44.660Z root             20 DAQInterface.py:985 sp1da=
q:
>>>>> Pixel 0 for amp5: 2462 (99e)
>>>>> 2013-02-15 05:57:44.660Z root             20 DAQInterface.py:986 sp1da=
q:
>>>>> Pixel 0 for amp6: 2487 (9b7)
>>>>> 2013-02-15 05:57:44.660Z root             20 DAQInterface.py:987 sp1da=
q:
>>>>> Pixel 0 for amp7: 2673 (a71)
>>>>> 2013-02-15 05:57:44.864Z root             20 CamProtocol.py:29 sp1cam >>>> 'FREAD '
>>>>> 2013-02-15 05:57:44.866Z root             20 CamProtocol.py:29 sp1cam >>>> FREAD     ok > ok'
>>>>> 2013-02-15 05:57:44.866Z cmds             20 CommandLink.py:117 > 2 51=
07 d
>>>>> text=3D"Completed message from sp1cam"
>>>>> 2013-02-15 05:57:44.867Z cmds             10 CommandLink.py:106 flushi=
ng
>>>>> queue to all outputs...
>>>>> 2013-02-15 05:57:44.867Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5107 d text=3D"Completed message from sp1cam"
>>>>> 2013-02-15 05:57:57.350Z root             20 DAQInterface.py:863 sp2da=
q:
>>>>> Last Response: LINE=3D500:PIXEL=3D0:PIXCNT=3D2176
>>>>> 2013-02-15 05:57:57.843Z root             20 DAQInterface.py:863 sp1da=
q:
>>>>> Last Response: LINE=3D500:PIXEL=3D0:PIXCNT=3D2176
>>>>> 2013-02-15 05:58:09.584Z root             40 DAQInterface.py:893 sp1da=
q:
>>>>> recv returns zero length string in receiveImage.
>>>>> 2013-02-15 05:58:09.584Z root             40 DAQInterface.py:903 Canno=
t
>>>>> obtain pixel data wc data (Line 946, Pixel 182) Invalid response from
>>>>> Netburner.
>>>>> 2013-02-15 05:58:09.585Z root             40 DAQInterface.py:665 sp1da=
q:
>>>>> Exception in receiveImage. Transfer aborted. Message: Invalid response=
>>>>> from Netburner.
>>>>> 2013-02-15 05:58:09.599Z cmds             20 CommandLink.py:117 > 2 51=
07 w
>>>>> text=3D"sp1daq readout failed: Invalid response from Netburner. (>>>> method Deferred.errback of >)."
>>>>> 2013-02-15 05:58:09.599Z cmds             10 CommandLink.py:106 flushi=
ng
>>>>> queue to all outputs...
>>>>> 2013-02-15 05:58:09.600Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5107 w text=3D"sp1daq readout failed: Invalid response f=
rom
>>>>> Netburner. (>=
)."
>>>>>=20
>>>>>=20
>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>=20
>>>>>=20
>>>>> Second error (~06:22:30):
>>>>>=20
>>>>> 2013-02-15 06:22:29.588Z root             20 MechProtocol.py:51 'sp1me=
ch':
>>>>> 'TempBlueCamBot 0.3 (0x)'
>>>>> 2013-02-15 06:22:29.588Z root             20 MechProtocol.py:51 'sp1me=
ch':
>>>>> 'TempBlueCamTop 0.4 (0x)'
>>>>> 2013-02-15 06:22:29.588Z root             20 MechProtocol.py:51 'sp1me=
ch':
>>>>> 'AD 1670 1294'
>>>>> 2013-02-15 06:22:29.589Z root             20 MechProtocol.py:51 'sp1me=
ch':
>>>>> 'OK'
>>>>> 2013-02-15 06:22:29.589Z cmds             20 CommandLink.py:117 > 2 51=
10 d
>>>>> text=3D"Completed message from sp1mech"
>>>>> 2013-02-15 06:22:29.590Z cmds             10 CommandLink.py:106 flushi=
ng
>>>>> queue to all outputs...
>>>>> 2013-02-15 06:22:29.590Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 d text=3D"Completed message from sp1mech"
>>>>> 2013-02-15 06:22:29.591Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> text=3D"Response parsed."
>>>>> 2013-02-15 06:22:29.592Z cmds             10 CommandLink.py:106 flushi=
ng
>>>>> queue to all outputs...
>>>>> 2013-02-15 06:22:29.592Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i text=3D"Response parsed."
>>>>> 2013-02-15 06:22:29.593Z cmds             20 CommandLink.py:117 > 2 51=
10 d
>>>>> text=3D"Noting the status from sp1mech."
>>>>> 2013-02-15 06:22:29.593Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> sp1LastShutterTime=3D0.720,0.550
>>>>> 2013-02-15 06:22:29.594Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> sp1Humidity=3D22.1,28.1
>>>>> 2013-02-15 06:22:29.594Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> sp1Temp=3D0.3,0.2,0.8,0.4,0.0,0.3
>>>>> 2013-02-15 06:22:29.595Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> sp2LastShutterTime=3D0.690,0.610
>>>>> 2013-02-15 06:22:29.595Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> sp2Humidity=3D26.7,26.4
>>>>> 2013-02-15 06:22:29.595Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> sp2Temp=3D-0.1,-0.1,-0.3,0.0,-0.1,-0.7
>>>>> 2013-02-15 06:22:29.596Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> motorStatus=3D0x84,0x84,0x85,0x84,0x84,0x85
>>>>> 2013-02-15 06:22:29.596Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> motorPosition=3D26470,26892,26698,27365,26852,27536
>>>>> 2013-02-15 06:22:29.597Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> shutterStatus=3D0x02,0x02
>>>>> 2013-02-15 06:22:29.597Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> screenStatus=3D0x0a,0x0a
>>>>> 2013-02-15 06:22:29.597Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> slitIDs=3D13,13
>>>>> 2013-02-15 06:22:29.598Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> specMechVersion=3D3.0.1,3.0.1
>>>>> 2013-02-15 06:22:29.598Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> specMechProtocol=3D3,3
>>>>> 2013-02-15 06:22:29.598Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> text=3D"Actual exposure time for sp1 =3D 900.830"
>>>>> 2013-02-15 06:22:29.599Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> text=3D"Actual exposure time for sp2 =3D 900.870"
>>>>> 2013-02-15 06:22:29.599Z root             20 ExposureCmd.py:647 156581=
 :
>>>>> Prepping the daq...
>>>>> 2013-02-15 06:22:29.600Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> text=3D"Prepping the daq..."
>>>>> 2013-02-15 06:22:29.600Z cmds             20 CommandLink.py:117 > 2 51=
10 d
>>>>> text=3D"initializing daq, just cuz"
>>>>> 2013-02-15 06:22:29.601Z root             10 DAQInterface.py:1377 sp1d=
aq:
>>>>> Sending SET DEBUG 0 to daq
>>>>> 2013-02-15 06:22:29.601Z root             40 DAQInterface.py:1434 sp1d=
aq:
>>>>> recv returns zero length string in getResponse.
>>>>> 2013-02-15 06:22:29.602Z root             40 DAQInterface.py:1406 sp1d=
aq:
>>>>> Cannot get response from command socket. Message: Invalid response fro=
m
>>>>> Netburner.
>>>>> 2013-02-15 06:22:29.602Z root             40 DAQInterface.py:1408 sp1d=
aq:
>>>>> Resetting Data Sock
>>>>> 2013-02-15 06:22:29.603Z cmds             20 CommandLink.py:117 > 2 51=
10 w
>>>>> text=3D'retrying readout prep. after DAQ connection re-established'
>>>>> 2013-02-15 06:22:29.603Z cmds             20 CommandLink.py:117 > 2 51=
10 d
>>>>> text=3D"initializing daq, just cuz"
>>>>> 2013-02-15 06:22:29.603Z root             10 DAQInterface.py:223
>>>>> Connecting to data link on host 192.168.1.4, port 7889
>>>>> 2013-02-15 06:22:29.604Z root             10 DAQInterface.py:228 sp1da=
q:
>>>>> Connected.
>>>>> 2013-02-15 06:22:29.605Z root             10 DAQInterface.py:1377 sp1d=
aq:
>>>>> Sending SET DEBUG 0 to daq
>>>>> 2013-02-15 06:22:29.606Z root             20 DAQInterface.py:1450 sp1d=
aq:
>>>>> Received response: CMD=3DSET:DEBUG=3D0
>>>>> 2013-02-15 06:22:29.606Z root             10 DAQInterface.py:1377 sp1d=
aq:
>>>>> Sending CLEAR to daq
>>>>> 2013-02-15 06:22:29.608Z root             20 DAQInterface.py:1450 sp1d=
aq:
>>>>> Received response: CMD=3DCLEAR:GCR=3D1:DSR=3D144
>>>>> 2013-02-15 06:22:29.608Z root             10 DAQInterface.py:1377 sp1d=
aq:
>>>>> Sending SET READOUTMODE ON to daq
>>>>> 2013-02-15 06:22:29.610Z root             20 DAQInterface.py:1450 sp1d=
aq:
>>>>> Received response: CMD=3DSET:GCR=3D1
>>>>> 2013-02-15 06:22:29.610Z root             10 DAQInterface.py:1377 sp1d=
aq:
>>>>> Sending SET EXPID 1 to daq
>>>>> 2013-02-15 06:22:29.611Z root             20 DAQInterface.py:1450 sp1d=
aq:
>>>>> Received response: CMD=3DSET:EXPID=3D1
>>>>> 2013-02-15 06:22:29.612Z root             10 DAQInterface.py:1377 sp1d=
aq:
>>>>> Sending SET ROMODE 0 to daq
>>>>> 2013-02-15 06:22:29.613Z root             20 DAQInterface.py:1450 sp1d=
aq:
>>>>> Received response: CMD=3DSET:ROMODE=3D0:
>>>>> 2013-02-15 06:22:29.614Z root             10 DAQInterface.py:1377 sp1d=
aq:
>>>>> Sending SET EXPID 156581 to daq
>>>>> 2013-02-15 06:22:29.615Z root             20 DAQInterface.py:1450 sp1d=
aq:
>>>>> Received response: CMD=3DSET:EXPID=3D156581
>>>>> 2013-02-15 06:22:29.615Z root             10 DAQInterface.py:1377 sp1d=
aq:
>>>>> Sending SET LINES 2112 to daq
>>>>> 2013-02-15 06:22:29.617Z root             20 DAQInterface.py:1450 sp1d=
aq:
>>>>> Received response: CMD=3DSET:LINES=3D2112:PIXELS=3D2176
>>>>> 2013-02-15 06:22:29.617Z root             10 DAQInterface.py:1377 sp1d=
aq:
>>>>> Sending SET PIXELS 2176 to daq
>>>>> 2013-02-15 06:22:29.619Z root             20 DAQInterface.py:1450 sp1d=
aq:
>>>>> Received response: CMD=3DSET:LINES=3D2112:PIXELS=3D2176
>>>>> 2013-02-15 06:22:29.619Z root             10 DAQInterface.py:1377 sp1d=
aq:
>>>>> Sending SET EXPOSURE ON to daq
>>>>> 2013-02-15 06:22:29.621Z root             20 DAQInterface.py:1450 sp1d=
aq:
>>>>> Received response: CMD=3DSET:EXPOSURE=3DON:GCR=3D129:DSR=3D128
>>>>> ...
>>>>>=20
>>>>> 2013-02-15 06:22:29.643Z root             20 ExposureCmd.py:679 156581=
 :
>>>>> Daq ready.
>>>>> 2013-02-15 06:22:29.644Z root             20 ExposureCmd.py:682 156581=
 :
>>>>> Clocking the CCDs.
>>>>> 2013-02-15 06:22:29.645Z root             20 Exposure.py:53
>>>>> exposureState=3DREADING,69.7,0.0
>>>>> 2013-02-15 06:22:29.646Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> exposureState=3DREADING,69.7,0.0
>>>>> 2013-02-15 06:22:29.646Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> text=3D"Arming the daq..."
>>>>> 2013-02-15 06:22:29.647Z root             20 ExposureCmd.py:695 156581=
 :
>>>>> Arming the daq...
>>>>> 2013-02-15 06:22:29.647Z root             20 DAQInterface.py:289 sp1da=
q:
>>>>> Starting receiver!
>>>>> 2013-02-15 06:22:29.647Z root             10 DAQInterface.py:223
>>>>> Connecting to data link on host 192.168.1.4, port 7890
>>>>> 2013-02-15 06:22:29.648Z root             10 DAQInterface.py:228 sp1da=
q:
>>>>> Connected.
>>>>> 2013-02-15 06:22:29.649Z root             20 DAQInterface.py:292 sp1da=
q:
>>>>> reconnecting data sock
>>>>> 2013-02-15 06:22:29.650Z root             10 DAQInterface.py:1377 sp1d=
aq:
>>>>> Sending ARM  to daq
>>>>> 2013-02-15 06:22:29.651Z root             20 DAQInterface.py:1450 sp1d=
aq:
>>>>> Received response: CMD=3DARM:STATUS=3DACCEPTED
>>>>> 2013-02-15 06:22:29.651Z root             40 DAQInterface.py:1434 sp1d=
aq:
>>>>> recv returns zero length string in getResponse.
>>>>> 2013-02-15 06:22:29.652Z root             20 DAQInterface.py:289 sp2da=
q:
>>>>> Starting receiver!
>>>>> 2013-02-15 06:22:29.652Z root             40 DAQInterface.py:699 sp1da=
q:
>>>>> Exception in receiveImage. Transfer Aborted. Message: Invalid response=
>>>>> from Netburner.
>>>>> 2013-02-15 06:22:29.653Z root             10 DAQInterface.py:1377 sp2d=
aq:
>>>>> Sending ARM  to daq
>>>>> 2013-02-15 06:22:29.654Z root             40 DAQInterface.py:665 sp1da=
q:
>>>>> Exception in receiveImage. Transfer aborted. Message: Invalid response=
>>>>> from Netburner.
>>>>> 2013-02-15 06:22:29.655Z root             20 DAQInterface.py:1450 sp2d=
aq:
>>>>> Received response: CMD=3DARM:STATUS=3DACCEPTED
>>>>> 2013-02-15 06:22:29.655Z root             20 ExposureCmd.py:711 156581=
 :
>>>>> Daq armed.
>>>>> 2013-02-15 06:22:29.655Z cmds             20 CommandLink.py:117 > 2 51=
10 i
>>>>> text=3D"Reading out..."
>>>>> 2013-02-15 06:22:29.656Z root             20 ExposureCmd.py:715 156581=
 :
>>>>> Waiting for readout to complete.
>>>>> 2013-02-15 06:22:29.658Z cmds             20 CommandLink.py:117 > 2 51=
10 w
>>>>> text=3D"sp1daq readout failed: Invalid response from Netburner. (>>>> method Deferred.errback of >)."
>>>>> 2013-02-15 06:22:29.658Z cmds             10 CommandLink.py:106 flushi=
ng
>>>>> queue to all outputs...
>>>>> 2013-02-15 06:22:29.659Z root             20 DAQInterface.py:1450 sp2d=
aq:
>>>>> Received response:
>>>>> EXPOSURE=3D156582:PIXCOUNT=3D36765696:LINES=3D2112:PIXPERLINE=3D2176
>>>>> 2013-02-15 06:22:29.660Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 d text=3D"Noting the status from sp1mech."
>>>>> 2013-02-15 06:22:29.661Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i sp1LastShutterTime=3D0.720,0.550
>>>>> 2013-02-15 06:22:29.660Z root             20 DAQInterface.py:717 sp2da=
q:
>>>>> Exposure ID 156582, total pixel count 36765696, Lines 2112, Pixels per=
>>>>> line 2176
>>>>> 2013-02-15 06:22:29.661Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i sp1Humidity=3D22.1,28.1
>>>>> 2013-02-15 06:22:29.743Z root             10 DAQInterface.py:751 sp2da=
q:
>>>>> Waiting for first line of exposure 156582
>>>>> 2013-02-15 06:22:29.744Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i sp1Temp=3D0.3,0.2,0.8,0.4,0.0,0.3
>>>>> 2013-02-15 06:22:29.745Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i sp2LastShutterTime=3D0.690,0.610
>>>>> 2013-02-15 06:22:29.745Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i sp2Humidity=3D26.7,26.4
>>>>> 2013-02-15 06:22:29.745Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i sp2Temp=3D-0.1,-0.1,-0.3,0.0,-0.1,-0.7
>>>>> 2013-02-15 06:22:29.746Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i motorStatus=3D0x84,0x84,0x85,0x84,0x84,0x85
>>>>> 2013-02-15 06:22:29.746Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i motorPosition=3D26470,26892,26698,27365,26852,275=
36
>>>>> 2013-02-15 06:22:29.746Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i shutterStatus=3D0x02,0x02
>>>>> 2013-02-15 06:22:29.747Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i screenStatus=3D0x0a,0x0a
>>>>> 2013-02-15 06:22:29.747Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i slitIDs=3D13,13
>>>>> 2013-02-15 06:22:29.747Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i specMechVersion=3D3.0.1,3.0.1
>>>>> 2013-02-15 06:22:29.748Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i specMechProtocol=3D3,3
>>>>> 2013-02-15 06:22:29.748Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i text=3D"Actual exposure time for sp1 =3D 900.830"=
>>>>> 2013-02-15 06:22:29.748Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i text=3D"Actual exposure time for sp2 =3D 900.870"=
>>>>> 2013-02-15 06:22:29.749Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i text=3D"Prepping the daq..."
>>>>> 2013-02-15 06:22:29.749Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 d text=3D"initializing daq, just cuz"
>>>>> 2013-02-15 06:22:29.749Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 w text=3D'retrying readout prep. after DAQ connecti=
on
>>>>> re-established'
>>>>> 2013-02-15 06:22:29.750Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 d text=3D"initializing daq, just cuz"
>>>>> 2013-02-15 06:22:29.750Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 d text=3D"initializing daq, just cuz"
>>>>> 2013-02-15 06:22:29.750Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i exposureState=3DREADING,69.7,0.0
>>>>> 2013-02-15 06:22:29.751Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i text=3D"Arming the daq..."
>>>>> 2013-02-15 06:22:29.751Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 i text=3D"Reading out..."
>>>>> 2013-02-15 06:22:29.751Z cmds             10 CommandLink.py:110 flushi=
ng
>>>>> queue line: 2 5110 w text=3D"sp1daq readout failed: Invalid response f=
rom
>>>>> Netburner. (>=
)."
>>>>> _____________________________________________________
>>>>> Message 1070 of sdss3-25mlog - Logs of nightly APO activities.
>>>>> https://trac.sdss3.org/tracmailman/browser/private/sdss3-25mlog/all/10=
70.html
>>>>> sdss3-25mlog@sdss3.org
>>>>> https://sdss3.org/mailman/listinfo/sdss3-25mlog
>>>=20
>>> --
>>> *************************
>>> John Parejko
>>> john.parejko@yale.edu
>>> http://www.astro.yale.edu/~jp727/
>>> 203 432-9759
>>> JWG 465
>>> Department of Physics
>>> Yale University
>>> New Haven, CT
>>> **************************
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>> _____________________________________________________
>>> Message 1072 of sdss3-25mlog - Logs of nightly APO activities.
>>> https://trac.sdss3.org/tracmailman/browser/private/sdss3-25mlog/all/1072=
.html
>>> sdss3-25mlog@sdss3.org
>>> https://sdss3.org/mailman/listinfo/sdss3-25mlog
>=20
> --
> *************************
> John Parejko
> john.parejko@yale.edu
> http://www.astro.yale.edu/~jp727/
> 203 432-9759
> JWG 465
> Department of Physics
> Yale University
> New Haven, CT
> **************************
>=20
>=20
>=20
>=20
>=20
>=20
> _____________________________________________________
> Message 1076 of sdss3-25mlog - Logs of nightly APO activities.
> https://trac.sdss3.org/tracmailman/browser/private/sdss3-25mlog/all/1076.h=
tml
> sdss3-25mlog@sdss3.org
> https://sdss3.org/mailman/listinfo/sdss3-25mlog