Subject: Re: [sloan-25mlog 4179] Re: 2.5m obslog 20180914 (SJD 58376)
From: John Downey
Date: Sat, 15 Sep 2018 12:34:29 -0600
Interesting, that does seem to rule out FFS position.  The odd thing is =
that you were apparently able to move the shutter with a manual command. =
 I wonder if the hartmann screens are stuck and that prevents the =
shutter from opening for an exposure.
Something to check anyway.
> On Sep 15, 2018, at 12:02 PM, Daniel Oravetz  =
wrote:
>=20
> Hi John,
>=20
> A little more about the sp2 shutter & hartmann misbehavior last =
night...
>=20
> I did consider that the state of the FFS PLC last night might have
> affected exposure behavior in unusual ways, but it still felt like =
there
> was something else going on in the sp2 mechanicals as it relates to
> Shutter and Hartmann that couldn't be explained by the FFS.
>=20
> I was running everything outside of SOP, using only the boss actor to =
take
> exposures, and operating that way usually circumvents a lot of the
> sanity-check type of preconditions that are folded-in to SOP; e.g., =
"are
> the petals open when one tries to integrate?" is not a condition I =
recall
> 'boss exposure ...' commands querying in the past.
>=20
> The sp2 images that wrote to disk yesterday (both the biases and the
> no-light arcs) actually have populated 'FFS' header keywords. I talked
> with Jose about this a little bit last night when we discussed the =
engCam
> and how it handles FFS status: whereas the ecameraICC can't write an
> exposure without the FFS status reporting 0's (closed) or 1's (open), =
the
> bossICC seems to be a bit more elegant in handling this and will =
actually
> report 'X' for each petal that's in an indeterminate state.
>=20
> When we had the problem with FF petal #5 about four months ago, where =
it
> was not registering as '0' or '1', we were still able to take data; =
i.e.,
> the shutters and hartmann screens behaved correctly
> (https://trac.sdss.org/ticket/2867) despite not getting a '0' or '1' =
from
> all eight FF petals.
>=20
> Also note that I was unable to move the SP2 hartmann screens 'Out'
> *independent of any exposure attempts*:
>=20
>>>>> I also tried a 'boss hartmann out spec=3Dsp2' to see if I could =
manually
>>>>> pull the
>>>>> hartmann screens out, but this command timed-out and the sp2 =
Hartmann
>>>>> status
>>>>> remained 'Right In'.
>=20
> Such a relatively low-level command should be independent of the FFS
> status, I would think.
>=20
> I was also surprised to see the sp1 Shutter and Hartmann status behave
> correctly. I would have expected that if the bossICC had something =
like a
> FFS check in place before commanding a shutter or hartmann screen that
> we'd see the same misbehavior in both sp1 and sp2, but that was not =
the
> case: sp1 Shutter and Hartmann were doing exactly what they should =
during
> an arc, but sp2 was not.
>=20
> Having mentioned all that, I also wouldn't necessarily be surprised if
> things started working again when the FFS work concludes. There have =
been
> inter-dependencies before that weren't exactly intuitive.
>=20
> Dan
>=20
>=20
>=20
>> Thanks for the info. We will see if it will work normally after the =
FFS
>> work today. We will
>> also try another MechIack on SP2.
>>=20
>>> On Sep 15, 2018, at 7:01 AM, John Downey  =
wrote:
>>>=20
>>>>>=20
>>>>> ---- Observing Summary ----
>>>>>=20
>>>>> A few test BOSS biases were taken to test aliveness, but only the =
sp2
>>>>> data would
>>>>> write to disk and the BOSS instrument status would hang at =
'Reading
>>>>> 0.0' on each
>>>>> attempt. There were no indications of Netburner or communication
>>>>> problems that
>>>>> might explain why the r1 data would not write to disk, and the
>>>>> controllers were
>>>>> all responsive. It seems plausible that some check or precondition
>>>>> isn't being
>>>>> met on sp1 as a whole (given that b1 is entirely disconnected) and
>>>>> that, in turn,
>>>>> is disrupting the readout/write-to-disk completion.
>>>>>=20
>>>>> Homed/re-zeroed the BOSS spectro collimators and then commanded =
them
>>>>> to the
>>>>> instructed target position a=3Db=3Dc=3D27000 steps.
>>>>>=20
>>>>> The sp2 Shutter and Hartmann behavior is abnormal during =
exposures.
>>>>> Please see
>>>>> the Log body for a description.
>>>>>=20
>>>>> In short, to sum-up tonight's BOSS exposure (mis)behavior: sp1 =
Shutter
>>>>> and
>>>>> Hartmann behavior is normal during exposures, but the r1 images =
are
>>>>> *not*
>>>>> written to disk. The sp2 images *do* write to disk, but Shutter
>>>>> remains 'Closed' and
>>>>> Hartmann does not change to 'Out' during exposures, so there's no
>>>>> light in the
>>>>> sp2 images.
>>>>=20
>>>> I don't think R1 images will read out without the blue saddlebag
>>>> installed.  We use a couple work arounds to get stuff like =
temperature
>>>> info out of the red camera without the blue one there, but if I
>>>> remember correctly data doesn't pass through the work around.  In =
the
>>>> opposite case (red camera removed, blue installed) it might still =
read
>>>> data.
>>>>=20
>>>> There is also a problem to be addressed in the SP1 daq connection =
that
>>>> Connie found when taking biases on Thursday afternoon.
>>>>=20
>>>> The sp2 shutter behavior is a new one.  Another thing to look into
>>>> today if we can get to it after the FFS.  Hopefully just an air
>>>> problem.
>>>=20
>>> Reading Dan's description below, I now suspect that the SP2 shutter
>>> function will be restored by resolving the FFS position... I suspect
>>> that something is preventing the shutter from opening due to it =
failing
>>> a check of FFS position.  Whether this happens in the PLC logic or =
at a
>>> higher level, I don't know.  But the higher level stuff gets it's
>>> physical system info from the PLC, so I suspect this problem will
>>> disappear.
>>>=20
>>>>=20
>>>>>=20
>>>>> ---- Observing Log ----
>>>>>=20
>>>>> All actors alive and responsive.
>>>>>=20
>>>>> APOGEE evening cals were taken and completed normally.
>>>>>=20
>>>>> A guider dark was also successfully obtained.
>>>>>=20
>>>>> A few test BOSS biases were taken to test aliveness, but only the =
sp2
>>>>> data would
>>>>> write to disk and the BOSS instrument status would hang at =
'Reading
>>>>> 0.0' on each
>>>>> attempt. There were no indications of Netburner or communication
>>>>> problems that
>>>>> might explain why the r1 data would not write to disk, and the
>>>>> controllers were
>>>>> all responsive. It seems plausible that some check or precondition
>>>>> isn't being
>>>>> met on sp1 as a whole given that b1 is entirely disconnected and =
that,
>>>>> in turn,
>>>>> is disrupting the readout/write-to-disk completion.
>>>>>=20
>>>>> Home Spectro Collimators
>>>>> ************************
>>>>> Homed sp1 collimator first (found reverse limit for each =
actuator).
>>>>> FYI, the
>>>>> actuator positions before re-zeroing were: a=3D-36002 b=3D-35783 =
c=3D-34585.
>>>>>=20
>>>>> Homed sp2 collimator. The actuator positions before re-zeroing =
were:
>>>>> a=3D-29533
>>>>> b=3D-31171 c=3D-30855.
>>>>>=20
>>>>> After homing and re-zeroing, I verified that the actuators for =
both
>>>>> the sp1 and
>>>>> sp2 collimators had at least 45,000 steps of travel from the home
>>>>> location: two
>>>>> successive moves totaling 45,000 steps completed successfully for =
all
>>>>> three
>>>>> actuators on both spectrographs.
>>>>>=20
>>>>> Lastly, the collimators were sent to the instructed target =
position of
>>>>> a=3Db=3Dc=3D27,000, which should be near enough to the desired =
"seasonal
>>>>> center."
>>>>>=20
>>>>> Focus SP2 and adjust spatial/spectral coverage
>>>>> **********************************************
>>>>> With the spectro collimators homed and centered, I was next going =
to
>>>>> focus r2
>>>>> and b2 (I wasn't going to do r1 because of the =
exposure-write-to-disk
>>>>> problem
>>>>> described above) using the focus rings, and then proceed to =
adjusting
>>>>> the
>>>>> spatial and spectral coverage, but the sp2 shutter and hartmann
>>>>> screens weren't
>>>>> behaving as expected during the exposure sequence.
>>>>>=20
>>>>> With the arc lamps manually turned on and the FFS physically =
closed
>>>>> (despite not
>>>>> being registered as closed in the MCP), I commanded an arc with =
'boss
>>>>> exposure
>>>>> arc itime=3D4', but there was no light in the images. I watched =
the
>>>>> exposure
>>>>> sequence on a subsequent attempt and saw that the Shutter status
>>>>> stayed 'Closed'
>>>>> throughout the integration and instead of the Hartmann status =
showing
>>>>> 'Out' the
>>>>> Hartmann status was 'Right In'. For comparison, the sp1 Shutter =
and
>>>>> Hartmann
>>>>> keywords were reporting the correct values during the integration
>>>>> ('Open' and
>>>>> 'Out', respectively).
>>>>>=20
>>>>> I tried a 'boss openShutter spec=3Dsp2' command and the BOSS =
Shutter
>>>>> status did in
>>>>> fact change to 'Open'. I issued a 'closeShutter' and tried another =
arc
>>>>> to see if
>>>>> the shutter status would now appropriately behave since I was
>>>>> successful with
>>>>> the 'openShutter' and 'closeShutter', but the Shutter status =
remained
>>>>> 'Closed'.
>>>>>=20
>>>>> I also tried a 'boss hartmann out spec=3Dsp2' to see if I could =
manually
>>>>> pull the
>>>>> hartmann screens out, but this command timed-out and the sp2 =
Hartmann
>>>>> status
>>>>> remained 'Right In'.
>>>>>=20
>>>>> In short, sp1 Shutter and Hartmann behavior is normal during
>>>>> exposures, but the
>>>>> r1 images are *not* written to disk. The sp2 images *do* write to
>>>>> disk, but
>>>>> Shutter remains 'Closed' and Hartmann does not change to 'Out' =
during
>>>>> exposures.
>>>>>=20
>>>>> Started APOGEE morning calibration sequence and called it a night.
>>>>>=20
>>>>> ---- Telescope Status ----
>>>>>=20
>>>>> Status at: 08:51Z
>>>>>=20
>>>>> Telescope stowed at:    121.0, 30.0, 0.0 mount
>>>>> Instrument mounted:     cartridge =3D 14
>>>>> Counterweights at:      428 427 427 429
>>>>> LN2 autofill systems:   Connected and turned on
>>>>> 180L LN2 dewar scales:  SP1  81 lbs, 22 psi
>>>>>                     SP2 141 lbs, 20 psi
>>>>>=20
>>>>> ---- Software Used ----
>>>>>=20
>>>>> Disk space:
>>>>>=20
>>>>> /dev/vdd        688G  600G   89G  88% /home
>>>>> /dev/vdc1        11T  9.0T  1.8T  84% /data
>>>>>=20
>>>>> --- STUI software ---
>>>>> STUI          : 1.7.3
>>>>> alerts        : trunk
>>>>> apo           : trunk
>>>>> boss          : v4_0
>>>>> -boss.specMec : 3.0.4
>>>>> gcamera       : v2_2
>>>>> guider        : 3.9.0
>>>>> hartmann      : v1_5
>>>>> hub           : Tag: v4.2
>>>>> mcp           : v7_0_4
>>>>> -mcp.plc      : 34
>>>>> -mcp.azFiduci : v1_131
>>>>> -mcp.altFiduc : v1_131
>>>>> -mcp.rotFiduc : v1_131
>>>>> platedb       : v2_7
>>>>> sop           : 3.10.2
>>>>> tcc           : 1.3.1
>>>>> apogee        : not availble
>>>>> apogeecal     : not availble
>>>>> apogeeql      : v1_6_1
>>>>> APO-local     : trunk
>>>>>=20
>>>>> --- Other versions ---
>>>>> idlspec2d         : v5_7_12
>>>>> plate-mapper3     : NOCVS:v6_0_11
>>>>> petunia           : git
>>>>> autoscheduler     : V8_1_hack/
>>>>> sdss_python_module: trunk
>>>>> mangadrp          : trunk exported
>>>>>=20
>>>>> ---- APOGEE Data Summary ----
>>>>>=20
>>>>> APOGEE data list,   mjd=3D58376,  all
>>>>> raw_data:  /data/apogee/utr_cdr/58376/apRaw-*.fits
>>>>> archive:  /data/apogee/archive/58376/apR-[a,b,c]-*.apz
>>>>> quickred:  /data/apogee/archive/58376/apR-[a,b,c]-*.apz
>>>>> =
--------------------------------------------------------------------------=
------
>>>>> mjd     i   UT   File/Exp  Imtype Nread A/B Dth  Ct-Plate Arch =
Offset
>>>>> %Flux Comm
>>>>> =
--------------------------------------------------------------------------=
------
>>>>> 58376   1 03:01  28140001    Dark    3  A  13.0  14-8733  a-b-c    =
-
>>>>> -
>>>>> 58376   2 03:06  28140002    Dark   60  A  13.0  14-8733  a-b-c    =
-
>>>>> -
>>>>> 58376   3 03:17  28140003  QuarFlat 10  A  13.0  14-8733  a-b-c    =
-
>>>>> 51
>>>>> 58376   4 03:19  28140004  Arc-Thar 12  A  13.0  14-8733  a-b-c  =
0.07
>>>>> -
>>>>> 58376   5 03:21  28140005  Arc-Une  40  A  13.0  14-8733  a-b-c    =
-
>>>>> -
>>>>> 58376   6 03:28  28140006  Arc-Thar 12  B  13.5  14-8733  a-b-c  =
0.55
>>>>> -
>>>>> 58376   7 03:31  28140007  Arc-Une  40  B  13.5  14-8733  a-b-c    =
-
>>>>> -
>>>>> 58376   8 03:38  28140008    Dark   60  A  13.0  14-8733  a-b-c    =
-
>>>>> -
>>>>> =
--------------------------------------------------------------------------=
------
>>>>>=20
>>>>> ____________________________________________________________
>>>>> Logs of Nightly APO Activities
>>>>> =
https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html
>>>>=20
>>>> ____________________________________________________________
>>>> Logs of Nightly APO Activities
>>>> =
https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html
>>>=20
>>> ____________________________________________________________
>>> Logs of Nightly APO Activities >> >
>>> =
https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html
>>> =
>>=20
>=20
>=20