Subject: 2.5m obslog 20140908 (MJD 56909)
From: viktorm@apo.nmsu.edu
Date: Tue,  9 Sep 2014 11:11:19 +0000 (UTC)
                       APACHE POINT OBSERVATORY SDSS 2.5M
                                 OBSERVING LOG
                          Monday September 08, 2014
                                  (MJD 56909)
----  Observing Team ----
Dmitry Bizyaev
Viktor Malanushenko
Elena Malanushenko
---- Observing Plan ----
Mon  8 SEP 2014    56909.0   1.00    01:53  12:09    18:02  04:19     Both
APOGEE-2 (10.3 h)
---- Observing Summary ----
In the evening we had problem with apogeeql actor, which has been solved with no
observing time loss, see afternoon part of the log.
During the night,occurred twice:
"The boss ICC's been dead ...", see in the night part of log.
---- Observing Log ----
Afternoon:
----------
All hardware, software, telescope, guider, plates, actors checked.
Apogeegl actor found dead, see below.
Enclosure = closed;
cwPositions =  423  423  423  425;
MCP instr =  11  (mounted);
Guider cart= 11,  plate= 7826,  side= A (loaded);
----------
motorPosition
     sp1:   25295,   25176,   24649;
     sp2:   27577,   26840,   28016;
----------
Median spectrographs temperatures:
     Tsp1= 15.0,  Tsp2= 14.4;
     Outside air temperature= 15.3;
----------
Boss CCD Temp :
    sp1r0= -125.0  (-126.6);
    sp1b2=  -99.6  (-100.0);
    sp2r0= -133.6  (-134.8);
    sp2b2= -101.2  (-100.0);
         sp1LN2: 10.0;
         sp2LN2: 10.0;
----------
gcameraTemp= -40.0 (-40.0)
gcamera simulating = Off
Rel. Focus =  0.0;
objArcOff = 0.0,  0.0;
----------
Apogee:
  vacuum = 6.792e-07 (  85 )
----------
Weather:
  humid25m =  79.9;
  airTemp25m =  15.3C,  dpTemp25m =  11.8C;
      diff= 3.5C;
  wind25m = n/a,  direction = n/a
----------
semOwner = None
stuiVer=(1.4.1b5 2014-08-29),  pid = 34210
memory = 210 MB
-- done --
Collimation with cart 11.
==> Collimate-56909-b1-00184341.log  Collimate-56909-b2-00184341.log  Collimate-56909-r1-00184341.log  Collimate-56909-r2-00184341.log <==
COLLIMATE: Predict (red) piston movement of  -42 steps for spectro-2
COLLIMATE: new  expected    motorPosition 27535 26798 27974
According to the science plan, the numbers are good, so blue rings were not moved.
Apogeeql was not working: no CDR-images in /utr_cdr/, and also "archive"
was not created even after evening calibs finished (we noticed the problem after
the first exposure, but let it go).
We tried to follow standard procedures, but found that:
- we did not have pwd to "apogee" account on sdss4-apogee
- another account that we have, "observer" doesn't allow us to stop or start
Called Jon Holtzman
Got pwd from JH.
- scripts on the new machine can be ran, but they don't stop existing apogeeql,
and apparently, don't start either.
Had a conversation involving JH, JB, Nathan. JP is on the way home yet.
JB restarted apogeeql actor using standard stageManager. - it worked. Our
procedures require to run a script. (start_apogeeql, apogeeql start ; apogeeql stop)
Jon and Nathan think that something got wrong with IDL scripts on sdss4-apogee.
They bundle, they populate headers (which have sone NaN values now).
02:01Z  Note that we are not loosing time because of high humidity on site.
03:40Z to summarize. something went wrong with either IDL or python scripts
(including availability of external packages), and apogeeql actor stopped doing
its job. After a few dozen minutes the initial problem disappeared, but it took
time to restart the actor properly after that. No time was lost because of high
humidity on site.
Useful notes: do not restart apogeeql via stageManager !
Night:
------
Ventilation and opening delayed because of high humidity.
09:24:58 .alerts 0 alerts e alert=boss.heartbeat,critical,"The boss ICC's been
dead for at least 603s",enabled,noack,""
-- We made the "boss reloadControllers + boss clearExposure" & restart bossActor
10:56:06 .alerts 0 alerts e alert=boss.heartbeat,critical,"The boss ICC's been
dead for at least 604s",enabled,noack,""
We repeated again.
---- Software Used ----
TUI          : 1.4.1b5
alerts        : trunk+svn149374
apo           : trunk+svn153334
boss          : v3_2
-boss.daq     : 1.3.2
-boss.specMec : 3.0.4
gcamera       : v2_0
guider        : trunk+svn158216
hub           : Tag: sdss3_v1_2_0
mcp           : v6_6_18
-mcp.plc      : 29
-mcp.azFiduci : v1_112
-mcp.altFiduc : v1_112
-mcp.rotFiduc : v1_112
platedb       : v2_3
sop           : v2_5
sos           : FAILED
-sos.idlspec2 :   FAILED
tcc           : 2.14.0
apogee        : not availble
apogeecal     : not availble
apogeeql      : FAILED
  -- Other software
sdssProcedure : v3_06
APO-local     : trunk
  -- done --
---- Mirror Numbers ----
PRIMARY:
--------
Scale: 1.000000
MIGS		TONIGHT	NOMINAL
Axial A:	0.035	0.037
Axial B:	-0.022	-0.042
Axial C:	-0.019	-0.022
Trans D:	3.21	3.21
Lateral E:	10.7188	10.7188
Lateral F:	12.3063	12.3063
GALILS
Commanded:  10850 -450 5750 13050 14750 14300
Actual:     10850 -450 5750 13050 14750 14300
SETMIR VALUES
PrimDesOrient: 0.00 0.00 0.00 651.00 937.40
PrimOrient:    0.00 0.00 0.00 651.08 936.54
SECONDARY:
----------
Focus:      0.00
Air temp:   13.2
Altitude:   30.2
MIGS		TONIGHT	NOMINAL
Axial A:	-7.061	-7.063
Axial B:	-7.121	-7.128
Axial C:	-7.032	-7.034
Trans D:	-0.178	-0.179
GALILS
Commanded: 1581300 1578850 1590700 1400 -2000
Actual:    1581337 1578874 1590691 1400 -2000
SETMIR VALUES
SecDesOrient: 1257.00 -3.80 0.00 0.00 9.58
SecOrient:    1257.03 -3.81 0.00 0.00 9.44