Subject: 2.5m obslog 20130114 (MJD 56307) From: kinemuchi@apo.nmsu.edu Date: Tue, 15 Jan 2013 12:20:58 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday January 14, 2013 (MJD 56307) ---- Observing Team ---- Viktor Malanushenko Karen Kinemuchi Kaike Pan (afternoon) Connie Rockosi (02:45Z-02:56Z) Joe Huehnerhoff (on the phone at 05:40Z) ---- Observing Plan ---- 14 Jan 2013 56307.0 0.26 01:31 12:54 02:07 13:31 BOSS 11.4 hours ---- Observing Summary ---- BOSS Observing: 00:00Z Closed due to temperatures below -10C 12:00Z Never opened due to temperature. Humidity rose in the last 2 hours of the night. 02:45Z Connie Rockosi called us in regards to the warning alert: boss w SP1LN2EmptyLatched=1358124872.39 boss w camCheck=SP1LN2EmptyLatched We resolved this problem by 03:00Z by issuing the boss command: boss ackErrors cams=sp1cam. Please see the Observing log for more details. Note to engineering staff: 05:00Z - We received a critical and serious alert for Sp1 for low pressure. We initiated a startfillseq and a force fill. Pressure on the secondary dewars were reading 5 lbs psi, but after force fill, it is up to 10 lbs psi. We called Joe for advice and we will monitor the pressure every 2 hours as well as do a force fill at the end of the night. Please see the Observing log and Problems in Detail for more details. ---- Observing Log ---- 00:00Z Snowing, not ventilating. Collimate: ============ cwPositions = 281 282 281 281; MCP instr = 11 (mounted); Guider cart= 11, plate= 5400, side= A (loaded); new hartmann is 154492, 154493 spCollimate 00154492 COLLIMATE: Predict blue camera ring movement of -4.7 degrees for spectro-1 COLLIMATE: Predict blue camera ring movement of 5.4 degrees for spectro-2 COLLIMATE: Predict (red) piston movement of 0 steps for spectro-1 COLLIMATE: Predict (red) piston movement of 0 steps for spectro-2 00:19Z Collimate is OK. -- Status -- 00:20:42 ---- Enclosure = closed; cwPositions = 282 281 281 278; MCP instr = 11 (mounted); Guider cart= 11, plate= 5400, side= A (loaded); ------------------------------ motorPosition sp1: -4620, -4624, -4919; sp2: -3831, -5017, -5132; ------------------------------ Median spectrographs temperatures: Tsp1= -3.2, Tsp2= -3.9; Outside air temperature= -10.1; ------------------------------ All systems and actors were checked, no problem to report. ========================= EVENING ======= Getting a warning alert with the BOSS instrument: 23:32:39 .boss 60713 boss w SP1LN2EmptyLatched=1358124872.39 23:32:39 .boss 60713 boss w camCheck=SP1LN2EmptyLatched 23:33:39 .boss 60715 boss w SP1LN2EmptyLatched=1358124872.39 23:33:39 .boss 60715 boss w camCheck=SP1LN2EmptyLatched 23:33:39 .alerts 0 alerts w text="sent MAIL WARN boss.SP1LN2EmptyLatched == Reported by camCheck" 23:34:39 .boss 60717 boss w SP1LN2EmptyLatched=1358124872.39 We have sent email to Connie and JohnP to get clarification on this warning. 01:45Z Still closed due to temperature. -11.8C! 02:45Z Connie Rockosi called in to help trouble shoot the warning alert about the sp1ln2EmptyLatched. Connie says that the instrument remembered the past "empty dewars" for Sp1 and was just complaining about it. We checked the status of the temperatures of the spectrograph by issuing the command: boss sp1cam raw=specstat boss sp1cam raw=camcheck We were able to resolve it by issuing the command: boss ackErrors cams=sp1cam This command cleared the "sticky and latched errors" and we are now no longer receiving the warnings. Problem resolved by 02:56Z. Thanks Connie! 04:59:40 .alerts 0 alerts e alert=boss.SP1SecondaryDewarPress,critical,"Reported by camCheck",enabled,noack,"" We issued a "startFillSeq" from the control room and then went down to the telescope to check. We found that the auto fill started on its own and the Sp1 dewar is filling as of 05:02Z. We started getting the following warning: 05:03:41 .boss 61411 boss w SP1LN2Fill=FAULT!!! 05:03:41 .boss 61411 boss w camCheck=SP1LN2Fill 05:03:41 .alerts 0 alerts w alert=boss.SP1LN2Fill,serious,"Reported by camCheck",enabled,noack,"" 05:04:40 .boss 61413 boss w SP1LN2Fill=FAULT!!! 05:04:40 .boss 61413 boss w camCheck=SP1LN2Fill We went down to the telescope again and did a force fill. The warning still appears. We called Joe at 05:45Z for advice on this warning. We checked the secondary dewar pressure. Before the force fill, it was at 5 (SP1) and 6 (SP2) lbs. After the forcefill, the pressure is now at 10.0psi for both cameras. Joe suspects this is related to the earlier problem reported with the low ambient temperature (we are at -13.4C) not allowing for a proper boil-off to have sufficient pressure. The day staff will look into this in the morning. We will check the pressure every two hours to monitor this pressure problem and we will issue another force fill in the morning. This is logged under Problems in Detail. The "boss w SP1LN2Fill=FAULT!!!" warning stopped showing up around 07:01Z. 08:00Z APOGEE morning calibrations started. 09:33Z APOGEE auto-scheduler has completed. 12:00Z We are calling it a night. The humidity has risen over 90% and the outside temperature is at -15.4C. Starting BOSS morning calibrations and doing a force-fill for the dewars. ---- Spectroscopy Data Summary ---- Reading FITS from /data/spectro/56307/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n time filename cart plt flav hrt exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 22:16 b1-00154484 11 5400 bia Clo 0.0 -5175 -5179 -5474 22:16 b2-00154484 11 5400 bia Clo 0.0 -11260 -12446 -12561 22:16 r1-00154484 11 5400 bia Clo 0.0 -5175 -5179 -5474 22:16 r2-00154484 11 5400 bia Clo 0.0 -11260 -12446 -12561 2 22:17 b1-00154485 11 5400 dar Clo 20.0 -5175 -5179 -5474 22:17 b2-00154485 11 5400 dar Clo 20.0 -11260 -12446 -12561 22:17 r1-00154485 11 5400 dar Clo 20.0 -5175 -5179 -5474 22:17 r2-00154485 11 5400 dar Clo 20.0 -11260 -12446 -12561 3 22:22 b1-00154486 11 5400 arc Lef 4.8 -5175 -5179 -5474 HART 22:22 b2-00154486 11 5400 arc Lef 4.8 -11260 -12446 -12561 HART 22:22 r1-00154486 11 5400 arc Lef 4.8 -5175 -5179 -5474 HART 22:22 r2-00154486 11 5400 arc Lef 4.8 -11260 -12446 -12561 HART 4 22:24 b1-00154487 11 5400 arc Rig 4.9 -5175 -5179 -5474 HART 22:24 b2-00154487 11 5400 arc Rig 4.9 -11260 -12446 -12561 HART 22:24 r1-00154487 11 5400 arc Rig 4.9 -5175 -5179 -5474 HART 22:24 r2-00154487 11 5400 arc Rig 4.9 -11260 -12446 -12561 HART 5 23:52 b1-00154488 11 5400 arc Lef 4.8 -5175 -5179 -5474 HART 23:52 b2-00154488 11 5400 arc Lef 4.8 -11260 -12446 -12561 HART 23:52 r1-00154488 11 5400 arc Lef 4.8 -5175 -5179 -5474 HART 23:52 r2-00154488 11 5400 arc Lef 4.8 -11260 -12446 -12561 HART 6 23:54 b1-00154489 11 5400 arc Rig 4.9 -5175 -5179 -5474 HART 23:54 b2-00154489 11 5400 arc Rig 4.9 -11260 -12446 -12561 HART 23:54 r1-00154489 11 5400 arc Rig 4.9 -5175 -5179 -5474 HART 23:54 r2-00154489 11 5400 arc Rig 4.9 -11260 -12446 -12561 HART 7 00:01 b1-00154490 11 5400 arc Lef 4.8 -5345 -5349 -5644 HART 00:01 b2-00154490 11 5400 arc Lef 4.9 -4727 -5912 -6028 HART 00:01 r1-00154490 11 5400 arc Lef 4.8 -5345 -5349 -5644 HART 00:01 r2-00154490 11 5400 arc Lef 4.9 -4727 -5912 -6028 HART 8 00:03 b1-00154491 11 5400 arc Rig 4.8 -5345 -5349 -5644 HART 00:03 b2-00154491 11 5400 arc Rig 4.9 -4727 -5912 -6028 HART 00:03 r1-00154491 11 5400 arc Rig 4.8 -5345 -5349 -5644 HART 00:03 r2-00154491 11 5400 arc Rig 4.9 -4727 -5912 -6028 HART 9 00:09 b1-00154492 11 5400 arc Lef 4.8 -4620 -4624 -4919 HART 00:09 b2-00154492 11 5400 arc Lef 4.9 -3831 -5017 -5132 HART 00:09 r1-00154492 11 5400 arc Lef 4.8 -4620 -4624 -4919 HART 00:09 r2-00154492 11 5400 arc Lef 4.9 -3831 -5017 -5132 HART 10 00:11 b1-00154493 11 5400 arc Rig 4.8 -4620 -4624 -4919 HART 00:11 b2-00154493 11 5400 arc Rig 4.9 -3831 -5017 -5132 HART 00:11 r1-00154493 11 5400 arc Rig 4.8 -4620 -4624 -4919 HART 00:11 r2-00154493 11 5400 arc Rig 4.9 -3831 -5017 -5132 HART 11 00:25 b1-00154494 11 5400 fla Out 30.8 -4620 -4624 -4919 00:25 b2-00154494 11 5400 fla Out 30.9 -3831 -5017 -5132 00:25 r1-00154494 11 5400 fla Out 30.8 -4620 -4624 -4919 00:25 r2-00154494 11 5400 fla Out 30.9 -3831 -5017 -5132 12 00:29 b1-00154495 11 5400 arc Out 4.8 -4620 -4624 -4919 00:29 b2-00154495 11 5400 arc Out 4.9 -3831 -5017 -5132 00:29 r1-00154495 11 5400 arc Out 4.8 -4620 -4624 -4919 00:29 r2-00154495 11 5400 arc Out 4.9 -3831 -5017 -5132 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --- OK --- Reading FITS from /data/apogee/quickred/56307/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - N UT File/Exp Cart Plate Imagetype Dth Nread Comment - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 08:05 07450001 11 5400 Dark A 60 None 2 08:16 07450002 11 5400 Dark A 60 None 3 08:26 07450003 11 5400 Dark A 60 None 4 08:37 07450004 11 5400 QuartzFlat A 10 None 5 08:39 07450005 11 5400 QuartzFlat A 10 None 6 08:41 07450006 11 5400 QuartzFlat A 10 None 7 08:43 07450007 11 5400 ArcLamp A 12 None 8 08:45 07450008 11 5400 ArcLamp A 40 None 9 08:52 07450009 11 5400 ArcLamp B 12 None 10 08:54 07450010 11 5400 ArcLamp B 40 None 11 09:02 07450011 11 5400 Dark A 30 None 12 09:07 07450012 11 5400 InternalFl A 30 None 13 09:12 07450013 11 5400 InternalFl A 30 None 14 09:18 07450014 11 5400 InternalFl A 30 None 15 09:23 07450015 11 5400 Dark A 30 None - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --- OK --- ---- Telescope Offsets and Scale ---- none ---- Focus Log ---- none ---- Weather Log ---- none ---- Telescope Status ---- Never opened No interlocks bypassed Status at: 00:46:19 Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 11 Counterweights at: 282, 281, 281, 278 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 127.8 lbs, 19.9 psi SP2 119.5 lbs, 20.5 psi ---- Software Used ---- getVersions, 2013-01-15 00:48:35 ----STUI software: STUI: 1.2 2012-12-19; alerts: v1_2_1; apo: v1_7; boss: v2_5_13; boss.daq: 1.3.2; boss.specMech: 3.0.1; gcamera: v1_2_10; guider: v1_4_1; hub: Tag: sdss3_v1_2_0; mcp: v6_6_16; mcp.plc: 28; mcp.azFiducialVersion: v1_107; mcp.altFiducialVersion: v1_107; mcp.rotFiducialVersion: v1_107; platedb: v1_0_5; sop: v1_2_16; sos: v1_2_0; sos.idlspec2d: v5_6_2; tcc: 2.14.0; apogee: ** not availble ** apogeecal: ** not availble ** apogeeql: v1_4_1 -----Other software sdssProcedure(sdss3): v3_06; - - - - - - - - - - - - - - - - - - - - Setting up: idlspec2d Flavor: Linux64 Version: v5_6_2 Setting up: |elodie Flavor: Linux64 Version: v1_3 Setting up: |spinspect Flavor: Linux64 Version: start Setting up: |specflat Flavor: NULL Version: v1_19 Setting up: |speclog Flavor: Linux64 Version: trunk Setting up: |platelist Flavor: Linux64 Version: trunk Setting up: |platedb Flavor: Linux64 Version: v1_3_1 Setting up: |hooloovookit Flavor: Linux64 Version: v3_4_1 Setting up: |sqlalchemy Flavor: Linux64 Version: 0.7.7 Setting up: |scipy Flavor: Linux64 Version: 0.7.1 Setting up: |psycopg Flavor: Linux64 Version: 2.0.11 Setting up: |actorcore Flavor: Linux64 Version: v1_17 Setting up: |opscore Flavor: Linux64 Version: v1_16 Setting up: |external Flavor: Linux64 Version: v1_2_0 Setting up: |actorkeys Flavor: Linux64 Version: trunk Setting up: |plc Flavor: Linux64 Version: v28_5 Setting up: |sdss3tools Flavor: Linux64 Version: v1_7_12 Setting up: |RO Flavor: Linux64 Version: RO-2.8.0 Setting up: |tron Flavor: Linux64 Version: sdss3_v1_1_9 Setting up: |idlutils Flavor: Linux64 Version: v5_5_8 Setting up: |idl Flavor: Linux64 Version: 7.1 Setting up: |dust Flavor: Linux64 Version: v0_1 Setting up: actorcore Flavor: Linux64 Version: v1_17 Setting up: |opscore Flavor: Linux64 Version: v1_16 Setting up: |external Flavor: Linux64 Version: v1_2_0 Setting up: |actorkeys Flavor: Linux64 Version: trunk Setting up: |plc Flavor: Linux64 Version: v28_5 Setting up: |sdsstools Flavor: Linux64 Version: v4_33 Setting up: |sdss3tools Flavor: Linux64 Version: v1_7_12 Setting up: |RO Flavor: Linux64 Version: RO-2.8.0 Setting up: |tron Flavor: Linux64 Version: sdss3_v1_1_9 -- Other versions ---- idlspec2d (sos3) --- v5_6_2 plate-mapper3 ------ NOCVS:v5_0_16 ---- Mirror Numbers ---- PRIMARY: -------- Scale: 0.999905 MIGS TONIGHT NOMINAL Axial A: 4.413 5.357 Axial B: 4.273 5.221 Axial C: 4.576 5.516 Trans D: 10.335 10.374 Lateral E: 10.6426 10.6299 Lateral F: 12.2809 12.2428 GALILS Commanded: -3450 -15400 -9250 21900 13500 13100 Actual: -3450 -15400 -9250 21900 13500 13100 SETMIR VALUES PrimDesOrient: 939.61 0.00 0.00 612.35 1500.90 PrimOrient: 939.73 0.00 0.00 612.21 1500.38 SECONDARY: ---------- Focus: 588.19 Air temp: -10.9 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: 8.402 7.041 Axial B: 8.42 7.059 Axial C: 8.394 7.041 Trans D: 0.358 0.33 GALILS Commanded: 3297750 3315750 3301150 -3350 -6500 Actual: 3298409 3323820 3303039 -3350 -6500 SETMIR VALUES SecDesOrient: 2623.18 3.00 5.00 0.00 115.81 SecOrient: 2626.00 4.30 7.03 1.66 114.78 ---- Problems in Detail ---- Status log of the secondary dewar this evening: This is when we were getting the warning :"boss 61477 boss w SP1LN2Fill=FAULT!!!" -- Status -- 05:33:10 ---- Median spectrographs temperatures: Tsp1= -3.8, Tsp2= -4.6; Outside air temperature= -13.2; ------------------------------ Boss CCD Temp : sp1r0= -125.0 (-126.6); sp1b2= -100.0 (-100.0); sp2r0= -134.8 (-134.8); sp2b2= -101.2 (-100.0); sp1LN2: 5.0; sp2LN2: 6.0; After doing the force-fill: - Status -- 06:04:39 ---- Median spectrographs temperatures: Tsp1= -3.9, Tsp2= -4.7; Outside air temperature= -13.4; ------------------------------ Boss CCD Temp : sp1r0= -124.6 (-126.6); sp1b2= -100.0 (-100.0); sp2r0= -134.8 (-134.8); sp2b2= -101.2 (-100.0); sp1LN2: 10.0; sp2LN2: 10.0; Checking every 2 hours: 07:00Z We stopped getting that warning. -- Status -- 07:20:08 ---- Median spectrographs temperatures: Tsp1= -4.0, Tsp2= -4.7; Outside air temperature= -13.2; ------------------------------ Boss CCD Temp : sp1r0= -125.0 (-126.6); sp1b2= -100.0 (-100.0); sp2r0= -134.8 (-134.8); sp2b2= -101.2 (-100.0); sp1LN2: 10.0; sp2LN2: 10.0; -- Status -- 09:05:28 ---- Median spectrographs temperatures: Tsp1= -4.0, Tsp2= -4.6; Outside air temperature= -13.6; ------------------------------ Boss CCD Temp : sp1r0= -124.6 (-126.6); sp1b2= -100.0 (-100.0); sp2r0= -134.8 (-134.8); sp2b2= -101.2 (-100.0); sp1LN2: 10.0; sp2LN2: 9.0; -- Status -- 11:11:50 ---- Median spectrographs temperatures: Tsp1= -4.0, Tsp2= -4.7; Outside air temperature= -14.9; ------------------------------ Boss CCD Temp : sp1r0= -125.0 (-126.6); sp1b2= -100.0 (-100.0); sp2r0= -134.8 (-134.8); sp2b2= -101.2 (-100.0); sp1LN2: 11.0; sp2LN2: 10.0; - Status -- 12:10:18 ---- Median spectrographs temperatures: Tsp1= -4.1, Tsp2= -4.7; Outside air temperature= -15.4; ------------------------------ Boss CCD Temp : sp1r0= -125.0 (-126.6); sp1b2= -99.6 (-100.0); sp2r0= -134.4 (-134.8); sp2b2= -101.2 (-100.0); sp1LN2: 10.0; sp2LN2: 10.0; ------------------------------