Subject: 2.5m obslog 20130510 (MJD 56423) From: kinemuchi@apo.nmsu.edu Date: Sat, 11 May 2013 11:02:53 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday May 10, 2013 (MJD 56423) ---- Observing Team ---- Kaike Pan Karen Kinemuchi Fritz Stauffer (hub25m reboot help) John Parejko (help with the aftermath of hub25m reboot; boss-icc heartbeat issues, and mcp log issues) Elena Malanushenko (help with mcp reboot) Jon Brinkman (help with mcp log issues) Craig Loomis (help with boss-icc critical alert issue) ---- Observing Plan ---- 10 May 2013 56423.0 0.08 03:08 10:51 11:21 19:05 BOSS 7.7 hours ---- Observing Summary ---- Entire night lost to clouds and high humidity. Lots of BOSS and APOGEE test calib data taken while systems were being brought back up after the hub25m reboot. Please see the observing log for details of the aftermath of rebooting hub25m. All systems seems to be now back up and running. However, we can't explain why the critical alert emails are still being sent out about the boss-icc heartbeat. Nothing is wrong with the boss-icc (no obvious warnings/errors from the logs). JohnP and Craig are on the case. ---- Observing Log ---- AFTERNOON CHECKOUT: ================== 23:54Z Heavy cloud cover and lots of radar return. 00:32Z Working with JohnP to restart several processes that have not come up or reconnected after the afternoon reboot of hub25m. Apogee-ql was rebooted, and we were able to successfully get the ThreeReadDark done. 00:35Z We restarted the sos process on sos3. The test bias (file 161507) we took after this restart is finally registered on the sos webpage. All previous biases taken before 00:30Z did not show up on the webpage. We have re-established the connection between hub25m and sos. We are taking a test arc to make sure there are no mechanical problems. We also tried to do a hard reboot of mcp from mcpMenu. The mcpMenu is running slow on minidru. We had been receiving boss-icc critical alerts, but no audio warnings. We ran the procedure to correct for this alert, but the warning emails are still being issued. There is no indication of any problems with the boss-icc (i.e. the outputs from boss camCheck and boss reloadControllers). JohnP is on the case. 00:46Z Started APOGEE evening cals and checking if connections are okay. Test BOSS arc failed. Still working out what is wrong with BOSS. 00:51Z Prepping scale and focus: getting the warnings: 00:50:13 APO.karen 70 tcc W Text="altitude unknown; new focus and/or scale recorded but not applied" 00:50:27 APO.karen 71 tcc W Text="altitude unknown; new focus and/or scale recorded but not applied" 00:59Z Actors tested - ok. gcamera/guider prep - done 02:31Z Rebooted apogee-ql, boss-icc, and tcc. We are still getting critical alert emails from the boss-icc. JohnP is still on the case. We've taken a set of BOSS calibs and all the files look fine and were displayed on the SOS webpage. mcp was rebooted again; this time pushing the button on the mcp crate. Elena was on hand to help with the mcp reboot. 02:32Z Finished sop hartmanns: COLLIMATE: Camera b1 appears to be IN-FOCUS (|mean| 85 ) ---------- Weather: humid25m = 87.0; airTemp25m = 2.4C, dpTemp25m = 0.5C; diff= 1.9C; wind25m = n/a, direction = n/a ---------- semOwner = None stuiVer=(1.3b4 2013-03-29), pid = 31740 memory = 754 MB 03:12Z JonB and JohnP are now working on the problem of why the mcp log isn't being written since the hub25m reboot. Three of the mcp directories mounted, but the 4th did not. We are still receiving boss-icc heartbeat is dead critical alert messages. JohnP has sent a message to Craig for more help. 03:38Z MCP log problem has been fixed by JonB and JohnP. We are running a set of BOSS and APOGEE calibs to see if everything is okay. 03:34:21 boss exposure 0.0, file=161519 03:35:43 boss exposure 20.0, file=161520 03:36:38 apogee.expose=apRaw-08610012, 3, A, Dark 03:37:26 boss exposure 30.0, file=161521 03:38:07 apogee.expose=apRaw-08610013, 10, A, QuartzFlat 03:40:05 apogee.expose=apRaw-08610014, 12, A, ArcLamp 03:41:49 boss exposure 4.0, file=161522 03:42:18 apogee.expose=apRaw-08610015, 40, A, ArcLamp 03:49:31 apogee.expose=apRaw-08610016, 12, B, ArcLamp 03:51:43 apogee.expose=apRaw-08610017, 40, B, ArcLamp 03:59:16 apogee.expose=apRaw-08610018, 10, A, Dark Images of the calibs all look okay! 04:05Z Testing to see if the telescope will move after all the mcp reboots. Discovered that we need to set the instrument block, which fixed the STUI status window. We were able to move the telescope. 05:14Z High humidity. Not ventilating. 07:30Z Craig online trying to fix the BOSS-icc critical alert message problem. 10:38Z High humidity and clouds. Giving up for the night. Sorry about all the critical alert emails for the boss-icc! BOSS cals started, auto-scheduler run. ---- Spectroscopy Data Summary ---- Reading FITS from /data/spectro/56423/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n time filename cart plt flav hrt exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 21:05 b1-00161502 10 6972 bia Clo 0.0 2605 2556 2634 21:05 b2-00161502 10 6972 bia Clo 0.0 5858 4617 4687 21:05 r1-00161502 10 6972 bia Clo 0.0 2605 2556 2634 21:05 r2-00161502 10 6972 bia Clo 0.0 5858 4617 4687 2 21:07 b1-00161503 10 6972 dar Clo 20.0 2605 2556 2634 21:07 b2-00161503 10 6972 dar Clo 20.0 5858 4617 4687 21:07 r1-00161503 10 6972 dar Clo 20.0 2605 2556 2634 21:07 r2-00161503 10 6972 dar Clo 20.0 5858 4617 4687 3 21:37 b1-00161504 10 6972 bia Clo 0.0 2605 2556 2634 21:37 b2-00161504 10 6972 bia Clo 0.0 5858 4617 4687 21:37 r1-00161504 10 6972 bia Clo 0.0 2605 2556 2634 21:37 r2-00161504 10 6972 bia Clo 0.0 5858 4617 4687 4 22:38 b1-00161505 10 6972 bia Clo 0.0 2605 2556 2634 22:38 b2-00161505 10 6972 bia Clo 0.0 5858 4617 4687 22:38 r1-00161505 10 6972 bia Clo 0.0 2605 2556 2634 22:38 r2-00161505 10 6972 bia Clo 0.0 5858 4617 4687 5 00:17 b1-00161506 10 6972 bia Clo 0.0 2605 2556 2634 00:17 b2-00161506 10 6972 bia Clo 0.0 5858 4617 4687 00:17 r1-00161506 10 6972 bia Clo 0.0 2605 2556 2634 00:17 r2-00161506 10 6972 bia Clo 0.0 5858 4617 4687 6 00:32 b1-00161507 10 6972 bia Clo 0.0 2605 2556 2634 00:32 b2-00161507 10 6972 bia Clo 0.0 5858 4617 4687 00:32 r1-00161507 10 6972 bia Clo 0.0 2605 2556 2634 00:32 r2-00161507 10 6972 bia Clo 0.0 5858 4617 4687 7 00:39 b1-00161508 10 6972 fla Out 30.8 2605 2556 2634 00:39 b2-00161508 10 6972 fla Out 30.9 5858 4617 4687 00:39 r1-00161508 10 6972 fla Out 30.8 2605 2556 2634 00:39 r2-00161508 10 6972 fla Out 30.9 5858 4617 4687 8 01:04 b1-00161509 10 6972 fla Out 30.9 2605 2556 2634 01:04 b2-00161509 10 6972 fla Out 30.9 5858 4617 4687 01:04 r1-00161509 10 6972 fla Out 30.9 2605 2556 2634 01:04 r2-00161509 10 6972 fla Out 30.9 5858 4617 4687 9 01:08 b1-00161510 10 6972 arc Out 4.8 2605 2556 2634 01:08 b2-00161510 10 6972 arc Out 4.9 5858 4617 4687 01:08 r1-00161510 10 6972 arc Out 4.8 2605 2556 2634 01:08 r2-00161510 10 6972 arc Out 4.9 5858 4617 4687 10 02:02 b1-00161511 10 6972 bia Clo 0.0 2605 2556 2634 02:02 b2-00161511 10 6972 bia Clo 0.0 5858 4617 4687 02:02 r1-00161511 10 6972 bia Clo 0.0 2605 2556 2634 02:02 r2-00161511 10 6972 bia Clo 0.0 5858 4617 4687 11 02:03 b1-00161512 10 6972 dar Clo 20.0 2605 2556 2634 02:03 b2-00161512 10 6972 dar Clo 20.0 5858 4617 4687 02:03 r1-00161512 10 6972 dar Clo 20.0 2605 2556 2634 02:03 r2-00161512 10 6972 dar Clo 20.0 5858 4617 4687 12 02:05 b1-00161513 10 6972 fla Out 30.9 2605 2556 2634 02:05 b2-00161513 10 6972 fla Out 30.9 5858 4617 4687 02:05 r1-00161513 10 6972 fla Out 30.9 2605 2556 2634 02:05 r2-00161513 10 6972 fla Out 30.9 5858 4617 4687 13 02:09 b1-00161514 10 6972 arc Out 4.9 2605 2556 2634 02:09 b2-00161514 10 6972 arc Out 4.9 5858 4617 4687 02:09 r1-00161514 10 6972 arc Out 4.9 2605 2556 2634 02:09 r2-00161514 10 6972 arc Out 4.9 5858 4617 4687 14 02:20 b1-00161515 10 6972 arc Lef 4.8 2605 2556 2634 HART 02:20 b2-00161515 10 6972 arc Lef 4.9 5858 4617 4687 HART 02:20 r1-00161515 10 6972 arc Lef 4.8 2605 2556 2634 HART 02:20 r2-00161515 10 6972 arc Lef 4.9 5858 4617 4687 HART 15 02:22 b1-00161516 10 6972 arc Rig 4.9 2605 2556 2634 HART 02:22 b2-00161516 10 6972 arc Rig 4.9 5858 4617 4687 HART 02:22 r1-00161516 10 6972 arc Rig 4.9 2605 2556 2634 HART 02:22 r2-00161516 10 6972 arc Rig 4.9 5858 4617 4687 HART 16 02:27 b1-00161517 10 6972 arc Lef 4.9 2477 2428 2506 HART 02:27 b2-00161517 10 6972 arc Lef 4.9 4622 3381 3449 HART 02:27 r1-00161517 10 6972 arc Lef 4.9 2477 2428 2506 HART 02:27 r2-00161517 10 6972 arc Lef 4.9 4622 3381 3449 HART 17 02:28 b1-00161518 10 6972 arc Rig 4.9 2477 2428 2506 HART 02:28 b2-00161518 10 6972 arc Rig 4.9 4622 3381 3449 HART 02:28 r1-00161518 10 6972 arc Rig 4.9 2477 2428 2506 HART 02:28 r2-00161518 10 6972 arc Rig 4.9 4622 3381 3449 HART 18 03:34 b1-00161519 10 6972 bia Clo 0.0 2477 2428 2506 03:34 b2-00161519 10 6972 bia Clo 0.0 4622 3381 3449 03:34 r1-00161519 10 6972 bia Clo 0.0 2477 2428 2506 03:34 r2-00161519 10 6972 bia Clo 0.0 4622 3381 3449 19 03:35 b1-00161520 10 6972 dar Clo 20.0 2477 2428 2506 03:35 b2-00161520 10 6972 dar Clo 20.0 4622 3381 3449 03:35 r1-00161520 10 6972 dar Clo 20.0 2477 2428 2506 03:35 r2-00161520 10 6972 dar Clo 20.0 4622 3381 3449 20 03:37 b1-00161521 10 6972 fla Out 30.8 2477 2428 2506 03:37 b2-00161521 10 6972 fla Out 30.9 4622 3381 3449 03:37 r1-00161521 10 6972 fla Out 30.8 2477 2428 2506 03:37 r2-00161521 10 6972 fla Out 30.9 4622 3381 3449 21 03:41 b1-00161522 10 6972 arc Out 4.8 2477 2428 2506 03:41 b2-00161522 10 6972 arc Out 4.9 4622 3381 3449 03:41 r1-00161522 10 6972 arc Out 4.8 2477 2428 2506 03:41 r2-00161522 10 6972 arc Out 4.9 4622 3381 3449 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --- OK --- Reading FITS from /data/apogee/quickred/56423/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - N UT File/Exp Cart Plate Imagetype Dth Nread Comment Archiving - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 00:30 08610004 10 6972 Dark A 3 None a-b-c 2 00:45 08610005 10 6972 Dark A 60 None a-b-c 3 00:56 08610006 10 6972 QuartzFlat A 10 None a-b-c 4 00:57 08610007 10 6972 ArcLamp A 12 None a-b-c 5 01:00 08610008 10 6972 ArcLamp A 40 None a-b-c 6 01:07 08610009 10 6972 ArcLamp B 12 None a-b-c 7 01:09 08610010 10 6972 ArcLamp B 40 None a-b-c 8 01:17 08610011 10 6972 Dark A 60 None a-b-c 9 03:36 08610012 10 6972 Dark A 3 None a-b-c 10 03:37 08610013 10 6972 QuartzFlat A 10 None a-b-c 11 03:39 08610014 10 6972 ArcLamp A 12 None a-b-c 12 03:42 08610015 10 6972 ArcLamp A 40 None a-b-c 13 03:49 08610016 10 6972 ArcLamp B 12 None a-b-c 14 03:51 08610017 10 6972 ArcLamp B 40 None a-b-c 15 03:59 08610018 10 6972 Dark A 10 None a-b-c - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ---- Telescope Status ---- Never opened. No interlocks bypassed Status at: 10:39:44 Telescope stowed at: 121.0, 30.4, 0.0 mount Instrument mounted: cartridge = 10 Counterweights at: 286, 285, 286, 285 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 195.3 lbs, 21.5 psi SP2 188.3 lbs, 21.6 psi --------------- ---- Software Used ---- Get Versions, 00:48:00 ----STUI software: STUI: 1.3b4 2013-03-29 alerts: v1_2_1 apo: v1_7 boss: v2_5_13 boss.daq: 1.3.2 boss.specMech: 3.0.1 gcamera: v1_2_10 guider: v1_4_5 hub: Tag: sdss3_v1_1_9 mcp: v6_6_16 mcp.plc: 28 mcp.azFiducialVersion: v1_107 mcp.altFiducialVersion: v1_107 mcp.rotFiducialVersion: v1_107 platedb: v1_0_5 sop: v1_2_18 sos: v1_2_0 sos.idlspec2d: v5_6_3 tcc: 2.14.0 apogee: ** not availble ** apogeecal: ** not availble ** apogeeql: v1_4_1 -----Other software sdssProcedure(sdss3): v3_06 Scripts : v3_0+trunk -- done -- Setting up: idlspec2d Flavor: Linux64 Version: v5_6_4 Setting up: |elodie Flavor: Linux64 Version: v1_3 Setting up: |spinspect Flavor: Linux64 Version: start Setting up: |specflat Flavor: NULL Version: v1_19 Setting up: |speclog Flavor: Linux64 Version: trunk Setting up: |platelist Flavor: Linux64 Version: trunk Setting up: |platedb Flavor: Linux64 Version: v1_3_1 Setting up: |hooloovookit Flavor: Linux64 Version: v3_4_1 Setting up: |sqlalchemy Flavor: Linux64 Version: 0.7.7 Setting up: |scipy Flavor: Linux64 Version: 0.7.1 Setting up: |psycopg Flavor: Linux64 Version: 2.0.11 Setting up: |actorcore Flavor: Linux64 Version: v1_17 Setting up: |opscore Flavor: Linux64 Version: v1_16 Setting up: |external Flavor: Linux64 Version: v1_2_0 Setting up: |actorkeys Flavor: Linux64 Version: trunk Setting up: |plc Flavor: Linux64 Version: v28_5 Setting up: |sdss3tools Flavor: Linux64 Version: v1_7_12 Setting up: |RO Flavor: Linux64 Version: RO-2.8.0 Setting up: |tron Flavor: Linux64 Version: sdss3_v1_1_9 Setting up: |idlutils Flavor: Linux64 Version: v5_5_8 Setting up: |idl Flavor: Linux64 Version: 7.1 Setting up: |dust Flavor: Linux64 Version: v0_1 Setting up: actorcore Flavor: Linux64 Version: v1_17 Setting up: |opscore Flavor: Linux64 Version: v1_16 Setting up: |external Flavor: Linux64 Version: v1_2_0 Setting up: |actorkeys Flavor: Linux64 Version: trunk Setting up: |plc Flavor: Linux64 Version: v28_5 Setting up: |sdsstools Flavor: Linux64 Version: v4_33 Setting up: |sdss3tools Flavor: Linux64 Version: v1_7_12 Setting up: |RO Flavor: Linux64 Version: RO-2.8.0 Setting up: |tron Flavor: Linux64 Version: sdss3_v1_1_9 -- Other versions ---- idlspec2d (sos3) --- v5_6_4 plate-mapper3 ------ NOCVS:v5_0_16 ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 5.359 5.362 Axial B: 5.206 5.212 Axial C: 5.512 5.517 Trans D: 10.357 10.353 Lateral E: 10.6553 10.6553 Lateral F: 12.2809 12.2809 GALILS Commanded: 11300 -650 5500 21900 13500 13050 Actual: 11300 -650 5500 21900 13500 13050 SETMIR VALUES PrimDesOrient: 0.00 0.00 0.00 612.35 1500.90 PrimOrient: 0.00 0.00 0.00 612.07 1500.38 SECONDARY: ---------- Focus: 0.00 Air temp: 2.7 Altitude: 30.1 MIGS TONIGHT NOMINAL Axial A: 7.039 7.036 Axial B: 7.044 7.041 Axial C: 7.035 7.031 Trans D: 0.336 0.334 GALILS Commanded: 1576550 1594600 1580000 -3550 -6650 Actual: 1576543 1594603 1580008 -3550 -6650 SETMIR VALUES SecDesOrient: 1257.00 3.00 5.00 0.00 115.81 SecOrient: 1257.00 3.00 5.02 -0.16 116.14 04:37Z After a reboot of tcc, taking a new set of mirror stats: PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 5.359 5.362 Axial B: 5.206 5.212 Axial C: 5.513 5.517 Trans D: 10.358 10.353 Lateral E: 10.6553 10.6553 Lateral F: 12.2809 12.2809 GALILS Commanded: 11300 -650 5500 21900 13500 13050 Actual: 11300 -650 5500 21900 13500 13050 SETMIR VALUES PrimDesOrient: 0.00 0.00 0.00 612.35 1500.90 PrimOrient: 0.00 0.00 0.00 612.07 1500.38 SECONDARY: ---------- Focus: 0.00 Air temp: 2.8 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: 7.038 7.036 Axial B: 7.044 7.041 Axial C: 7.035 7.031 Trans D: 0.335 0.334 GALILS Commanded: 1576550 1594600 1580000 -3550 -6650 Actual: 1576543 1594603 1580058 -3550 -6650 SETMIR VALUES SecDesOrient: 1257.00 3.00 5.00 0.00 115.81 SecOrient: 1256.99 3.01 5.01 -0.16 116.13