Subject: 2.5m obslog 20151107 (SJD 57334) From: gfreischlad@apo.nmsu.edu Date: Sun, 8 Nov 2015 11:54:16 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Saturday November 07, 2015 (SJD 57334) ---- Observing Team ---- Kaike Pan Gordon Freischlad Fritz (afternoon/evening server restart help) ---- Observing Summary ---- Lost the entire night to high humidity. Notes: -Multiple server disk failures and reboots during afternoon checkout and into the evening. See Afternoon section of the log. ---- Observing Plan ---- Sat 7 NOV 2015 57334.0 0.14 01:18 12:18 21:22 08:24 ---- eBOSS until 07:21 ( 6.1 h), then MaNGA ( 4.9 h) ---- Observing Log ---- Afternoon: =========== Boss Collimation was checked with cart 1. No adjustment was made. 22:32:08 Hartmann collimate output on cart #1 sp1: offset: r= 0 (In Focus); b= -5.2 (Out of focus) sp2: offset: r= 2209 (Out of focus); b= -5.1 (Out of focus) sp1: pred. move: spAverageMove= 835 sp2: pred. move: spAverageMove= 1924 sp1: pred. spResiduals: r= -835, b= -2.6, txt= OK, spTemp = 6.4 sp2: pred. spResiduals: r= 285, b= 0.9, txt= OK, spTemp = 5.9 Guider cart= 1, plate= 8147, side= A (loaded); ---------- motorPosition sp1: 18153, 18489, 18171; sp2: -5747, -6402, -6240; ---------- Median spectrographs temperatures: Tsp1= 6.4, Tsp2= 6.0; Outside air temperature= 5.1; APOGEE evening calibs were taken, but were not properly archived due to the problems below. Before cart 10 could be mounted for the beginning of the night we were disconnected from STUI and all sdss machines. We called Fritz who looked into it and said there was a disk error and restarted the server for us. (Meanwhile the humidity spiked and we stopped ventilation) We restarted: dos, hartmann and platedb actors, and bossICC(complete with port logouts). And had to hub startNubs tcc, apogee, apogeecal and apogeeql. Right when we had everything working again and were taking some test biases the server failed again. Another disk failure. Fritz restarted the server and ran some disk analysis. Then we restarted hartmann, bossICC and dos again. Apogee was ok. We then noticed the Humidity, Temp, Dewpoint and Dewpoint Differential fields of the weather status page weren't available and that the TPM wasn't displaying dewar weights or mirror numbers or temps and likely more. We rebooted TPM2 and the tpm window and the weather status page are back. Fritz says that the logs show the disk storage system checking the disks and trying to "fix" them. He had to stop the check using a default configuration and then run fsck on the filesystem to fix the problem. ===== Night ===== A critical alert about the SP1 dewar pressure was triggered during the a primary fill, but the pressure returned to normal very quickly (within a couple minutes). Continuing the server issues from the afternoon: Fritz found the same problem so he rebooted the server again and fsck'd and then we resarted platedb, hartmann, bossICC, DOS and did hub startNubs for tcc, apogee, apogeecal and apogeeql. 04:33Z All seems to be working fine, but humidity is still very high. ---- APOGEE Data Summary ---- APOGEE data list, mjd=57334, all raw_data: /data/apogee/utr_cdr/57334/apRaw-*.fits archive: /data/apogee/archive/57334/apR-[a,b,c]-*.apz quickred: /data/apogee/archive/57334/apR-[a,b,c]-*.apz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 57334 1 22:25 17720001 Dark 3 A 13.0 1-8147 a-b-c - - 57334 2 22:46 17720002 Dark 60 A 13.0 ------- x-x-x - - 57334 3 22:47 17720003 QuarFlat 10 A 13.0 ------- x-x-x - 19 57334 4 01:01 17720004 Dark 60 A 13.0 1-8147 a-b-c - - 57334 5 01:12 17720005 QuarFlat 10 A 13.0 1-8147 a-b-c - 78 57334 6 01:14 17720006 Arc-Thar 12 A 13.0 1-8147 a-b-c -0.99 - 57334 7 01:16 17720007 Arc-Une 40 A 13.0 1-8147 a-b-c - - 57334 8 01:23 17720008 Arc-Thar 12 B 13.5 1-8147 a-b-c -0.48 - 57334 9 01:33 17720009 Arc---- 40 B 13.5 ------- x-x-x - - cannot read file : /data/apogee/utr_cdr/57334/apRaw-17720010.fits 57334 11 04:12 17720011 Dark 3 A 13.0 1-8147 a-b-c - - 57334 12 05:32 17720012 Dark 3 A 13.0 1-8147 a-b-c - - -------------------------------------------------------------------------------- ---- eBOSS Data Summary ---- Reading FITS from /data/spectro/57334/ 1 02:16 b1-00207747 11 8768 bia Clo 0.0 18153 18489 18171 02:16 b2-00207747 11 8768 bia Clo 0.0 -5747 -6402 -6240 02:16 r1-00207747 11 8768 bia Clo 0.0 18153 18489 18171 02:16 r2-00207747 11 8768 bia Clo 0.0 -5747 -6402 -6240 2 02:17 b1-00207748 11 8768 dar Clo 20.0 18153 18489 18171 02:17 b2-00207748 11 8768 dar Clo 20.0 -5747 -6402 -6240 02:17 r1-00207748 11 8768 dar Clo 20.0 18153 18489 18171 02:17 r2-00207748 11 8768 dar Clo 20.0 -5747 -6402 -6240 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --- OK --- ---- MaNGA Data Summary ---- Reading FITS from /data/spectro/57334/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n time filename crt plt flav Shu Dth exp colA colB colC Hrt b1 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 22:25 b1-00207736 1 8147 bia Clo C 0.0 17317 17650 17334 22:25 b2-00207736 1 8147 bia Clo C 0.0 -7671 -8326 -8164 22:25 r1-00207736 1 8147 bia Clo C 0.0 17317 17650 17334 22:25 r2-00207736 1 8147 bia Clo C 0.0 -7671 -8326 -8164 2 22:26 b1-00207737 1 8147 dar Clo C 20.0 17317 17650 17334 22:26 b2-00207737 1 8147 dar Clo C 20.0 -7671 -8326 -8164 22:26 r1-00207737 1 8147 dar Clo C 20.0 17317 17650 17334 22:26 r2-00207737 1 8147 dar Clo C 20.0 -7671 -8326 -8164 3 22:29 b1-00207738 1 8147 arc Lef C 4.1 17317 17650 17334 HART 22:29 b2-00207738 1 8147 arc Lef C 4.2 -7671 -8326 -8164 HART 22:29 r1-00207738 1 8147 arc Lef C 4.1 17317 17650 17334 HART 22:29 r2-00207738 1 8147 arc Lef C 4.2 -7671 -8326 -8164 HART 4 22:30 b1-00207739 1 8147 arc Rig C 4.1 17317 17650 17334 HART 22:30 b2-00207739 1 8147 arc Rig C 4.2 -7671 -8326 -8164 HART 22:30 r1-00207739 1 8147 arc Rig C 4.1 17317 17650 17334 HART 22:30 r2-00207739 1 8147 arc Rig C 4.2 -7671 -8326 -8164 HART 5 22:33 b1-00207740 1 8147 fla Out C 25.1 18153 18489 18171 22:33 b2-00207740 1 8147 fla Out C 25.1 -5747 -6402 -6240 22:33 r1-00207740 1 8147 fla Out C 25.1 18153 18489 18171 22:33 r2-00207740 1 8147 fla Out C 25.1 -5747 -6402 -6240 6 22:37 b1-00207741 1 8147 arc Out C 4.1 18153 18489 18171 22:37 b2-00207741 1 8147 arc Out C 4.2 -5747 -6402 -6240 22:37 r1-00207741 1 8147 arc Out C 4.1 18153 18489 18171 22:37 r2-00207741 1 8147 arc Out C 4.2 -5747 -6402 -6240 7 01:01 b1-00207743 1 8147 bia Clo 0.0 18153 18489 18171 01:01 b2-00207743 1 8147 bia Clo 0.0 -5747 -6402 -6240 01:01 r1-00207743 1 8147 bia Clo 0.0 18153 18489 18171 01:01 r2-00207743 1 8147 bia Clo 0.0 -5747 -6402 -6240 8 01:02 b1-00207744 1 8147 dar Clo 20.0 18153 18489 18171 01:02 b2-00207744 1 8147 dar Clo 20.0 -5747 -6402 -6240 01:02 r1-00207744 1 8147 dar Clo 20.0 18153 18489 18171 01:02 r2-00207744 1 8147 dar Clo 20.0 -5747 -6402 -6240 9 02:04 b1-00207745 1 8147 bia Clo 0.0 18153 18489 18171 02:04 b2-00207745 1 8147 bia Clo 0.0 -5747 -6402 -6240 02:04 r1-00207745 1 8147 bia Clo 0.0 18153 18489 18171 02:04 r2-00207745 1 8147 bia Clo 0.0 -5747 -6402 -6240 10 02:05 b1-00207746 1 8147 dar Clo 20.0 18153 18489 18171 02:05 b2-00207746 1 8147 dar Clo 20.0 -5747 -6402 -6240 02:05 r1-00207746 1 8147 dar Clo 20.0 18153 18489 18171 02:05 r2-00207746 1 8147 dar Clo 20.0 -5747 -6402 -6240 11 04:11 b1-00207749 1 8147 bia Clo 0.0 18153 18489 18171 04:11 b2-00207749 1 8147 bia Clo 0.0 -5747 -6402 -6240 04:11 r1-00207749 1 8147 bia Clo 0.0 18153 18489 18171 04:11 r2-00207749 1 8147 bia Clo 0.0 -5747 -6402 -6240 12 04:12 b1-00207750 1 8147 dar Clo 20.0 18153 18489 18171 04:12 b2-00207750 1 8147 dar Clo 20.0 -5747 -6402 -6240 04:12 r1-00207750 1 8147 dar Clo 20.0 18153 18489 18171 04:12 r2-00207750 1 8147 dar Clo 20.0 -5747 -6402 -6240 13 04:18 b1-00207751 1 8147 bia Clo 0.0 18153 18489 18171 04:18 b2-00207751 1 8147 bia Clo 0.0 -5747 -6402 -6240 04:18 r1-00207751 1 8147 bia Clo 0.0 18153 18489 18171 04:18 r2-00207751 1 8147 bia Clo 0.0 -5747 -6402 -6240 14 05:32 b1-00207752 1 8147 bia Clo 0.0 18153 18489 18171 05:32 b2-00207752 1 8147 bia Clo 0.0 -5747 -6402 -6240 05:32 r1-00207752 1 8147 bia Clo 0.0 18153 18489 18171 05:32 r2-00207752 1 8147 bia Clo 0.0 -5747 -6402 -6240 15 05:34 b1-00207753 1 8147 dar Clo 20.0 18153 18489 18171 05:34 b2-00207753 1 8147 dar Clo 20.0 -5747 -6402 -6240 05:34 r1-00207753 1 8147 dar Clo 20.0 18153 18489 18171 05:34 r2-00207753 1 8147 dar Clo 20.0 -5747 -6402 -6240 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --- OK --- ---- eBOSS Collimation Summary ---- --- Hartmann --- Time Inst r1 b1 move1 b1pred Tsp1 r2 b2 move2 b2pred Tsp2 -------------- ---------------------------- ---------------------------- 22:31 1-8147A 0 -5.2 835 -2.6 6.4 2209 -5.1 1924 0.9 5.9 ---- Telescope Status ---- No interlocks bypassed Status at: 11:53Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 1 Counterweights at: 438 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 165.1 lbs, 15.5 psi SP2 198.3 lbs, 20.9 psi ---- Software Used ---- STUI : 1.7.1b2 alerts : trunk+svn149374 apo : trunk+svn158476M boss : v3_6 -boss.specMec : 3.0.4 gcamera : v2_1 guider : v3_3 hartmann : v1_4_3 hub : Tag: sdss4_v3_0 mcp : v6_6_18 -mcp.plc : 31 -mcp.azFiduci : v1_118 -mcp.altFiduc : v1_118 -mcp.rotFiduc : v1_118 platedb : v2_3 sop : v3_4 tcc : 1.2.2 apogee : not availble apogeecal : not availble apogeeql : v1_6_1 APO-local : trunk idlspec2d : v5_7_12 plate-mapper3 : NOCVS:v6_0_6 petunia : v2_5_2 autoscheduler : v5_36 sdss_python_module: v2_2 mangadrp : trunk exported ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000; MIGS TONIGHT NOMINAL Axial A: -0.122 -0.118 Axial B: 0.121 0.118 Axial C: -0.028 -0.025 Trans D: -1.356 -1.345 Lateral E: 10.9601 10.9601 Lateral F: 12.5984 12.5857 GALILS Commanded: 11500 -1200 5850 19600 24150 23900 Actual: 11500 -1200 5850 19600 24150 23900 SETMIR VALUES PrimDesOrient: 0.0 -24.21 6.52 356.1 1323.0 PrimOrient: 0.0 -24.31 6.14 356.65 1322.6 SECONDARY: ---------- Focus: 0.0000 Air temp: 5.2 Altitude: 29.8 MIGS TONIGHT NOMINAL Axial A: 0.007 0 Axial B: -0.043 -0.04 Axial C: -0.039 -0.041 Trans D: -0.001 -0.007 GALILS Commanded: 1613250 1567850 1570050 -1700 -8100 Actual: 1613211 1567806 1570080 -1700 -8100 SETMIR VALUES SecDesOrient: 1257.0 4.35 -15.05 25.0 111.05 SecOrient: 1256.98 4.31 -15.0 24.72 110.66