Subject: 2.5m obslog 20140306 (MJD 56723) From: kinemuchi@apo.nmsu.edu Date: Fri, 7 Mar 2014 13:06:05 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday March 06, 2014 (MJD 56723) ---- Observing Team ---- Patrick Gaulme Karen Kinemuchi ---- Observing Plan ---- 6 Mar 2014 56723.0 0.42 02:12 12:16 06:08 16:14 BOSS 10.1 hours ---- Observing Summary ---- First half of the night was clouded out. Opened for the 2nd half to clear skies. BOSS Observing: Cart 16 (Plate 7123) - not done Cart 17 (Plate 6963) - DONE! We had a netburner problem during the cart change (16->17). Please see the Problems in Detail for more information. One data file for Cart 16 was not recovered. APOGEE Observing: Cart 3 (Plate 5763) - 2xABBA [no missing fibers; faint fibers #173,197,248,250] Cart 7 (Plate 6855) - 1xAB [no missing fibers; faint fibers #197,270] ---- Observing Log ---- ============================================================================ AFTERNOON CHECKOUT ============================================================================ 22:45Z Started checkout %... Disk space Filesystem Size Used Avail Use% Mounted on /dev/sda2 95G 19G 71G 22% / /dev/sda4 793G 337G 415G 45% /home /dev/sda1 99M 36M 58M 39% /boot tmpfs 5.9G 0 5.9G 0% /dev/shm %... BOSS Hartmann COLLIMATE: Predict blue camera ring movement of -5.0 degrees for spectro-1 COLLIMATE: Predict blue camera ring movement of -9.0 degrees for spectro-2 COLLIMATE: Predict (red) piston movement of 1110 steps for spectro-1 COLLIMATE: Predict (red) piston movement of 3843 steps for spectro-2 red cameras adjusted Target B1: -3 B2: +2, COLLIMATE: Predict blue camera ring movement of -1.0 degrees for spectro-1 COLLIMATE: Predict blue camera ring movement of 1.4 degrees for spectro-2 COLLIMATE: Predict (red) piston movement of -170 steps for spectro-1 COLLIMATE: Predict (red) piston movement of 85 steps for spectro-2 Collimation Finished %... Status --------- motorPosition sp1: 21997, 22135, 21885; sp2: 20054, 19355, 20544; ---------- Median spectrographs temperatures: Tsp1= 6.9, Tsp2= 6.2; Outside air temperature= 5.7; 00:53Z Whoops! Sorry - we accidentally queued up 30 BOSS flats! Please ignore image files 176368-176384 (these are all BOSS flats). The BOSS arc is file #176385. ============================================================================ NIGHT ============================================================================ 01:00Z Ventilation begins. Delaying opening due to clouds. 05:08Z Ventilation ends. Nearby radar return. 06:30Z Ventilation begins again. 08:05:41 encl25m : open; 08:17:15 loadCart: ct=16, pl=7123, sd=A; HARTMANN: 08:31:53 boss exposure 4.0, file=176386 08:32:37 boss exposure 4.0, file=176387 08:34:14 hartSp2: r=413, b=1.4, txt=OK, sp2Temp = 4.2 08:34:14 sos.sp2AverageMove = 2263 08:34:22 sp2.motor.move= 2263, 2263, 2262 08:34:22 hartSp1: r=-162, b=-0.6, txt=OK, sp1Temp = 4.8 08:34:22 sos.sp1AverageMove = 296 08:34:26 sp1.motor.move= 298, 297, 297 CALIBS: 08:34:46 boss exposure 4.0, file=176388 08:36:19 boss exposure 30.0, file=176389 GSOGTF; clear skies; seeing is 1.4" 08:40:28 boss exposure 900.0, file=176390 08:56:50 boss exposure 900.0, file=176391 09:13:12 boss exposure 900.0, file=176392 plate3 7123 (S/N)^2 accumulated for plate 7123 ---- logfile-56722.html ---- ---- logfile-56723.html ---- --------------------------------------------------------------- MJD b1 r1 b2 r2 Texp Sum Plugfile --------------------------------------------------------------- 56723 4.13 9.68 3.56 7.50 900.1 24.87 7123-56721-01 56723 3.70 8.30 2.95 6.01 900.1 20.96 7123-56721-01 --------------------------------------------------------------- s/n2= 7.83 17.98 6.51 13.51 1800.2 --> 45.83 [above 0.50] --------------------------------------------------------------- Plate 7123 is not complete. ================================================================================ 09:40Z We got a netburner problem during the goToField command and the Hartmann exposures failed. We followed the procedures to recover from the netburner problem. However, upon investigation of the log, it looks like the netburner problem happened during the readout of the last image for Cart 16 (Plate 7123). We were unable to recover image 176392 (only data from sp2 are on /data/spectro/56723). See Problems in Detail for more information. 09:32:50 loadCart: ct=17, pl=6963, sd=A; 09:33:01 sop.gotoField = running; 09:34:04 sop.gotoField = failed; 09:34:39 sop.gotoField = running; 09:34:49 sop.gotoField = failed; 09:35:55 sop.gotoField = running; HARTMANN: 09:36:24 boss exposure 4.0, file=176392 09:37:35 sop.gotoField = failed; 09:38:03 sop.gotoField = running; 09:38:05 sop.gotoField = failed; 09:40:38 boss exposure 0.0, file=176392 09:42:42 sop.gotoField = running; 09:43:11 boss exposure 4.0, file=176395 09:43:55 boss exposure 4.0, file=176396 09:44:58 hartSp1: r=-497, b=-1.7, txt=OK, sp1Temp = 4.6 09:44:58 sos.sp1AverageMove = -328 09:45:02 sp1.motor.move= -329, -328, -331 09:45:02 hartSp2: r=-438, b=-1.5, txt=OK, sp2Temp = 3.9 09:45:02 sos.sp2AverageMove = -3608 09:45:11 sp2.motor.move= -3609, -3607, -3608 CALIBS: 09:45:32 boss exposure 4.0, file=176397 09:47:06 boss exposure 30.0, file=176398 GSOGTF; clear skies; seeing is 1.3" 09:52:55 boss exposure 900.0, file=176399 10:09:17 boss exposure 900.0, file=176400 10:25:39 boss exposure 900.0, file=176401 plate3 6963 (S/N)^2 accumulated for plate 6963 ---- logfile-56723.html ---- --------------------------------------------------------------- MJD b1 r1 b2 r2 Texp Sum Plugfile --------------------------------------------------------------- 56723 3.97 7.64 3.26 6.59 900.1 21.46 6963-56723-01 56723 4.38 8.45 3.52 7.51 900.1 23.85 6963-56723-01 56723 4.45 8.65 3.46 7.64 900.1 24.19 6963-56723-01 --------------------------------------------------------------- s/n2= 12.79 24.74 10.24 21.74 2700.3 --> 69.51 [above 0.50] --------------------------------------------------------------- Plate 6963 is COMPLETE! ================================================================================ 10:44:32 loadCart: ct=3, pl=5763, sd=A; 10:46:39 sop.gotoField = running; 10:49:52 sop.gotoField = done; GSOGTF; clear skies; seeing is 1.4" 11 10:54 11610011 Object 47 A-13.0 3-5763 a-b-c - - 12 11:03 11610012 Object 47 B-13.5 3-5763 a-b-c - - 13 11:11 11610013 Object 47 B-13.5 3-5763 a-b-c - - 14 11:20 11610014 Object 47 A-13.0 3-5763 a-b-c - - 15 11:28 11610015 Object 47 A-13.0 3-5763 a-b-c - - 16 11:36 11610016 Object 47 B-13.5 3-5763 a-b-c - - 17 11:45 11610017 Object 47 B-13.5 3-5763 a-b-c - - 18 11:53 11610018 Object 47 A-13.0 3-5763 a-b-c - - 19 12:02 11610019 DomeFlat 5 A-13.0 3-5763 a-b-c - 95 20 12:04 11610020 Dark 10 A-13.0 3-5763 a-b-c - - 21 12:06 11610021 Dark 10 A-13.0 3-5763 a-b-c - - aptest 56723 11610019 flat= /data/apogee/utr_cdr/56723/apRaw-11610019.fits master= /home/observer/bin/apRaw-03720068.fits nfibers= 300 average ratio= 0.781201582137 missing fibers (flat/master -------------------------------------------------------------------------------- 08:39 16-7123A 1.000180 -1777 137 445 161.7 55.6 4.2 13 266 1.5 boss 08:56 16-7123A 1.000185 -1815 105 456 164.4 56.6 3.9 11 263 1.4 boss 09:12 16-7123A 1.000174 -1720 189 454 167.3 57.4 3.9 12 266 1.5 boss 09:52 17-6963A 1.000129 -1271 602 494 190.0 56.7 3.6 15 279 1.3 boss 10:08 17-6963A 1.000127 -1248 621 494 192.8 56.0 3.6 14 294 1.4 boss 10:25 17-6963A 1.000123 -1200 646 486 195.5 55.1 3.7 21 288 1.4 boss 10:54 3-5763A 1.000324 -3198 -1120 396 67.5 70.1 4.0 16 298 1.6 apog 11:02 3-5763A 1.000290 -2851 -890 354 64.2 71.7 4.3 20 294 1.4 apog 11:11 3-5763A 1.000274 -2707 -783 330 60.6 73.2 4.3 23 300 1.4 apog 11:19 3-5763A 1.000263 -2593 -722 300 56.2 74.8 4.6 21 296 1.3 apog 11:28 3-5763A 1.000258 -2545 -690 294 51.0 76.2 4.4 19 293 1.3 apog 11:36 3-5763A 1.000262 -2583 -709 309 44.9 77.5 4.3 20 295 1.3 apog 11:44 3-5763A 1.000261 -2573 -693 317 37.4 78.6 4.3 13 274 1.3 apog 11:53 3-5763A 1.000268 -2643 -748 322 28.5 79.6 4.2 20 294 1.4 apog 12:13 7-6855A 1.000215 -2121 -225 371 156.5 56.9 4.0 15 276 1.4 apog 12:21 7-6855A 1.000222 -2188 -269 384 157.9 57.6 4.3 16 278 1.6 apog ---- Weather Log ---- --- Weather --- Time Inst Temp DP Dif Humid Wind Dir Dust,1um IRSC IRSCm -------------------------------------------------------------------------------- 08:39 16-7123A 4.2 -9.3 13.5 36 13 266 524 92.7 819 1.5 boss 08:56 16-7123A 3.9 -9.1 13.0 38 11 263 524 138.7 808 1.4 boss 09:12 16-7123A 3.9 -8.7 12.6 39 12 266 524 87.6 813 1.5 boss 09:52 17-6963A 3.6 -8.9 12.5 39 15 279 524 35.2 802 1.3 boss 10:08 17-6963A 3.6 -8.7 12.3 40 14 294 524 79.8 803 1.4 boss 10:25 17-6963A 3.7 -8.7 12.4 40 21 288 524 41.5 802 1.4 boss 10:54 3-5763A 4.0 -8.9 12.9 38 16 298 524 45.1 804 1.6 apog 11:02 3-5763A 4.3 -8.4 12.7 39 20 294 524 86.2 804 1.4 apog 11:11 3-5763A 4.3 -8.4 12.7 38 23 300 524 37.9 807 1.4 apog 11:19 3-5763A 4.6 -8.4 13.0 38 21 296 524 78.0 806 1.3 apog 11:28 3-5763A 4.4 -8.4 12.8 38 19 293 524 53.5 801 1.3 apog 11:36 3-5763A 4.3 -9.0 13.3 37 20 295 524 27.8 799 1.3 apog 11:44 3-5763A 4.3 -9.6 13.9 35 13 274 524 22.2 802 1.3 apog 11:53 3-5763A 4.2 -9.7 13.9 35 20 294 524 11.4 801 1.4 apog 12:13 7-6855A 4.0 -9.8 13.8 35 15 276 524 8.4 801 1.4 apog 12:21 7-6855A 4.3 -10.3 14.6 33 16 278 524 9.5 800 1.6 apog ---- Telescope Status ---- Started ventilation 08:05Z encl25m : open 12:45Z encl25m : closed No interlocks bypassed Status at: 12:54:07 Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 11 Counterweights at: 445, 445, 446, 446 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 220.5 lbs, 20.5 psi SP2 253.5 lbs, 22.9 psi ---- Software Used ---- ----STUI software: STUI: 1.3 2013-09-05 alerts: v1_3_0 apo: FAILED boss: v2_5_15 boss.daq: 1.3.2 boss.specMech: 3.0.4 gcamera: v1_3_7 guider: v1_6_3 hub: Tag: sdss3_v1_2_0 mcp: v6_6_18 mcp.plc: 29 mcp.azFiducialVersion: v1_111 mcp.altFiducialVersion: v1_111 mcp.rotFiducialVersion: v1_111 platedb: v1_1_0 sop: v1_3_2 sos: v1_2_0 sos.idlspec2d: v5_6_9 tcc: 2.14.0 apogee: -not availble- apogeecal: -not availble- apogeeql: v1_5_3 -----Other software sdssProcedure(sdss3): v3_06 APO-local scripts : trunk -- Other versions ---- idlspec2d (sos3) --- v5_6_9 plate-mapper3 ------ NOCVS:v5_0_14a ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 15.01 15.011 Axial B: 16.004 16.003 Axial C: 15.529 15.529 Trans D: -0.007 -0.003 Lateral E: 10.6299 10.6299 Lateral F: 12.2809 12.2809 GALILS Commanded: 11300 -650 5500 21900 13500 13050 Actual: 11300 -650 5500 21900 13500 13050 SETMIR VALUES PrimDesOrient: 0.00 0.00 0.00 612.35 1500.90 PrimOrient: 0.00 0.00 0.00 612.07 1500.38 SECONDARY: ---------- Focus: 0.00 Air temp: 6.4 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -7.062 -7.063 Axial B: -7.036 -7.036 Axial C: -7.078 -7.078 Trans D: 0.029 0.024 GALILS Commanded: 1575950 1583000 1592550 -8800 -3350 Actual: 1575951 1583019 1592537 -8800 -3350 SETMIR VALUES SecDesOrient: 1257.00 -4.10 3.00 -100.00 143.14 SecOrient: 1256.98 -4.10 2.99 -100.04 143.62 ---- Problems in Detail ---- A couple of strange occurrences happened in between the cart change from Cart 16 to Cart 17. The goToInstrumentChange command failed. It was issued while the last image (file #176392) was reading out. Error from the log (in red): 09:28:12 APO.Patrick 125 sop f text="will not go to instrument change: slewing disallowed for BOSS, with 1 science exposures left; exposureState=INTEGRATING" 09:28:12 CmdDone: APO.Patrick 125 sop gotoInstrumentChange 09:28:12 client.v689.guider 639 gcamera i stack=1 We reissued the goToInstrumentChange when there was a few seconds left in the readout, and the slew executed fine. 09:29:22 .hub 0 cmds d CmdQueued=1441390,1394184562.64,"APO.Patrick",126,"sop",710,"gotoInstrumentChange " However: 09:29:37 APO.Patrick.sop 479 guider w text="TCC motion aborted guiding" 09:30:10 APO.Patrick 126 sop w text="in slew with halted=False slewing=False" 09:30:14 APO.Patrick.sop 482 boss w text="Readout timeout expired, aborting exposure." Another warning, but this one is perplexing as we were finishing our sequence of 3 BOSS exposures: 09:30:47 APO.Patrick 123 sop w text="1 tasks failed to respond: boss" 09:30:47 APO.Patrick 123 sop f text="Failed to take science exposure" 09:30:47 CmdDone: APO.Patrick 123 sop doScience nExp=3 expTime=900.0 We got a netburner problem in the meantime: 09:28:44 APO.Patrick.sop 482 boss w text="sp1daq readout failed: Invalid response from Netburner. (>)." This apparently caused file #176392 to not be completely written to disk. We didn't notice anything out of the ordinary with the readout in the BOSS instrument window at that time. We noticed that something was wrong when we issued a goToField command. The goToField command failed at the Hartmann step. 09:34:02 client.v689.sos 226 boss f text="exposure already in progress! Do a 'boss clearExposure' if you want to kill and or clear it." 09:34:02 CmdDone: client.v689.sos 226 boss exposure arc hartmann=left itime=4 window=850,1400 09:34:02 APO.Patrick.sop 492 sos f text="failed to take left hartmann exposure" 09:34:02 CmdDone: APO.Patrick.sop 492 sos doHartmann 09:34:04 APO.Patrick 128 sop f text="Failed to do Hartmann sequence" 09:34:04 CmdDone: APO.Patrick 128 sop gotoField flatTime=30.0 guiderFlatTime=0.5 guiderExpTime=5.0 arcTime=4.0 At this point, we did not realize that the netburner problem had occurred, and we tried to figure out why the Hartmanns failed. Upon investigation of the log, we discovered there was a netburner problem. Unfortunately, we could not recover the lost data file of Cart 16 (Plate 7123) because the first Hartmann was successful when we issued a goToField command -- the Hartmann file overwrote the original file 176392. Here is the output from LogFun: First try: 09:33:01 sop.gotoField = running; 09:34:04 sop.gotoField = failed; Second try: 09:34:39 sop.gotoField = running; 09:34:49 sop.gotoField = failed; Third try: 09:35:55 sop.gotoField = running; 09:36:24 boss exposure 4.0, file=176392 09:37:35 sop.gotoField = failed; 09:38:03 sop.gotoField = running; 09:38:05 sop.gotoField = failed; Here is where we realized a netburner problem happened. We followed the procedure to recover from this. We thought we just lost one of the Hartmann images, so we did not try to recover it. Note the file number is the same as the last data file for Cart 16. We realized that the data file 176392 had not completely finished writing to disk, and at this point, the data had been overwritten(maybe?) by the Hartmann and test bias. We could not find the original Cart 16 data file #176392 on /data/spectro/56723 or on hub25m. 09:40:38 boss exposure 0.0, file=176392 09:42:42 sop.gotoField = running; 09:43:11 boss exposure 4.0, file=176395 09:43:55 boss exposure 4.0, file=176396 09:44:58 hartSp1: r=-497, b=-1.7, txt=OK, sp1Temp = 4.6 09:44:58 sos.sp1AverageMove = -328 09:45:02 sp1.motor.move= -329, -328, -331 09:45:02 hartSp2: r=-438, b=-1.5, txt=OK, sp2Temp = 3.9 09:45:02 sos.sp2AverageMove = -3608 09:45:11 sp2.motor.move= -3609, -3607, -3608 09:45:32 boss exposure 4.0, file=176397 09:47:06 boss exposure 30.0, file=176398 We had missed the netburner warning in the log window (with only the warnings and alerts) because it was imbedded with other blue warnings that were the standard warnings we see related to BOSS. Perhaps it would be good to have the netburner warning come up in red so we can see it more readily? In our warnings/alert window, all of the following messages were in blue, and when clustered together, the netburner warning can be hard to parse out: 09:28:44 APO.Patrick.sop 482 boss w text="sp1daq readout failed: Invalid response from Netburner. (>)." 09:28:53 .boss 91231 boss w SP2R0VSc1Read=-4.023; SP2R1VSc2Read=-3.945 09:28:53 .boss 91231 boss w camCheck=SP2R0VSc1Read,SP2R1VSc2Read 09:28:53 .alerts 0 alerts w text="sent MAIL WARN boss.SP2R1VSc2Read == Reported by camCheck"