Subject: 2.5m obslog 20220218 (SJD 59629) From: dmbiz@apo.nmsu.edu Date: Sat, 19 Feb 2022 12:35:08 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday February 18, 2022 (SJD 59629) ---- Observing Team ---- Viktor Malanushenko Dmitry Bizyaev Jose Sanchez-Gallego John Donor Joe Burchett David Nidever (evening, in STUI) ---- Observing Plan ---- https://wiki.sdss.org/pages/viewpage.action?spaceKey=COM&title=Commissioning+Nightly+Plan Friday February 18 1. Observe BOSS science commissioning design 4823 from the queue. 2 Continue observing from the queue simulating normal operations. Use Boson alpha.36. At the beginning of the night do a focus sweep but after that keep the auto-focus feature enable to test if it keeps focus as it drifts. At the end of each exposure preload the next design, then as soon as the BOSS readout begins load the preloaded design (from the blue banner on top of the HAL window), then immediately start the new goto sequence. 3. For an hour at some point in the night, observe as in 2. but without taking any calibrations. 4. Throughout the night, keep track of the time spent on the different goto steps and lost time due to hardware/software issues. ---- Observing Summary ---- Clear, productive night The whole night we observed science field via pre-loading designs in Boson, and then loading them the same way. ** Science designs completed: Desgn Field Cfg Exposures Overhead(*) --------------------------------------------- 4823-17057 3583 B: 900s A: AB(43) start of the night 4823-17057 3584 B: 900s A: AB(43) 4823-17057 3585 B: 900s A: AB(43) 4823-17057 3586 B: 900s A: AB(43) 12952-19154 3588 B: 900s A: AB(43) 13 min [includes BOSS calibs] 15599-19642 3590 B: 900s A: AB(43) 20 min [includes BOSS calibs] Lower than expected S/N on APOGEE, but adding one more failed because of APOGEE-BOSS exposures, see log. 15680-19655 3591 B:2x900s A:2xAB(43) 52 min [calibs + problems] 15268-19583 3592 B: 900s A: AB(43) 19 min [includes BOSS calibs] 721-16340 3593 B: 900s A: AB(43) 11 min [includes BOSS calibs] 10510-18365 3594 B: 900s A: AB(43) 12 min 1218-16577 3595 B: 900s A: AB(43) 12 min 1219-16577 3600 B: 900s A: AB(43) 11 min [includes BOSS calibs] 13696-19348 3601 B: 900s 12 min 13695-19348 3604 B: 900s 9 min 8110-17818 3605 B: 900s 17 min [fvc loop problems] 4583-16963 3608 B: 900s 10 min 4582-16963 3611 B: 900s 10 min 5684-17315 3614 B: 900s A: AB(43) 11 min (*) Overhead is time from loading design to starting first science exposure. ** Notes - what we learned is that most of bright time fields have S/N**2 > 2025 completion criterion. Kronos marks done fields that meet the criterion with green. - APOGEE instrument was acting up. It stopped working in the middle of science exposures 4 times: at 4:47, 6:43, 08:22, 08:31. Second time APOGEE was initialized. Jose tried to restart APOGEE actor, but the problem returned back. We sent email to Matt Nelson. - 08:07Z, 09:27Z, and 3 more out of closed loop AZ and ROT during a field acquisition. - dust accumulated: 1.4K ---- Observing Log ---- Afternoon: ========== --Test Actors, 23:28:06 -- APOGEE Short Dark - 23:28:57 -- troubles with apogeeql, Jose and David fixed it -- boss bias, ff - ok 3:56:39 Hartmann collimate output sp1: offset: r= -329 (In Focus); b= -1.2 (In Focus) sp1: pred. move: spAverageMove= 28 sp1: pred. resid: r= -357, b= -1.1, txt= OK, spTemp = 4.2 hartmann ok -- 23:58:19 -- APOGEE: Evening Cals started -- boss arc -ok 00:10Z - opened doors, started ventilation Night ===== 01:05Z enclosure open =========================================== Science Design 4823, loading from the queue =========================================== 01:27:21 Config Loaded: 4823-3583 The field was loaded via the button in Boson/Design widget 01:31:55 Hartmann collimate output sp1: offset: r= 3628 (Out of focus); b= -10.3 (Out of focus) sp1: pred. move: spAverageMove= 3453 sp1: pred. resid: r= 175, b= 0.5, txt= OK, spTemp = 3.3 01:32:26 APO.viktorm.jaeger 124 jaeger i text="[FVC]: RMS full fit 47.040 um." 01:33:10 APO.viktorm.jaeger 124 jaeger i text="[FVC]: RMS full fit 19.400 um." 01:33:51 APO.viktorm.jaeger 124 jaeger i text="[FVC]: RMS full fit 16.390 um." 01:34:32 APO.viktorm.jaeger 124 jaeger i text="[FVC]: RMS full fit 16.460 um." 01:34:33 APO.viktorm.jaeger 124 jaeger d text="Saving processed image /data/fcam/59629/proc-fimg-fvc1n-0004.fits" We set initial focus (450), then Jose enabled the auto-focus loop. Adjusted its PID's. It started converging. 2:05:05 CmdDone: APO.viktorm.apogee 154 apogee expose time=459.0 object=object 02:05:05 APO.viktorm 51 hal e error="Command 'apogee expose time=459.0 object=object' failed." At the first attempt, it exposed APOGEE for 6 reads and then failed. We took test APOGEE exposure, then BOSS bias, and next attempt to start was OK. 02:00:52 boss exposure 900.0, file=337679 - not written 02:04:35 boss exposure 900.0, file=337679 - attempted again - not written 02:13:40 boss exposure 0.0, file=337679 - test bias Science exposures: 02:15:51 boss exposure 900.0, file=337682 02:15:52 apogee.expose=apRaw-40670020, 43, A, Object 02:23:33 apogee.expose=apRaw-40670021, 43, B, Object 02:33:10 Config Loaded: 4823-3584 02:34:25 boss exposure 900.0, file=337683 02:34:27 apogee.expose=apRaw-40670022, 43, A, Object 02:42:08 apogee.expose=apRaw-40670023, 43, B, Object 02:49:26 Config Loaded: 4823-3585 03:06:18 Config Loaded: 4823-3586 03:08:02 boss exposure 900.0, file=337685 03:08:04 apogee.expose=apRaw-40670026, 43, A, Object 03:15:44 apogee.expose=apRaw-40670027, 43, B, Object 03:24:42 apogee.expose=apRaw-40670028, 5, B, DomeFlat 03:37:57 APO.viktorm.jaeger 203 jaeger i text="[FVC]: RMS full fit 50.500 um." 03:40:01 APO.viktorm.jaeger 214 jaeger i text="[FVC]: RMS full fit 22.240 um." 03:40:42 APO.viktorm.jaeger 214 jaeger i text="[FVC]: RMS full fit 17.980 um." 03:41:22 APO.viktorm.jaeger 214 jaeger i text="[FVC]: RMS full fit 17.640 um." =================================== Second science field, 1 exposure 03:33:57 Config Loaded: 12952-3588 =================================== 03:37:26 Hartmann collimate output sp1: offset: r= -1649 (Out of focus); b= 2.8 (In Focus) sp1: pred. move: spAverageMove= -1274 sp1: pred. resid: r= -375, b= -1.2, txt= OK, spTemp = 1.7 03:37:57 APO.viktorm.jaeger 203 jaeger i text="[FVC]: RMS full fit 50.500 um." FVC loop failed because of missing exposure (Jose was here, he knows about the problem) We manually started FVC loop and further steps from the Boson 03:40:01 APO.viktorm.jaeger 214 jaeger i text="[FVC]: RMS full fit 22.240 um." 03:40:42 APO.viktorm.jaeger 214 jaeger i text="[FVC]: RMS full fit 17.980 um." 03:41:22 APO.viktorm.jaeger 214 jaeger i text="[FVC]: RMS full fit 17.640 um." 03:41:52 boss exposure 4.0, file=337689 03:43:26 boss exposure 25.0, file=337690 03:47:38 boss exposure 900.0, file=337691 03:47:40 apogee.expose=apRaw-40670029, 43, A, Object 03:55:21 apogee.expose=apRaw-40670030, 43, B, Object 04:04:38 apogee.expose=apRaw-40670031, 5, B, DomeFlat =================================== Third science field, 1 exposure 04:05:48 Config Loaded: 15599-3590 =================================== 4:09:29 Hartmann collimate output sp1: offset: r= 0 (In Focus); b= 1.8 (In Focus) sp1: pred. move: spAverageMove= -289 sp1: pred. resid: r= 289, b= 0.9, txt= OK, spTemp = 1.4 We lost ~7 min on waiting fr the guider rms to converge, but it stays at 0.27, and not going down to under 0.07. Decided to start science exposure any way. 04:26:03 boss exposure 900.0, file=337696 04:26:04 apogee.expose=apRaw-40670032, 43, A, Object 04:33:45 apogee.expose=apRaw-40670033, 43, B, Object 04:44:47 boss exposure 900.0, file=337697 04:44:48 apogee.expose=apRaw-40670034, 43, A, Object 04:50:28 apogee.expose=apRaw-40670035, 5, B, DomeFlat Comment: we added one more exposure because S/N**2 for A+B was less than expected (4000, according to JohnD) 04:56:56 client.v1451.hartmann 531 boss e text="Readout timeout expired, aborting exposure." ==================================== Fourth science exposure 04:52:50 Config Loaded: 15680-3591 ==================================== 04:54:47 boss exposure 4.0, file=337697 04:56:26 Hartmann collimate output Hartmann failed sp1: offset: r= ? (?); b= ? (?) sp1: pred. move: spAverageMove= ? sp1: pred. resid: r= ?, b= ?, txt= ?, spTemp = 1.2 Problem with APOGEE/BOSS exposures. See Problem in detail section. We showed it to Jose, and finally restarted BOSS actor. 05:31:42 Hartmann collimate output sp1: offset: r= 0 (In Focus); b= 5.8 (Out of focus) sp1: pred. move: spAverageMove= -931 sp1: pred. resid: r= 931, b= 2.9, txt= OK, spTemp = 1.1 05:32:14 APO.viktorm.jaeger 310 jaeger i text="[FVC]: RMS full fit 52.140 um." 05:32:57 APO.viktorm.jaeger 310 jaeger i text="[FVC]: RMS full fit 18.780 um." 05:33:38 APO.viktorm.jaeger 310 jaeger i text="[FVC]: RMS full fit 11.750 um." 05:34:21 APO.viktorm.jaeger 310 jaeger i text="[FVC]: RMS full fit 11.060 um." 05:34:22 APO.viktorm.jaeger 310 jaeger d text="Saving processed image /data/fcam/59629/proc-fimg-fvc1n-0017.fits" 05:43:29 boss exposure 900.0, file=337706 05:43:31 apogee.expose=apRaw-40670037, 43, A, Object 05:51:11 apogee.expose=apRaw-40670038, 43, B, Object Added one more exposure. 06:00:02 boss exposure 900.0, file=337707 06:00:04 apogee.expose=apRaw-40670039, 43, A, Object 06:07:44 apogee.expose=apRaw-40670040, 43, B, Object =================================== Fifth science field 06:18:48 Config Loaded: 15268-3592 =================================== 06:20:16 Jaeger collision: 184 6:25:10 APO.viktorm.jaeger 362 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" Unwound and reconfigured again. It worked from the second attempt. 6:29:59 Hartmann collimate output sp1: offset: r= 329 (In Focus); b= 5.8 (Out of focus) sp1: pred. move: spAverageMove= -767 sp1: pred. resid: r= 1096, b= 3.4, txt= OK, spTemp = 1.0 APOGEE again stopped at 33rd read out of 43: 06:43:43 APO.viktorm.apogee 397 apogee f errorMessage="cannot execute command: Controller is busy..." 06:43:43 CmdDone: APO.viktorm.apogee 397 apogee expose time=459.0 object=object 06:43:43 APO.viktorm 58 hal e error="Command 'apogee expose time=459.0 object=object' failed." 6:43:48 APO.viktorm.boss 393 boss f text="exposure aborted" 59629 42 06:44 40670042 Object 33 A 13.0 15268-3592 a-b-c -0.05 - We issued "apogee init" , took short 3-read exposure (looks ok), and started science again. 06:53:51 boss exposure 900.0, file=337712 06:53:53 apogee.expose=apRaw-40670044, 43, A, Object 07:01:34 apogee.expose=apRaw-40670045, 43, B, Object =================================== Sixth science field 07:11:31 Config Loaded: 721-3593 =================================== 07:14:55 Hartmann collimate output sp1: offset: r= -329 (In Focus); b= 5.8 (Out of focus) sp1: pred. move: spAverageMove= -1096 sp1: pred. resid: r= 767, b= 2.4, txt= OK, spTemp = 0.9 07:15:25 APO.viktorm.jaeger 425 jaeger i text="[FVC]: RMS full fit 46.480 um." 07:16:10 APO.viktorm.jaeger 425 jaeger i text="[FVC]: RMS full fit 15.270 um." 07:16:51 APO.viktorm.jaeger 425 jaeger i text="[FVC]: RMS full fit 12.300 um." 07:17:32 APO.viktorm.jaeger 425 jaeger i text="[FVC]: RMS full fit 10.440 um." 07:18:13 APO.viktorm.jaeger 425 jaeger i text="[FVC]: RMS full fit 11.830 um." 07:18:14 APO.viktorm.jaeger 425 jaeger d text="Saving processed image /data/fcam/59629/proc-fimg-fvc1n-0026.fits" 7:22:30 boss exposure 900.0, file=337718 07:22:32 apogee.expose=apRaw-40670047, 43, A, Object 07:30:12 apogee.expose=apRaw-40670048, 43, B, Object 07:39:01 apogee.expose=apRaw-40670049, 5, B, DomeFlat =================================== Seventh science field 07:40:07 Config Loaded: 10510-3594 =================================== 07:43:05 APO.viktorm.jaeger 467 jaeger i text="[FVC]: RMS full fit 47.730 um." 07:43:49 APO.viktorm.jaeger 467 jaeger i text="[FVC]: RMS full fit 15.500 um." 07:44:32 APO.viktorm.jaeger 467 jaeger i text="[FVC]: RMS full fit 12.410 um." 07:45:13 APO.viktorm.jaeger 467 jaeger i text="[FVC]: RMS full fit 10.800 um." 07:45:54 APO.viktorm.jaeger 467 jaeger i text="[FVC]: RMS full fit 12.570 um." 07:45:55 APO.viktorm.jaeger 467 jaeger d text="Saving processed image /data/fcam/59629/proc-fimg-fvc1n-0031.fits" Second attempt: 08:08:57 Jaeger collision: 184 08:09:02 APO.viktorm.jaeger 492 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 37.24 seconds after start" Again, APOGEE exposure stopped at 22nd read. 8:22:18 APO.viktorm.apogee 522 apogee f errorMessage="cannot execute command: Controller is busy..." 08:22:18 CmdDone: APO.viktorm.apogee 522 apogee expose time=459.0 object=object 08:22:18 APO.viktorm 39 hal e error="Command 'apogee expose time=459.0 object=object' failed." 08:31:24 boss exposure 900.0, file=337722 but APOGEE failed at 23rd read, again Jose was unable to restart the APOGEE actor, but then we've got next APOGEE exposure stuck in the middle , again. Jose asked to expose BOSS only till the rest of the night. 08:50:29 boss exposure 900.0, file=337723 =================================== Eighth science field 09:01:31 Config Loaded: 1219-3600 =================================== 09:04:48 Hartmann collimate output sp1: offset: r= 0 (In Focus); b= -0.2 (In Focus) sp1: pred. move: spAverageMove= 32 sp1: pred. resid: r= -32, b= -0.1, txt= OK, spTemp = 0.8 9:05:18 APO.viktorm.jaeger 551 jaeger i text="[FVC]: RMS full fit 49.720 um." 09:06:02 APO.viktorm.jaeger 551 jaeger i text="[FVC]: RMS full fit 17.230 um." 09:06:44 APO.viktorm.jaeger 551 jaeger i text="[FVC]: RMS full fit 11.100 um." 09:07:25 APO.viktorm.jaeger 551 jaeger i text="[FVC]: RMS full fit 11.940 um." 09:07:26 APO.viktorm.jaeger 551 jaeger d text="Saving processed image /data/fcam/59629/proc-fimg-fvc1n-0039.fits" 09:11:56 apogee.expose=apRaw-40670057, 43, A, Object 09:12:02 boss exposure 900.0, file=337728 09:19:37 apogee.expose=apRaw-40670058, 43, B, Object =================================== Ninth science field 09:27:13 Config Loaded: 13696-3601 =================================== 09:29:41 APO.viktorm.jaeger 585 jaeger i text="[FVC]: RMS full fit 48.670 um." 09:30:27 APO.viktorm.jaeger 585 jaeger i text="[FVC]: RMS full fit 17.390 um." 09:31:08 APO.viktorm.jaeger 585 jaeger i text="[FVC]: RMS full fit 16.170 um." 09:32:38 APO.viktorm.jaeger 591 jaeger i text="[FVC]: RMS full fit 16.330 um." 09:33:20 APO.viktorm.jaeger 591 jaeger i text="[FVC]: RMS full fit 12.380 um." 09:34:01 APO.viktorm.jaeger 591 jaeger i text="[FVC]: RMS full fit 11.740 um." 09:34:02 APO.viktorm.jaeger 591 jaeger d text="Saving processed image /data/fcam/59629/proc-fimg-fvc1n-0046.fits" 09:39:05 boss exposure 900.0, file=337729 =================================== Tenth science field 09:55:44 Config Loaded: 13695-3604 =================================== 9:58:00 APO.viktorm.jaeger 607 jaeger i text="[FVC]: RMS full fit 47.760 um." 09:58:45 APO.viktorm.jaeger 607 jaeger i text="[FVC]: RMS full fit 16.410 um." 09:59:26 APO.viktorm.jaeger 607 jaeger i text="[FVC]: RMS full fit 14.800 um." 10:00:08 APO.viktorm.jaeger 607 jaeger i text="[FVC]: RMS full fit 13.500 um." 10:00:49 APO.viktorm.jaeger 607 jaeger i text="[FVC]: RMS full fit 12.750 um." 10:00:50 APO.viktorm.jaeger 607 jaeger d text="Saving processed image /data/fcam/59629/proc-fimg-fvc1n-0051.fits" 10:04:30 boss exposure 900.0, file=337730 ================================== 10:20:03 Config Loaded: 8110-3605 ================================== 0:24:46 APO.viktorm.jaeger 630 jaeger i text="[FVC]: RMS full fit 43.840 um." 10:25:30 APO.viktorm.jaeger 630 jaeger i text="[FVC]: RMS full fit 24.370 um." 10:26:15 APO.viktorm.jaeger 630 jaeger i text="[FVC]: RMS full fit 27.150 um." 10:28:26 APO.viktorm.jaeger 637 jaeger i text="[FVC]: RMS full fit 26.860 um." 10:29:09 APO.viktorm.jaeger 637 jaeger i text="[FVC]: RMS full fit 708.980 um." - accidentally ROT axes left alive. Re-done below. 10:32:09 APO.viktorm.jaeger 649 jaeger i text="[FVC]: RMS full fit 27.500 um." 10:32:52 APO.viktorm.jaeger 649 jaeger i text="[FVC]: RMS full fit 20.390 um." 10:33:35 APO.viktorm.jaeger 649 jaeger i text="[FVC]: RMS full fit 17.040 um." 10:34:18 APO.viktorm.jaeger 649 jaeger i text="[FVC]: RMS full fit 16.090 um." 10:37:24 boss exposure 900.0, file=337731 0:53:23 .jaeger 0 jaeger w text="Positioner 985 replied to SEND_TRAJECTORY_DATA UID=5712 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 10:53:23 APO.viktorm.jaeger 662 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59629/trajectory-59629-0076.json 10:53:23 APO.viktorm.jaeger 662 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" 10:53Z ALT axis got out of closed loop ================================== 10:52:32 Config Loaded: 4583-3608 ================================== 11:00:05 APO.viktorm.jaeger 671 jaeger i text="[FVC]: RMS full fit 45.230 um." 11:00:50 APO.viktorm.jaeger 671 jaeger i text="[FVC]: RMS full fit 17.940 um." 11:01:33 APO.viktorm.jaeger 671 jaeger i text="[FVC]: RMS full fit 14.090 um." 11:02:17 APO.viktorm.jaeger 671 jaeger i text="[FVC]: RMS full fit 16.590 um." 11:02:18 APO.viktorm.jaeger 671 jaeger d text="Saving processed image /data/fcam/59629/proc-fimg-fvc1n-0064.fits" 11:05:30 boss exposure 900.0, file=337732 ================================== 11:20:51 Config Loaded: 4582-3611 ================================== 11:23:09 APO.viktorm.jaeger 687 jaeger i text="[FVC]: RMS full fit 43.070 um." 11:23:54 APO.viktorm.jaeger 687 jaeger i text="[FVC]: RMS full fit 14.290 um." 11:24:35 APO.viktorm.jaeger 687 jaeger i text="[FVC]: RMS full fit 12.700 um." 11:25:17 APO.viktorm.jaeger 687 jaeger i text="[FVC]: RMS full fit 10.570 um." 11:25:58 APO.viktorm.jaeger 687 jaeger i text="[FVC]: RMS full fit 10.240 um." 11:25:59 APO.viktorm.jaeger 687 jaeger d text="Saving processed image /data/fcam/59629/proc-fimg-fvc1n-0069.fits" 11:30:57 boss exposure 900.0, file=337733 ================================== 11:46:11 Config Loaded: 5684-3614 ================================== 11:46Z all axes went out of closed loop 11:51:17 APO.viktorm.jaeger 728 jaeger i text="[FVC]: RMS full fit 45.900 um." 11:52:02 APO.viktorm.jaeger 728 jaeger i text="[FVC]: RMS full fit 16.770 um." 11:52:46 APO.viktorm.jaeger 728 jaeger i text="[FVC]: RMS full fit 13.690 um." 11:53:29 APO.viktorm.jaeger 728 jaeger i text="[FVC]: RMS full fit 16.250 um." 11:53:30 APO.viktorm.jaeger 728 jaeger d text="Saving processed image /data/fcam/59629/proc-fimg-fvc1n-0073.fits" 11:57:19 boss exposure 900.0, file=337734 11:57:21 apogee.expose=apRaw-40670059, 43, A, Object 12:05:01 apogee.expose=apRaw-40670060, 43, B, Object 12:22Z enclosure closed calibrations started ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0040 0.0040 Axial B: 0.0000 0.0030 Axial C: -0.1120 -0.1150 Trans D: -0.0343 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 3.00 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6550 -9.6570 Axial B: -9.6680 -9.6710 Axial C: -9.6950 -9.6970 Trans D: 0.1240 0.1140 GALILS Commanded: 4893900 4892800 4910450 -50 -3450 Actual: 4859444 4859324 4859060 -50 -3450 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3857.05 0.11 -0.06 -0.04 42.8 ---- Telescope Status ---- 00:10Z Started ventilation 01:05Z encl25m : open 12:22Z encl25m : closed No interlocks bypassed Status at: 12:32Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = FPS Counterweights at: 687/704 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 199 lbs, 22 psi ---- Software Used ---- Get Versions, 02/19/22, 00:01:45 -- STUI software: STUI : 1.7.17b1 alerts : 2.2.0 boss : 6.0.1 -boss.specMec : 3.0.4 guider : FAILED ecamera : FAILED apo : 3.0.1 fliswarm : 0.2.1-alpha.0 jaeger : 0.14.1-alpha.0 kaiju : 1.2.2-alpha.1 coordio : 1.2.2-alpha.1 fps_calibrati : APO.2021.12.25+960cb05 cherno : 0.1.1-alpha.0 hal : 0.1.1-alpha.0 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.12 --------------Other Versions-------------- sdss-obstools : 3.7.10 TPM : 1.2-31-gd7eb MCP : 2.1 PLC : V34 Fiducials : v1_148 idlspec2d : v6_0_5 Kronos : 0.1.0 roboscheduler : 0.11.1 /dev/sdb1 2.0T 169G 1.9T 9% /home 10.25.1.230:/data 21T 17T 3.1T 85% /data ---- APOGEE Data Summary ---- APOGEE data list, mjd=59629, all raw_data: /data/apogee/utr_cdr/59629/apRaw-*.fits archive: /data/apogee/archive/59629/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59629/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59629 1 no file found: /data/apogee/utr_cdr/59629/apRaw-40670001.fits cannot read file : /data/apogee/utr_cdr/59629/apRaw-40670002.fits 59629 3 no file found: /data/apogee/utr_cdr/59629/apRaw-40670003.fits cannot read file : /data/apogee/utr_cdr/59629/apRaw-40670004.fits 59629 5 23:47 40670005 Dark 3 A 13.0 ---------- a-b-c - - 59629 6 23:52 40670006 Dark 3 A 13.0 ---------- a-b-c - - test 59629 7 23:53 40670007 Object 3 A 13.0 --------- a-b-c -6.55 - test 59629 8 23:55 40670008 Object 6 A 13.0 --------- a-b-c err - test 59629 9 00:08 40670009 Dark 60 A 13.0 ---------- a-b-c - - 59629 10 00:10 40670010 QuarFlat 10 A 13.0 ---------- a-b-c - 100 59629 11 00:13 40670011 Arc-Thar 12 B 13.5 ---------- a-b-c 0.48 - 59629 12 00:20 40670012 Arc-Une 40 B 13.5 ---------- a-b-c - - 59629 13 00:22 40670013 Arc-Thar 12 A 13.0 ---------- a-b-c -0.10 - 59629 14 00:29 40670014 Arc-Une 40 A 13.0 ---------- a-b-c - - 59629 15 00:40 40670015 Dark 60 A 13.0 ---------- a-b-c - - 59629 16 00:46 40670016 Arc---- 30 A 13.0 ---------- a-b-c - - FPI 59629 17 02:01 40670017 Object 6 A 13.0 4823-3583 a-b-c 0.17 - 59629 18 02:05 40670018 Object 6 A 13.0 4823-3583 a-b-c 0.19 - 59629 19 02:13 40670019 Arc---- 3 B 13.5 ---------- a-b-c - - test 59629 20 02:23 40670020 Object 43 A 13.0 4823-3583 a-b-c -0.10 - 59629 21 02:30 40670021 Object 43 B 13.5 4823-3583 a-b-c 0.40 - 59629 22 02:41 40670022 Object 43 A 13.0 4823-3584 a-b-c -0.10 - 59629 23 02:49 40670023 Object 43 B 13.5 4823-3585 a-b-c 0.39 - 59629 24 02:58 40670024 Object 43 A 13.0 4823-3585 a-b-c -0.09 - 59629 25 03:06 40670025 Object 43 B 13.5 4823-3586 a-b-c 0.40 - 59629 26 03:15 40670026 Object 43 A 13.0 4823-3586 a-b-c -0.10 - 59629 27 03:22 40670027 Object 43 B 13.5 4823-3586 a-b-c 0.40 - 59629 28 03:25 40670028 DomeFlat 5 B 13.5 4823-3586 a-b-c - 144 59629 29 03:54 40670029 Object 43 A 13.0 12952-3588 a-b-c -0.09 - 59629 30 04:02 40670030 Object 43 B 13.5 12952-3588 a-b-c 0.41 - 59629 31 04:05 40670031 DomeFlat 5 B 13.5 12952-3588 a-b-c - 144 59629 32 04:33 40670032 Object 43 A 13.0 15599-3590 a-b-c -0.10 - 59629 33 04:40 40670033 Object 43 B 13.5 15599-3590 a-b-c 0.40 - 59629 34 04:47 40670034 Object 20 A 13.0 15599-3590 a-b-c -0.04 - 59629 35 04:50 40670035 DomeFlat 5 B 13.5 15599-3590 a-b-c - 146 59629 36 05:30 40670036 Arc---- 3 B 13.5 ---------- a-b-c - - test 59629 37 05:50 40670037 Object 43 A 13.0 15680-3591 a-b-c -0.10 - 59629 38 05:58 40670038 Object 43 B 13.5 15680-3591 a-b-c 0.41 - 59629 39 06:07 40670039 Object 43 A 13.0 15680-3591 a-b-c -0.10 - 59629 40 06:14 40670040 Object 43 B 13.5 15680-3591 a-b-c 0.41 - 59629 41 06:17 40670041 DomeFlat 5 B 13.5 15680-3591 a-b-c - 147 59629 42 06:44 40670042 Object 33 A 13.0 15268-3592 a-b-c -0.05 - 59629 43 06:53 40670043 Object 3 B 13.5 15268-3592 a-b-c 0.40 - test 59629 44 07:01 40670044 Object 43 A 13.0 15268-3592 a-b-c -0.09 - 59629 45 07:08 40670045 Object 43 B 13.5 15268-3592 a-b-c 0.41 - 59629 46 07:10 40670046 DomeFlat 5 B 13.5 15268-3592 a-b-c - 147 59629 47 07:29 40670047 Object 43 A 13.0 721-3593 a-b-c -0.09 - 59629 48 07:37 40670048 Object 43 B 13.5 721-3593 a-b-c 0.41 - 59629 49 07:39 40670049 DomeFlat 5 B 13.5 721-3593 a-b-c - 145 59629 50 07:59 40670050 Object 43 A 13.0 10510-3594 a-b-c -0.09 - 59629 51 08:06 40670051 Object 43 B 13.5 10510-3594 a-b-c 0.41 - 59629 52 08:22 40670052 Object 22 A 13.0 1218-3595 a-b-c -0.04 - 59629 53 08:30 40670053 Object 3 B 13.5 1218-3595 a-b-c 0.42 - test cannot read file : /data/apogee/utr_cdr/59629/apRaw-40670054.fits 59629 55 08:47 40670055 Dark 3 A 13.0 ---------- a-b-c - - 59629 56 08:54 40670056 Object 23 A 13.0 1218-3595 a-b-c -0.03 - 59629 57 09:19 40670057 Object 43 A 13.0 1219-3600 a-b-c 0.41 - 59629 58 09:26 40670058 Object 43 B 13.5 1219-3600 a-b-c 0.94 - 59629 59 12:04 40670059 Object 43 A 13.0 5684-3614 a-b-c 0.42 - 59629 60 12:12 40670060 Object 43 B 13.5 5684-3614 a-b-c 0.93 - 59629 61 12:14 40670061 DomeFlat 5 B 13.5 5684-3614 a-b-c - 145 cannot read file : /data/apogee/utr_cdr/59629/apRaw-40670062.fits ------------------------------------------------------------------------------ ---- eBOSS Data Summary ---- Reading FITS from /data/spectro//59629 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 23:43 b1-00337670 ----------- ---- bia Clo 0.0 -5392 -5246 -6634 2 23:43 r1-00337670 ----------- ---- bia Clo 0.0 -5392 -5246 -6634 3 23:52 b1-00337671 21652-20867 3582 fla Out 20.1 -5392 -5246 -6634 4 23:52 r1-00337671 21652-20867 3582 fla Out 20.1 -5392 -5246 -6634 5 23:55 b1-00337672 21652-20867 3582 arc Lef 4.1 -5392 -5246 -6634 HART 6 23:55 r1-00337672 21652-20867 3582 arc Lef 4.1 -5392 -5246 -6634 HART 7 23:55 b1-00337673 21652-20867 3582 arc Rig 4.1 -5392 -5246 -6634 HART 8 23:55 r1-00337673 21652-20867 3582 arc Rig 4.1 -5392 -5246 -6634 HART 9 23:59 b1-00337674 21652-20867 3582 arc Out 4.1 -5361 -5217 -6608 10 23:59 r1-00337674 21652-20867 3582 arc Out 4.1 -5361 -5217 -6608 11 01:30 b1-00337675 4823-17057 3583 arc Lef 4.1 -5361 -5217 -6608 HART 12 01:30 r1-00337675 4823-17057 3583 arc Lef 4.1 -5361 -5217 -6608 HART 13 01:31 r1-00337676 4823-17057 3583 arc Rig 4.1 -5361 -5217 -6608 HART 14 01:31 b1-00337676 4823-17057 3583 arc Rig 4.1 -5361 -5217 -6608 HART 15 01:32 b1-00337677 4823-17057 3583 arc Out 4.1 -1909 -1765 -3155 16 01:32 r1-00337677 4823-17057 3583 arc Out 4.1 -1909 -1765 -3155 17 01:35 b1-00337678 4823-17057 3583 fla Out 25.1 -1909 -1765 -3155 18 01:35 r1-00337678 4823-17057 3583 fla Out 25.1 -1909 -1765 -3155 19 02:13 r1-00337681 ----------- ---- bia Clo 0.0 -1909 -1765 -3155 20 02:13 b1-00337681 ----------- ---- bia Clo 0.0 -1909 -1765 -3155 21 02:15 r1-00337682 4823-17057 3583 sci Out 900.1 -1909 -1765 -3155 22 02:15 b1-00337682 4823-17057 3583 sci Out 900.1 -1909 -1765 -3155 23 02:34 r1-00337683 4823-17057 3584 sci Out 900.1 -1909 -1765 -3155 24 02:34 b1-00337683 4823-17057 3584 sci Out 900.1 -1909 -1765 -3155 25 02:51 r1-00337684 4823-17057 3585 sci Out 900.1 -1909 -1765 -3155 26 02:51 b1-00337684 4823-17057 3585 sci Out 900.1 -1909 -1765 -3155 27 03:08 r1-00337685 4823-17057 3586 sci Out 900.1 -1909 -1765 -3155 28 03:08 b1-00337685 4823-17057 3586 sci Out 900.1 -1909 -1765 -3155 29 03:36 b1-00337686 12952-19154 3588 arc Lef 4.1 -1909 -1765 -3155 HART 30 03:36 r1-00337686 12952-19154 3588 arc Lef 4.1 -1909 -1765 -3155 HART 31 03:36 r1-00337687 12952-19154 3588 arc Rig 4.1 -1909 -1765 -3155 HART 32 03:36 b1-00337687 12952-19154 3588 arc Rig 4.1 -1909 -1765 -3155 HART 33 03:37 b1-00337688 12952-19154 3588 arc Out 4.1 -3183 -3039 -4429 34 03:37 r1-00337688 12952-19154 3588 arc Out 4.1 -3183 -3039 -4429 35 03:41 r1-00337689 12952-19154 3588 arc Out 4.1 -3183 -3039 -4429 36 03:41 b1-00337689 12952-19154 3588 arc Out 4.1 -3183 -3039 -4429 37 03:43 b1-00337690 12952-19154 3588 fla Out 25.1 -3183 -3039 -4429 38 03:43 r1-00337690 12952-19154 3588 fla Out 25.1 -3183 -3039 -4429 39 03:47 r1-00337691 12952-19154 3588 sci Out 900.1 -3183 -3039 -4429 40 03:47 b1-00337691 12952-19154 3588 sci Out 900.1 -3183 -3039 -4429 41 04:08 r1-00337692 15599-19642 3590 arc Lef 4.1 -3183 -3039 -4429 HART 42 04:08 b1-00337692 15599-19642 3590 arc Lef 4.1 -3183 -3039 -4429 HART 43 04:08 b1-00337693 15599-19642 3590 arc Rig 4.1 -3183 -3039 -4429 HART 44 04:08 r1-00337693 15599-19642 3590 arc Rig 4.1 -3183 -3039 -4429 HART 45 04:09 r1-00337694 15599-19642 3590 arc Out 4.1 -3472 -3328 -4718 46 04:09 b1-00337694 15599-19642 3590 arc Out 4.1 -3472 -3328 -4718 47 04:12 r1-00337695 15599-19642 3590 fla Out 25.1 -3472 -3328 -4718 48 04:12 b1-00337695 15599-19642 3590 fla Out 25.1 -3472 -3328 -4718 49 04:26 r1-00337696 15599-19642 3590 sci Out 900.1 -3472 -3328 -4718 50 04:26 b1-00337696 15599-19642 3590 sci Out 900.1 -3472 -3328 -4718 51 05:27 r1-00337701 ----------- ---- bia Clo 0.0 -3472 -3328 -4718 52 05:27 b1-00337701 ----------- ---- bia Clo 0.0 -3472 -3328 -4718 53 05:30 b1-00337702 15680-19655 3591 arc Lef 4.1 -3472 -3328 -4718 HART 54 05:30 r1-00337702 15680-19655 3591 arc Lef 4.1 -3472 -3328 -4718 HART 55 05:31 b1-00337703 15680-19655 3591 arc Rig 4.1 -3472 -3328 -4718 HART 56 05:31 r1-00337703 15680-19655 3591 arc Rig 4.1 -3472 -3328 -4718 HART 57 05:32 b1-00337704 15680-19655 3591 arc Out 4.1 -4403 -4260 -5652 58 05:32 r1-00337704 15680-19655 3591 arc Out 4.1 -4403 -4260 -5652 59 05:35 r1-00337705 15680-19655 3591 fla Out 25.1 -4403 -4260 -5652 60 05:35 b1-00337705 15680-19655 3591 fla Out 25.1 -4403 -4260 -5652 61 05:43 b1-00337706 15680-19655 3591 sci Out 900.1 -4403 -4260 -5652 62 05:43 r1-00337706 15680-19655 3591 sci Out 900.1 -4403 -4260 -5652 63 06:00 b1-00337707 15680-19655 3591 sci Out 900.1 -4403 -4260 -5652 64 06:00 r1-00337707 15680-19655 3591 sci Out 900.1 -4403 -4260 -5652 65 06:28 b1-00337708 15268-19583 3592 arc Lef 4.1 -4403 -4260 -5652 HART 66 06:28 r1-00337708 15268-19583 3592 arc Lef 4.1 -4403 -4260 -5652 HART 67 06:29 r1-00337709 15268-19583 3592 arc Rig 4.1 -4403 -4260 -5652 HART 68 06:29 b1-00337709 15268-19583 3592 arc Rig 4.1 -4403 -4260 -5652 HART 69 06:30 b1-00337710 15268-19583 3592 arc Out 4.1 -5170 -5029 -6421 70 06:30 r1-00337710 15268-19583 3592 arc Out 4.1 -5170 -5029 -6421 71 06:33 b1-00337711 15268-19583 3592 fla Out 25.1 -5170 -5029 -6421 72 06:33 r1-00337711 15268-19583 3592 fla Out 25.1 -5170 -5029 -6421 73 06:53 b1-00337713 15268-19583 3592 sci Out 900.1 -5170 -5029 -6421 74 06:53 r1-00337713 15268-19583 3592 sci Out 900.1 -5170 -5029 -6421 75 07:13 b1-00337714 721-16340 3593 arc Lef 4.1 -5170 -5029 -6421 HART 76 07:13 r1-00337714 721-16340 3593 arc Lef 4.1 -5170 -5029 -6421 HART 77 07:14 b1-00337715 721-16340 3593 arc Rig 4.1 -5170 -5029 -6421 HART 78 07:14 r1-00337715 721-16340 3593 arc Rig 4.1 -5170 -5029 -6421 HART 79 07:15 r1-00337716 721-16340 3593 arc Out 4.1 -6266 -6122 -7513 80 07:15 b1-00337716 721-16340 3593 arc Out 4.1 -6266 -6122 -7513 81 07:18 b1-00337717 721-16340 3593 fla Out 25.1 -6266 -6122 -7513 82 07:18 r1-00337717 721-16340 3593 fla Out 25.1 -6266 -6122 -7513 83 07:22 r1-00337718 721-16340 3593 sci Out 900.1 -6266 -6122 -7513 84 07:22 b1-00337718 721-16340 3593 sci Out 900.1 -6266 -6122 -7513 85 07:51 r1-00337719 10510-18365 3594 sci Out 900.1 -6266 -6122 -7513 86 07:51 b1-00337719 10510-18365 3594 sci Out 900.1 -6266 -6122 -7513 87 08:25 b1-00337721 ----------- ---- bia Clo 0.0 -6266 -6122 -7513 88 08:25 r1-00337721 ----------- ---- bia Clo 0.0 -6266 -6122 -7513 89 08:31 r1-00337722 1218-16577 3595 sci Out 900.1 -6266 -6122 -7513 90 08:31 b1-00337722 1218-16577 3595 sci Out 900.1 -6266 -6122 -7513 91 09:03 b1-00337724 1219-16577 3600 arc Lef 4.1 -6266 -6122 -7513 HART 92 09:03 r1-00337724 1219-16577 3600 arc Lef 4.1 -6266 -6122 -7513 HART 93 09:04 b1-00337725 1219-16577 3600 arc Rig 4.1 -6266 -6122 -7513 HART 94 09:04 r1-00337725 1219-16577 3600 arc Rig 4.1 -6266 -6122 -7513 HART 95 09:05 r1-00337726 1219-16577 3600 arc Out 4.1 -6231 -6089 -7482 96 09:05 b1-00337726 1219-16577 3600 arc Out 4.1 -6231 -6089 -7482 97 09:08 r1-00337727 1219-16577 3600 fla Out 25.1 -6231 -6089 -7482 98 09:08 b1-00337727 1219-16577 3600 fla Out 25.1 -6231 -6089 -7482 99 09:12 r1-00337728 1219-16577 3600 sci Out 900.0 -6231 -6089 -7482 100 09:12 b1-00337728 1219-16577 3600 sci Out 900.0 -6231 -6089 -7482 101 09:39 r1-00337729 13696-19348 3601 sci Out 900.1 -6231 -6089 -7482 102 09:39 b1-00337729 13696-19348 3601 sci Out 900.1 -6231 -6089 -7482 103 10:04 r1-00337730 13695-19348 3604 sci Out 900.1 -6231 -6089 -7482 104 10:04 b1-00337730 13695-19348 3604 sci Out 900.1 -6231 -6089 -7482 105 10:37 r1-00337731 8110-17818 3605 sci Out 900.1 -6231 -6089 -7482 106 10:37 b1-00337731 8110-17818 3605 sci Out 900.1 -6231 -6089 -7482 107 11:05 r1-00337732 4583-16963 3608 sci Out 900.1 -6231 -6089 -7482 108 11:05 b1-00337732 4583-16963 3608 sci Out 900.1 -6231 -6089 -7482 109 11:30 b1-00337733 4582-16963 3611 sci Out 900.1 -6231 -6089 -7482 110 11:30 r1-00337733 4582-16963 3611 sci Out 900.1 -6231 -6089 -7482 111 11:57 b1-00337734 5684-17315 3614 sci Out 900.1 -6231 -6089 -7482 112 11:57 r1-00337734 5684-17315 3614 sci Out 900.1 -6231 -6089 -7482 113 12:24 b1-00337735 ----------- ---- bia Clo 0.0 -6231 -6089 -7482 114 12:24 r1-00337735 ----------- ---- bia Clo 0.0 -6231 -6089 -7482 115 12:26 b1-00337736 ----------- ---- bia Clo 0.0 -6231 -6089 -7482 116 12:26 r1-00337736 ----------- ---- bia Clo 0.0 -6231 -6089 -7482 117 12:27 b1-00337737 ----------- ---- bia Clo 0.0 -6231 -6089 -7482 118 12:27 r1-00337737 ----------- ---- bia Clo 0.0 -6231 -6089 -7482 119 12:28 b1-00337738 ----------- ---- bia Clo 0.0 -6231 -6089 -7482 120 12:28 r1-00337738 ----------- ---- bia Clo 0.0 -6231 -6089 -7482 121 12:30 r1-00337739 ----------- ---- bia Clo 0.0 -6231 -6089 -7482 122 12:30 b1-00337739 ----------- ---- bia Clo 0.0 -6231 -6089 -7482 123 12:31 b1-00337740 ----------- ---- bia Clo 0.0 -6231 -6089 -7482 124 12:31 r1-00337740 ----------- ---- bia Clo 0.0 -6231 -6089 -7482 ---- BOSS Spectrographs Collimation ---- Time TSP1 R1 B1 Move1 B1Resid ============================================= 23:56:30 4.2 -329 -1.2 28 -1.1 01:31:40 3.1 3628 -10.3 3453 0.5 03:37:20 1.3 -1649 2.8 -1274 -1.2 04:09:20 1.1 0 1.8 -289 0.9 05:31:30 1.0 0 5.8 -931 2.9 06:29:50 0.9 329 5.8 -767 3.4 07:14:40 0.9 -329 5.8 -1096 2.4 09:04:40 0.4 0 -0.2 32 -0.1 ---- Weather Log ---- Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma 19:00 0.8 -10.8 1.1 201.5 330 19:15 0.1 -10.0 1.5 302.9 431 19:30 0.4 -9.7 11.5 2.1 377 19:45 0.8 -7.8 11.7 22.5 259 20:00 1.4 -15.8 12.7 355.0 354 20:15 1.4 -11.2 8.0 356.4 316 20:30 1.3 -7.6 4.4 326.3 368 20:45 1.6 -11.6 5.7 3.2 331 21:00 1.8 -8.7 8.9 327.3 394 21:15 1.6 -10.0 10.7 359.6 392 21:30 1.4 -7.7 5.7 343.8 473 21:45 2.0 -8.7 4.0 331.6 437 22:00 2.2 -12.9 12.4 2.6 329 22:15 2.1 -13.8 12.5 358.5 207 22:30 2.0 -9.7 9.9 346.6 379 22:45 2.3 -12.8 1.8 352.0 293 23:00 2.2 -11.4 8.4 331.0 298 23:15 1.6 -10.4 7.6 337.5 435 23:30 1.7 -11.4 9.0 355.2 313 23:45 1.8 -13.8 6.5 356.4 246 00:00 1.3 -11.2 2.5 348.4 322 00:15 1.3 -12.9 0.8 223.3 234 00:30 1.7 -15.6 6.3 5.0 228 00:45 1.6 -17.2 0.9 273.1 174 01:00 1.2 -16.8 11.2 353.6 159 01:15 1.1 -18.5 7.1 356.2 127 01:30 0.8 -18.9 12.0 353.9 98 84 7.05 01:45 0.1 -17.1 7.4 2.0 111 94 8.36 02:00 0.0 -17.2 9.6 356.0 109 141 3.61 02:15 0.4 -18.6 1.5 20.1 115 117 5.22 02:30 0.1 -18.2 3.8 0.6 90 84 6.74 02:45 0.1 -17.8 6.2 7.9 126 193 7.81 03:00 -0.5 -16.5 9.9 23.4 103 207 12.80 03:15 -0.1 -16.6 8.6 0.0 152 183 2.20 03:30 -0.4 -16.9 7.5 357.5 158 84 8.11 03:45 -0.4 -18.0 5.9 353.0 94 136 6.09 04:00 -0.9 -16.8 7.5 354.4 117 122 3.17 04:15 -1.2 -15.1 1.0 82.5 155 84 7.93 04:30 -1.4 -16.2 2.0 347.1 145 84 13.10 04:45 -0.6 -16.7 7.1 353.8 107 89 11.60 05:00 -0.3 -15.4 10.4 359.0 93 136 3.36 05:15 0.4 -15.2 17.3 13.7 73 113 5.76 05:30 0.4 -14.6 15.3 7.5 110 122 3.09 05:45 0.3 -14.3 14.6 44.1 167 160 4.63 06:00 0.6 -14.3 16.9 32.3 145 84 7.96 06:15 0.6 -13.8 6.1 8.8 195 84 6.41 06:30 1.0 -13.8 9.5 357.0 164 89 7.99 06:45 0.7 -13.8 11.0 350.8 184 160 4.74 07:00 0.5 -14.0 6.4 2.6 153 141 2.26 07:15 0.5 -13.3 12.1 14.0 159 226 13.30 07:30 0.7 -12.9 12.0 355.0 204 240 18.13 07:45 -0.1 -13.4 0.6 294.0 191 117 9.18 08:00 0.2 -12.9 12.8 20.8 196 84 4.49 08:15 0.1 -12.7 12.5 358.5 166 183 7.95 08:30 0.2 -12.2 9.9 28.5 181 155 2.47 08:45 0.2 -12.1 16.8 29.2 207 84 10.61 09:00 0.4 -12.4 14.7 48.7 191 169 7.55 09:15 0.2 -12.9 11.8 44.5 136 84 10.04 09:30 1.1 -14.3 7.4 52.8 106 198 8.94 09:45 0.8 -14.7 9.4 48.3 97 155 2.37 10:00 1.4 -15.6 6.0 96.3 75 84 6.22 10:15 0.2 -14.7 9.1 85.7 115 89 12.68 10:30 0.7 -14.9 7.4 91.2 104 165 8.51 10:45 0.3 -14.9 9.1 92.5 102 150 4.13 11:00 0.1 -14.7 7.4 92.0 93 150 2.93 11:15 0.6 -14.9 5.2 119.6 103 94 13.45 11:30 0.8 -15.3 2.1 124.6 99 108 6.80 11:45 0.7 -15.4 4.1 163.5 71 174 3.84 12:00 0.8 -15.6 9.7 140.9 67 146 4.30 12:15 1.0 -15.6 8.4 130.5 83 198 6.91 12:30 0.6 -15.4 9.7 133.8 76 179 3.82 ---- Problems in Detail ---- Around 04:47 APOGEE stopped working on 20th read of science exposure: 59629 34 04:47 40670034 Object 20 A 13.0 15599-3590 a-b-c -0.04 - It communicated via HAL with BOSS side, and ongoing exposure was aborted. The abortion process was not clear, so the BOSS controller hanged. Note that HAL widget in Boson did not show exposures in red, and we thought that science exposures were going. Finally, to fix it, we tried short APOGEE 3-read dark - it worked; then restarted bossICC. ==== APOGEE exposure failures attempted to fix by restarting apogee actor (by Jose) . After that we got serious alert on APOGEE arrayPower, like 8:56:35 .hub 1020 apogee i dspload="?"; arrayPower=?; cameraState=done; utrReadTime=10.647; dayNumber=4067 and fixed it by issuing apogee arrayPower state=on