Subject: 2.5m obslog 20211217 (SJD 59566) From: dmbiz@apo.nmsu.edu Date: Sat, 18 Dec 2021 11:59:01 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday December 17, 2021 (SJD 59566) ---- Observing Team ---- Karen Kinemuchi Dmitry Bizyaev Jose Sanchez-Gallego (Commissioning, support in STUI later) Observers in Zoom: Kaike, DanO, Audrey, Viktor, Katie ---- Observing Plan ---- FPS commissioning ---- Observing Summary ---- THIS IS THE CORRECT LOG - first one was sent too early Partly cloudy night, then completely overcast in its second half. We attempted to do the robot dither data in design 158 (BOSS) but it interfered with clouds. The robot dither data were obtained for designs 44 (BOSS) and 43 (APOGEE) The #43 was significantly affected by clouds for the dithers #8 and #9. The first 7 went on mostly clear sky. During a wave of clouds we performed the test "Improve performance of FVC loop" (from the cloudy night tasks) for -m -k 5 0.7 8 0.8 10 0.9 15 1.0 Later we completed the whole grid for an APOGEE design, so it was -m -k 5 0.7 0.8 0.9 1.0 8 0.7 0.8 0.9 1.0 10 0.7 0.8 0.9 1.0 15 0.7 0.8 0.9 1.0 This completes Jose's cloudy night task. We also attempted to operate the "Wht" lamp via the STUI MCP window, but it did not work. We'll check again tomorrow during the day time. Notes ----- - we restarted jaeger on sdss5-fps 3 times because it lost connection to ieb. According to Jose, this is a software communication issue and not a real power off of ieb. See the log and Problems in Detail. - At the first connection to ieb loss, Grafana stopped displaying new information about FPS. We were unable to restart Grafana. Software developers and day staff, please look at those 2 items. - dust collected: 4561. - Updated Jose's command for the dome flatturns FF lamp correctly for the 3rd read, but also turns it on again for the 4th read. ---- Observing Log ---- 22:35Z Clear on site AFTERNOON CHECKOUT: =================== 23:26Z Evening cals going. Grabbed mirror numbers - ok; BOSS SPECTROGRAPH CALS: 23:39:58 boss exposure 0.0, file=335406 23:42:04 boss exposure 20.0, file=335407 23:45:49 boss exposure 4.0, file=335410 23:46:29 boss exposure 4.0, file=335411 23:47:10 sp1.motor.move= 345, 346, 345 23:47:10 Hartmann collimate output on cart #19 sp1: offset: r= -659 (In Focus); b= -4.2 (Out of focus) sp1: pred. move: spAverageMove= 345 sp1: pred. resid: r= -1004, b= -3.1, txt= OK, spTemp = 4.8 Will check hartmann later in the night to account for ambient temp change. 23:49:13 boss exposure 4.0, file=335412 23:51:04 boss exposure 30.0, file=335413 23:52Z Ventilation begins. Telescope ready. 00:16Z All APOGEE cals are done. Night ----- 0:34Z enclosure open The sky is getting cloudy becaus eof orographic clouds, but we stil have a chance to at least a BOSS dithering test. 02:00:56 CmdStarted: OBSERVER.karen 88 tcc track 345, 20 icrs /rota=0 /rotty=object 02:02:54 CmdStarted: OBSERVER.karen 89 jaeger unwind - error, unlock first! 2:03:18 CmdDone: OBSERVER.karen 90 jaeger unlock 02:03:29 CmdStarted: OBSERVER.karen 91 jaeger unwind focusing first 02:05:15 CmdStarted: OBSERVER.karen 93 fliswarm talk -c gfa expose 15 The best focus is at 382 02:13:30 CmdStarted: OBSERVER.karen 109 jaeger configuration load 158 02:13:37 OBSERVER.karen 109 jaeger i text="Assigned targets 464. Valid targets 461." 02:23:05 Hartmann collimate output on cart #19 sp1: offset: r= 659 (In Focus); b= 3.8 (In Focus) sp1: pred. move: spAverageMove= -281 sp1: pred. resid: r= 940, b= 2.9, txt= OK, spTemp = 2.3 Took BOSS arc and flat here 02:27:04 CmdDone: OBSERVER.karen 124 tcc axis stop 02:27:12 CmdStarted: jaeger.fliswarm 6 fliswarm talk -c fvc expose 5.0 02:27:38 OBSERVER.karen 125 jaeger i text="[FVC]: RMS full fit 72.740 um." 02:28:20 OBSERVER.karen 125 jaeger i text="[FVC]: RMS full fit 31.540 um." 02:29:02 OBSERVER.karen 125 jaeger i text="[FVC]: RMS full fit 24.040 um." 02:29:44 OBSERVER.karen 125 jaeger i text="[FVC]: RMS full fit 23.110 um." delta target is reached, but not rms yet. Started second time 02:30:36 OBSERVER.karen 126 jaeger i text="[FVC]: RMS full fit 4.270 um." 02:29:48 CmdDone: OBSERVER.karen 125 jaeger fvc loop 02:34:00 CmdStarted: OBSERVER.karen 127 jaeger configuration slew Applying offsets in cherno: 02:34:37 CmdDone: OBSERVER.karen 128 cherno offset 0 0 -432 02:34:55 CmdStarted: OBSERVER.karen 129 cherno acquire -c -t 15 way off 02:36:51 CmdDone: OBSERVER.karen 130 cherno set pid radec k 1.0 02:37:03 CmdDone: OBSERVER.karen 131 cherno set pid rot k 1.0 Changing PID doesn't help. Stopping cherno 02:38:10 CmdDone: OBSERVER.karen 132 cherno stop Was: 02:37:54 OBSERVER.karen 129 cherno i astrometry_fit=14,5,-999.0,-999.0,1.08,0.09,-0.379,8.958,7.825,-958.4,0.999872 Trying another direction for the offset 02:39:03 CmdDone: OBSERVER.karen 133 cherno offset 0 0 500 Now: 02:38:17 OBSERVER.karen 129 cherno i astrometry_fit=15,2,-999.0,-999.0,1.03,0.06,-0.27,9.119,7.068,-989.1,0.999966 Now ot looks better 02:40:03 OBSERVER.karen 134 cherno d guide_rms=17,0.048,0.052,0.071 2:40:48 CmdStarted: OBSERVER.karen 137 cherno show 02:40:48 OBSERVER.karen 137 cherno > 02:40:48 OBSERVER.karen 137 cherno i text="status: " 02:40:48 OBSERVER.karen 137 cherno i text="offset: (0.0, 0.0, 500.0)" 02:40:48 OBSERVER.karen 137 cherno i text="exposure_time: 15.0" 02:40:48 OBSERVER.karen 137 cherno i text="guide_loop: {'radec': {'max_correction': 600, 'min_correction': 0.05, 'pid': {'k': 0.7}}," 02:40:48 OBSERVER.karen 137 cherno i text=" 'rot': {'max_correction': 600," 02:40:48 OBSERVER.karen 137 cherno i text=" 'min_correction': 3," 02:40:48 OBSERVER.karen 137 cherno i text=" 'min_isolated_correction': 30," 02:40:48 OBSERVER.karen 137 cherno i text=" 'pid': {'k': 0.7}}}" Starting BOSS Base exposure 02:42:13 boss exposure 300.0, file=335422 02:49:24 CmdDone: jaeger.jaeger 19 jaeger configuration execute 02:49:24 OBSERVER.karen 140 jaeger : text="Dither configuration 364 loaded." 02:49:24 CmdDone: OBSERVER.karen 140 jaeger configuration dither 0.1 02:55:47 CmdStarted: OBSERVER.karen 143 jaeger configuration reverse 02:56:00 CmdStarted: OBSERVER.karen 144 jaeger configuration dither 0.1 02:56:08 CmdStarted: jaeger.jaeger 20 jaeger configuration execute 02:56:21 OBSERVER.karen 144 jaeger : text="Dither configuration 365 loaded." We note that it is very cloudy in our direction. 03:02:58 CmdDone: OBSERVER.karen 146 jaeger configuration reverse 03:02:58 CmdStarted: OBSERVER.karen 147 jaeger configuration dither 0.1 03:03:07 CmdStarted: jaeger.jaeger 21 jaeger configuration execute 03:03:20 OBSERVER.karen 147 jaeger : text="Dither configuration 366 loaded." 03:09:45 CmdStarted: OBSERVER.karen 150 jaeger configuration reverse 03:09:59 CmdStarted: OBSERVER.karen 151 jaeger configuration dither 0.1 03:10:07 CmdStarted: jaeger.jaeger 22 jaeger configuration execute 03:10:18 OBSERVER.karen 151 jaeger : text="Dither configuration 367 loaded." 03:10:18 CmdDone: OBSERVER.karen 151 jaeger configuration dither 0.1 03:16:41 CmdStarted: OBSERVER.karen 153 jaeger configuration reverse 03:16:52 CmdStarted: OBSERVER.karen 154 jaeger configuration dither 0.1 03:17:00 CmdStarted: jaeger.jaeger 23 jaeger configuration execute 03:17:13 OBSERVER.karen 154 jaeger : text="Dither configuration 368 loaded." 03:17:13 CmdDone: OBSERVER.karen 154 jaeger configuration dither 0.1 03:23:36 CmdStarted: OBSERVER.karen 156 jaeger configuration reverse 03:23:49 CmdStarted: OBSERVER.karen 157 jaeger configuration dither 0.1 03:23:58 CmdStarted: jaeger.jaeger 24 jaeger configuration execute 3:24:11 OBSERVER.karen 157 jaeger : text="Dither configuration 369 loaded." 03:24:11 CmdDone: OBSERVER.karen 157 jaeger configuration dither 0.1 03:30:34 CmdStarted: OBSERVER.karen 162 jaeger configuration reverse 03:30:47 CmdStarted: OBSERVER.karen 163 jaeger configuration dither 0.1 03:30:55 CmdStarted: jaeger.jaeger 25 jaeger configuration execute 03:31:08 OBSERVER.karen 163 jaeger : text="Dither configuration 370 loaded." 03:31:08 CmdDone: OBSERVER.karen 163 jaeger configuration dither 0.1 03:37:31 CmdStarted: OBSERVER.karen 165 jaeger configuration reverse 03:37:44 CmdStarted: OBSERVER.karen 166 jaeger configuration dither 0.1 03:37:52 CmdStarted: jaeger.jaeger 26 jaeger configuration execute 03:38:05 OBSERVER.karen 166 jaeger : text="Dither configuration 371 loaded." 03:38:06 CmdDone: OBSERVER.karen 166 jaeger configuration dither 0.1 03:44:29 CmdStarted: OBSERVER.karen 169 jaeger configuration reverse 03:44:41 CmdStarted: OBSERVER.karen 170 jaeger configuration dither 0.1 03:44:50 CmdStarted: jaeger.jaeger 27 jaeger configuration execute 3:45:02 OBSERVER.karen 170 jaeger : text="Dither configuration 372 loaded." 03:45:02 CmdDone: OBSERVER.karen 170 jaeger configuration dither 0.1 03:51:25 CmdStarted: OBSERVER.karen 172 jaeger configuration reverse 2:49:43 boss exposure 300.0, file=335423 02:56:40 boss exposure 300.0, file=335424 03:03:39 boss exposure 300.0, file=335425 03:10:37 boss exposure 300.0, file=335426 03:17:32 boss exposure 300.0, file=335427 03:24:30 boss exposure 300.0, file=335428 03:31:27 boss exposure 300.0, file=335429 03:38:24 boss exposure 300.0, file=335430 03:45:21 boss exposure 300.0, file=335431 This design is finished, but its results are severely affected by clouds. Jose suggested a cloudy night task. == Cloudy night tasks == Improve performance of FVC loop We go to a designed field, to take test at a real position in the sky 04:09:55 CmdDone: OBSERVER.karen 179 jaeger unwind APOGEE design 46, BOSS design 47 04:10:44 CmdStarted: OBSERVER.karen 180 jaeger configuration load 46 Axes Halted via the tcc axix stop ! Max corr k 5 0.7 [DEFAULT] ----------------------- it 1 04:13:53 OBSERVER.karen 182 jaeger i text="[FVC]: RMS full fit 65.420 um." /data/fcam/59566/proc-fimg-fvc1n-0006.fits" it 2 04:14:35 OBSERVER.karen 182 jaeger i text="[FVC]: RMS full fit 21.170 um." /data/fcam/59566/proc-fimg-fvc1n-0007.fits" it 3 04:15:17 OBSERVER.karen 182 jaeger i text="[FVC]: RMS full fit 7.730 um." /data/fcam/59566/proc-fimg-fvc1n-0008.fits" it 4 04:15:59 OBSERVER.karen 182 jaeger i text="[FVC]: RMS full fit 5.000 um." /data/fcam/59566/proc-fimg-fvc1n-0009.fits" it 5 04:16:41 OBSERVER.karen 182 jaeger i text="[FVC]: RMS full fit 4.640 um." /data/fcam/59566/proc-fimg-fvc1n-0010.fits" 04:16:45 CmdDone: OBSERVER.karen 182 jaeger fvc loop -k 0.7 -m 5 04:18:29 CmdDone: OBSERVER.karen 183 jaeger configuration reverse 04:20:31 CmdStarted: OBSERVER.karen 184 jaeger configuration execute 15 1.0 --------- 04:26:38 CmdStarted: OBSERVER.karen 185 jaeger fvc loop -k 1.0 -m 15 it 1 04:27:04 OBSERVER.karen 185 jaeger i text="[FVC]: RMS full fit 63.600 um." /data/fcam/59566/proc-fimg-fvc1n-0011.fits" it 2 ... failed 04:27:58 OBSERVER.karen 185 jaeger f error="Command 'fvc loop -k 1.0 -m 15' failed because of an uncaught error 'ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 13 is different from 18)'. See traceback in the log for more information." 04:27:58 CmdDone: OBSERVER.karen 185 jaeger fvc loop -k 1.0 -m 15 go back via reverse: 04:28:36 CmdStarted: OBSERVER.karen 186 jaeger configuration reverse trying again: 04:30:36 CmdStarted: OBSERVER.karen 187 jaeger configuration execute 04:31:33 CmdStarted: OBSERVER.karen 188 jaeger fvc loop -k 1.0 -m 15 it 1 04:31:59 OBSERVER.karen 188 jaeger i text="[FVC]: RMS full fit 63.870 um." /data/fcam/59566/fimg-fvc1n-0013.fits" it 2 04:32:40 OBSERVER.karen 188 jaeger i text="[FVC]: RMS full fit 6.700 um." image 14 it 3 04:33:22 OBSERVER.karen 188 jaeger i text="[FVC]: RMS full fit 4.660 um." image 15 04:33:26 OBSERVER.karen 188 jaeger : text="FVC target RMS reached." 04:34:07 CmdStarted: OBSERVER.karen 189 jaeger configuration reverse 04:35:30 CmdStarted: OBSERVER.karen 190 jaeger configuration execute 8 0.8 --------- 04:36:56 CmdStarted: OBSERVER.karen 191 jaeger fvc loop -k 0.8 -m 8 it 1 04:37:21 OBSERVER.karen 191 jaeger i text="[FVC]: RMS full fit 63.310 um." image 16 it 2 04:38:04 OBSERVER.karen 191 jaeger i text="[FVC]: RMS full fit 14.120 um." image 17 it 3 04:38:45 OBSERVER.karen 191 jaeger i text="[FVC]: RMS full fit 5.090 um." image 18 it 4 04:39:27 OBSERVER.karen 191 jaeger i text="[FVC]: RMS full fit 4.350 um." image 19 04:39:32 OBSERVER.karen 191 jaeger : text="FVC target RMS reached." 04:39:52 CmdStarted: OBSERVER.karen 192 jaeger configuration reverse 04:41:15 CmdStarted: OBSERVER.karen 193 jaeger configuration execute 10 0.9 --------- 04:42:16 CmdStarted: OBSERVER.karen 194 jaeger fvc loop -k 0.9 -m 10 it 1 04:42:41 OBSERVER.karen 194 jaeger i text="[FVC]: RMS full fit 61.840 um." image 20 it 2 04:43:23 OBSERVER.karen 194 jaeger i text="[FVC]: RMS full fit 8.430 um." image 21 it 3 04:44:04 OBSERVER.karen 194 jaeger i text="[FVC]: RMS full fit 3.310 um." image 22 04:44:09 OBSERVER.karen 194 jaeger : text="FVC target RMS reached." 04:44:33 CmdStarted: OBSERVER.karen 195 jaeger configuration reverse === The sky is clear again. We resume the dithering tests 04:46:29 CmdStarted: OBSERVER.karen 196 jaeger unwind 04:47:14 CmdDone: OBSERVER.karen 197 tcc axis init design RA Dec PA Targets 43 | 75 | 0 | 0 | 275 | APOGEE 44 | 75 | 0 | 0 | 448 | BOSS We decided to refocus first. 04:47:52 CmdStarted: OBSERVER.karen 198 tcc track 75, 0 icrs /rota=0 /rotty=object Best focus: 336 (not a big change) Starting with BOSS: 04:57:40 CmdStarted: OBSERVER.karen 214 jaeger configuration load 44 05:00:21 CmdStarted: OBSERVER.karen 215 jaeger configuration execute Lamps on, Petals closed 05:01:37 CmdStarted: OBSERVER.karen 219 hartmann collimate 05:03:20 Hartmann collimate output on cart #19 sp1: offset: r= -329 (In Focus); b= 8.9 (Out of focus) sp1: pred. move: spAverageMove= -1578 sp1: pred. resid: r= 1249, b= 3.9, txt= OK, spTemp = 1.9 Moved blue ring +4 deg 05:09:00 Hartmann collimate output on cart #19 sp1: offset: r= 1319 (Out of focus); b= -1.2 (In Focus) sp1: pred. move: spAverageMove= 852 sp1: pred. resid: r= 467, b= 1.5, txt= OK, spTemp = 1.9 05:09:50 boss exposure 4.0, file=335436 05:11:21 boss exposure 30.0, file=335437 05:13:11 CmdStarted: OBSERVER.karen 228 tcc axis stop 05:14:03 CmdStarted: OBSERVER.karen 229 jaeger fvc loop -k 0.7 -m 5 05:14:29 OBSERVER.karen 229 jaeger i text="[FVC]: RMS full fit 56.690 um." 05:15:10 OBSERVER.karen 229 jaeger i text="[FVC]: RMS full fit 20.490 um." 05:15:52 OBSERVER.karen 229 jaeger i text="[FVC]: RMS full fit 11.280 um." 05:16:33 OBSERVER.karen 229 jaeger i text="[FVC]: RMS full fit 9.470 um." 05:17:16 OBSERVER.karen 229 jaeger i text="[FVC]: RMS full fit 9.410 um." 05:17:20 OBSERVER.karen 229 jaeger w text="Delta RMS reached. Cancelling the FVC loop." 05:17:33 CmdStarted: OBSERVER.karen 230 jaeger configuration slew 05:17:59 CmdDone: OBSERVER.karen 231 cherno offset 0 0 -432 05:18:19 CmdStarted: OBSERVER.karen 232 cherno acquire -c -t 15 05:22:30 OBSERVER.karen 232 cherno d guide_rms=149,0.046,0.051,0.069 Base BOSS exposure 05:22:48 CmdStarted: OBSERVER.karen 233 boss exposure science itime=300 05:23:08 boss exposure 300.0, file=335438 05:29:52 CmdDone: jaeger.jaeger 81 jaeger configuration execute 05:29:52 CmdDone: OBSERVER.karen 234 jaeger configuration dither 0.1 05:29:52 CmdDone: jaeger.jaeger 81 jaeger configuration execute 05:29:52 OBSERVER.karen 234 jaeger : text="Dither configuration 375 loaded." 05:29:52 CmdDone: OBSERVER.karen 234 jaeger configuration dither 0.1 05:36:15 CmdStarted: OBSERVER.karen 236 jaeger configuration reverse 05:36:28 CmdStarted: OBSERVER.karen 237 jaeger configuration dither 0.1 05:36:49 CmdDone: jaeger.jaeger 82 jaeger configuration execute 05:36:49 OBSERVER.karen 237 jaeger : text="Dither configuration 376 loaded." 05:36:49 CmdDone: OBSERVER.karen 237 jaeger configuration dither 0.1 05:43:25 CmdDone: OBSERVER.karen 239 jaeger configuration reverse 05:43:25 CmdStarted: OBSERVER.karen 240 jaeger configuration dither 0.1 05:43:46 OBSERVER.karen 240 jaeger : text="Dither configuration 377 loaded." 05:43:46 CmdDone: OBSERVER.karen 240 jaeger configuration dither 0.1 05:50:09 CmdStarted: OBSERVER.karen 242 jaeger configuration reverse 05:50:43 CmdDone: jaeger.jaeger 84 jaeger configuration execute 05:50:43 OBSERVER.karen 243 jaeger : text="Dither configuration 378 loaded." 05:50:43 CmdDone: OBSERVER.karen 243 jaeger configuration dither 0.1 05:57:09 CmdStarted: OBSERVER.karen 245 jaeger configuration reverse 05:57:22 CmdStarted: OBSERVER.karen 246 jaeger configuration dither 0.1 05:57:30 CmdStarted: jaeger.jaeger 85 jaeger configuration execute 05:57:43 OBSERVER.karen 246 jaeger : text="Dither configuration 379 loaded." 05:57:43 CmdDone: OBSERVER.karen 246 jaeger configuration dither 0.1 It was pretty clear up to now. From this point we're getting some faint clouds. 06:04:08 CmdStarted: OBSERVER.karen 248 jaeger configuration reverse 06:04:21 CmdStarted: OBSERVER.karen 249 jaeger configuration dither 0.1 06:04:30 CmdStarted: jaeger.jaeger 86 jaeger configuration execute 06:04:42 OBSERVER.karen 249 jaeger : text="Dither configuration 380 loaded." 06:04:42 CmdDone: OBSERVER.karen 249 jaeger configuration dither 0.1 06:11:06 CmdStarted: OBSERVER.karen 251 jaeger configuration reverse 06:11:18 CmdStarted: OBSERVER.karen 252 jaeger configuration dither 0.1 06:11:26 CmdStarted: jaeger.jaeger 87 jaeger configuration execute 06:11:39 OBSERVER.karen 252 jaeger : text="Dither configuration 381 loaded." 06:11:39 CmdDone: OBSERVER.karen 252 jaeger configuration dither 0.1 The sky has cleared again 06:18:02 CmdStarted: OBSERVER.karen 254 jaeger configuration reverse 06:18:15 CmdStarted: OBSERVER.karen 255 jaeger configuration dither 0.1 06:18:24 CmdStarted: jaeger.jaeger 88 jaeger configuration execute 06:18:36 OBSERVER.karen 255 jaeger : text="Dither configuration 382 loaded." 06:18:36 CmdDone: OBSERVER.karen 255 jaeger configuration dither 0.1 Some clouds came again 06:24:59 CmdStarted: OBSERVER.karen 257 jaeger configuration reverse 06:25:12 CmdStarted: OBSERVER.karen 258 jaeger configuration dither 0.1 06:25:33 OBSERVER.karen 258 jaeger : text="Dither configuration 383 loaded." 06:25:33 CmdDone: OBSERVER.karen 258 jaeger configuration dither 0.1 06:31:58 CmdStarted: OBSERVER.karen 260 jaeger configuration reverse The script is done 06:32:31 CmdStarted: OBSERVER.karen 261 cherno stop 06:32:48 CmdStarted: OBSERVER.karen 262 jaeger unwind Corresponding images: 05:30:11 boss exposure 300.0, file=335439 05:37:08 boss exposure 300.0, file=335440 05:44:04 boss exposure 300.0, file=335441 05:51:04 boss exposure 300.0, file=335442 05:58:04 boss exposure 300.0, file=335443 06:05:01 boss exposure 300.0, file=335444 06:11:58 boss exposure 300.0, file=335445 06:18:55 boss exposure 300.0, file=335446 06:25:52 boss exposure 300.0, file=335447 == Now doing APOGEE-loaded design 43 06:34:48 CmdStarted: OBSERVER.karen 263 jaeger configuration load 43 06:36:39 CmdStarted: OBSERVER.karen 264 jaeger configuration execute Focus sweep 06:37:59 CmdStarted: OBSERVER.karen 266 fliswarm talk -c gfa expose 15 Best focus = 252 06:44:59 CmdDone: OBSERVER.karen 282 tcc axis stop 06:45:20 CmdStarted: OBSERVER.karen 283 jaeger fvc loop 06:45:46 OBSERVER.karen 283 jaeger i text="[FVC]: RMS full fit 59.480 um." 06:46:27 OBSERVER.karen 283 jaeger i text="[FVC]: RMS full fit 20.650 um." 06:47:08 OBSERVER.karen 283 jaeger i text="[FVC]: RMS full fit 10.940 um." 06:47:51 OBSERVER.karen 283 jaeger i text="[FVC]: RMS full fit 8.850 um." 06:48:32 OBSERVER.karen 283 jaeger i text="[FVC]: RMS full fit 9.160 um." 06:48:37 OBSERVER.karen 283 jaeger w text="Delta RMS reached. Cancelling the FVC loop." 06:48:55 CmdStarted: OBSERVER.karen 284 jaeger configuration slew Opened APOGEE cold shitter: 06:49:16 CmdDone: OBSERVER.karen 285 apogee shutter open 06:49:48 CmdStarted: OBSERVER.karen 286 cherno offset 0 0 -432 06:49:59 CmdStarted: OBSERVER.karen 287 cherno acquire -c -t 15 06:54:29 OBSERVER.karen 287 cherno d guide_rms=316,0.051,0.06,0.079 Base Exposures ; the sky is clear now 06:54:55 apogee.expose=apRaw-40040010, 30, A, Object 07:06:16 CmdStarted: OBSERVER.karen 289 jaeger configuration dither 0.1 07:06:24 CmdStarted: jaeger.jaeger 101 jaeger configuration execute 07:06:37 OBSERVER.karen 289 jaeger : text="Dither configuration 385 loaded." 07:06:37 CmdDone: OBSERVER.karen 289 jaeger configuration dither 0.1 07:11:57 CmdStarted: OBSERVER.karen 291 jaeger configuration reverse 07:12:10 CmdStarted: OBSERVER.karen 292 jaeger configuration dither 0.1 07:12:18 CmdStarted: jaeger.jaeger 102 jaeger configuration execute 07:12:31 OBSERVER.karen 292 jaeger : text="Dither configuration 386 loaded." 07:12:31 CmdDone: OBSERVER.karen 292 jaeger configuration dither 0.1 07:17:51 CmdStarted: OBSERVER.karen 294 jaeger configuration reverse 07:18:04 CmdStarted: OBSERVER.karen 295 jaeger configuration dither 0.1 07:18:25 OBSERVER.karen 295 jaeger : text="Dither configuration 387 loaded." 07:18:25 CmdDone: OBSERVER.karen 295 jaeger configuration dither 0.1 More thin clouds, with the Moon at high ALT 07:23:58 CmdDone: OBSERVER.karen 297 jaeger configuration reverse 07:23:58 CmdStarted: OBSERVER.karen 298 jaeger configuration dither 0.1 07:24:18 CmdDone: jaeger.jaeger 104 jaeger configuration execute 07:24:18 OBSERVER.karen 298 jaeger : text="Dither configuration 388 loaded." 07:24:18 CmdDone: OBSERVER.karen 298 jaeger configuration dither 0.1 07:29:38 CmdStarted: OBSERVER.karen 300 jaeger configuration reverse 07:29:51 CmdStarted: OBSERVER.karen 301 jaeger configuration dither 0.1 07:30:12 CmdDone: jaeger.jaeger 105 jaeger configuration execute 07:30:12 OBSERVER.karen 301 jaeger : text="Dither configuration 389 loaded." 07:30:12 CmdDone: OBSERVER.karen 301 jaeger configuration dither 0.1 07:35:32 CmdStarted: OBSERVER.karen 303 jaeger configuration reverse 07:35:45 CmdStarted: OBSERVER.karen 304 jaeger configuration dither 0.1 07:36:07 CmdDone: jaeger.jaeger 106 jaeger configuration execute 07:36:07 OBSERVER.karen 304 jaeger : text="Dither configuration 390 loaded." 07:36:07 CmdDone: OBSERVER.karen 304 jaeger configuration dither 0.1 07:41:27 CmdStarted: OBSERVER.karen 306 jaeger configuration reverse 07:41:39 CmdStarted: OBSERVER.karen 307 jaeger configuration dither 0.1 07:42:00 CmdDone: jaeger.jaeger 107 jaeger configuration execute 07:42:00 OBSERVER.karen 307 jaeger : text="Dither configuration 391 loaded." 07:42:00 CmdDone: OBSERVER.karen 307 jaeger configuration dither 0.1 07:44:21 OBSERVER.karen 287 cherno w text="Camera gfa3: not enough sources." 07:44:21 OBSERVER.karen 287 cherno w text="Camera gfa6: not enough sources." 07:44:21 OBSERVER.karen 287 cherno w text="Camera gfa2: not enough sources." 07:44:21 OBSERVER.karen 287 cherno w text="Camera gfa5: not enough sources." 07:44:21 OBSERVER.karen 287 cherno w text="Camera gfa4: not enough sources." 07:44:21 OBSERVER.karen 287 cherno e acquisition_valid=F; correction_applied=F Stars disappeared because of clouds We increased the exposure time to 45 sec: cherno set exposure-time 45 07:47:20 CmdStarted: OBSERVER.karen 309 jaeger configuration reverse 07:47:34 CmdStarted: OBSERVER.karen 311 jaeger configuration dither 0.1 07:47:42 CmdStarted: jaeger.jaeger 108 jaeger configuration execute 07:47:55 OBSERVER.karen 311 jaeger : text="Dither configuration 392 loaded." 07:47:55 CmdDone: OBSERVER.karen 311 jaeger configuration dither 0.1 07:53:15 CmdStarted: OBSERVER.karen 313 jaeger configuration reverse 07:53:28 CmdStarted: OBSERVER.karen 314 jaeger configuration dither 0.1 07:53:36 CmdStarted: jaeger.jaeger 109 jaeger configuration execute 07:53:49 OBSERVER.karen 314 jaeger : text="Dither configuration 393 loaded." 07:53:49 CmdDone: OBSERVER.karen 314 jaeger configuration dither 0.1 07:56:23 CmdDone: OBSERVER.karen 316 cherno stop 07:59:21 CmdDone: OBSERVER.karen 317 jaeger configuration reverse 07:06:49 apogee.expose=apRaw-40040011, 30, A, Object 07:12:43 apogee.expose=apRaw-40040012, 30, A, Object 07:18:38 apogee.expose=apRaw-40040013, 30, A, Object 07:24:31 apogee.expose=apRaw-40040014, 30, A, Object 07:30:25 apogee.expose=apRaw-40040015, 30, A, Object 07:36:19 apogee.expose=apRaw-40040016, 30, A, Object 07:42:13 apogee.expose=apRaw-40040017, 30, A, Object 07:48:08 apogee.expose=apRaw-40040018, 30, A, Object 07:54:01 apogee.expose=apRaw-40040019, 30, A, Object We ran modified Jose's script for the dome flat. 08:01:23 CmdDone: OBSERVER.karen.apogee 42 apogee expose time=50.0 object=DomeFlat the command is nice - it closes petals on its own and then opens at the end. The FF lampl was turned on as expected during the 3-rd read, and then was off, and then... it did the same with the FF lamp during the 4-th read, again. Double exposed dome flats: 59566 20 08:01 40040020 DomeFlat 5 A 13.0 FPS- 0 a-b-c - 255 59566 21 08:02 40040021 DomeFlat 5 A 13.0 FPS- 0 a-b-c - 258 Next, we forgot to turn the FF lamp on manually 59566 22 08:05 40040022 DomeFlat 5 A 13.0 ------- x-x-x - 0 then shutter was closed... 59566 23 08:06 40040023 DomeFlat 5 A 13.0 FPS- 0 a-b-c - 0 We then took a regular dome flat, manually 59566 24 08:09 40040024 DomeFlat 5 A 13.0 FPS- 0 x-x-x - 287 09:23Z Resuming Jose's cloudy night test since it is cloudy. Telescope is still open since we do not have any immediate weather dangers. Dmitry will do this test to train with the command structure. Karen will log. FVC loop test while changing the -k and -m parameters. Loading an APOGEE configuration. 09:24:33 jaeger configuration load 61 09:27:09 jaeger configuration execute k = 0.8, m = 5 =============== 09:29:10 OBSERVER.dmbiz 19 jaeger i text="[FVC]: RMS full fit 69.830 um." 09:29:52 OBSERVER.dmbiz 19 jaeger i text="[FVC]: RMS full fit 30.370 um." 09:30:34 OBSERVER.dmbiz 19 jaeger i text="[FVC]: RMS full fit 27.850 um." 09:31:15 OBSERVER.dmbiz 19 jaeger i text="[FVC]: RMS full fit 27.240 um." fimg-fvc1n-0033.fits to fimg-fvc1n-0036.fits 09:31:58 jaeger configuration reverse 09:35:24 jaeger configuration execute k = 0.9 , m = 5 =============== 09:37:03 OBSERVER.dmbiz 22 jaeger i text="[FVC]: RMS full fit 55.210 um." 09:37:45 OBSERVER.dmbiz 22 jaeger i text="[FVC]: RMS full fit 9.050 um." 09:38:26 OBSERVER.dmbiz 22 jaeger i text="[FVC]: RMS full fit 4.740 um." fimg-fvc1n-0037.fits to fimg-fvc1n-0039.fits Received an error: 09:38:48 jaeger.jaeger 126 jaeger f error="Command 'ieb fbi led2 0' failed because of an uncaught error 'DriftError: Failed connecting to the IEB. Disabling it.'. See traceback in the log for more information." 09:38:48 CmdDone: jaeger.jaeger 126 jaeger ieb fbi led2 0 09:40:00 jaeger configuration reverse 09:40:15 jaeger configuration execute 09:41:55 jaeger.jaeger 131 jaeger f error="ieb not connected." 09:41:55 CmdDone: jaeger.jaeger 131 jaeger ieb fbi led1 0 09:41:55 .hub 0 cmds d CmdQueued=427058,1639820514.41,"jaeger.jaeger",132,"jaeger",183,"ieb fbi led2 0" 09:41:55 jaeger.jaeger 132 jaeger f error="ieb not connected." 09:41:55 CmdDone: jaeger.jaeger 132 jaeger ieb fbi led2 0 09:41:55 OBSERVER.dmbiz 25 jaeger w text="Cannot write processed image." 09:41:55 OBSERVER.dmbiz 25 jaeger f error="Command 'fvc loop -k 1.0 -m 5' failed because of an uncaught error 'ValueError: zero-size array to reduction operation maximum which has no identity'. See traceback in the log for more information." 09:41:55 CmdDone: OBSERVER.dmbiz 25 jaeger fvc loop -k 1.0 -m 5 10:05Z Jose joined us on STUI. We had to restart the jaeger actor completely and that restored communication to the ieb. We discovered that Grafana kept updating all the telemetry from FPS, so it was ok. Jose thinks that the drop in communication to the IEB was long enough that it ended up in a disabled/disconnected state. See Problems in Detail on what commands we issued to try to revive the IEB. Resuming the test: k = 1.0, m=5 ============= 10:08:33 OBSERVER.dmbiz 42 jaeger i text="[FVC]: RMS full fit 54.060 um." 10:09:18 OBSERVER.dmbiz 42 jaeger i text="[FVC]: RMS full fit 6.580 um." 10:09:59 OBSERVER.dmbiz 42 jaeger i text="[FVC]: RMS full fit 4.240 um." fimg-fvc1n-0042.fits to fimg-fvc1n-0044.fits 10:11:23 jaeger configuration reverse 10:12:53 jaeger configuration execute k=0.7, m=8 ========== 10:14:25 OBSERVER.dmbiz 45 jaeger i text="[FVC]: RMS full fit 53.620 um." 10:15:06 OBSERVER.dmbiz 45 jaeger i text="[FVC]: RMS full fit 18.490 um." 10:15:48 OBSERVER.dmbiz 45 jaeger i text="[FVC]: RMS full fit 7.740 um." 10:16:30 OBSERVER.dmbiz 45 jaeger i text="[FVC]: RMS full fit 5.020 um." 10:17:12 OBSERVER.dmbiz 45 jaeger i text="[FVC]: RMS full fit 4.360 um." fimg-fvc1n-0045.fits to fimg-fvc1n-0049.fits 10:17:21 jaeger configuration reverse 10:18:47 jaeger configuration execute k=0.9, m=8 ========== 10:20:15 OBSERVER.dmbiz 48 jaeger i text="[FVC]: RMS full fit 52.690 um." 10:20:57 OBSERVER.dmbiz 48 jaeger i text="[FVC]: RMS full fit 8.560 um." 10:21:39 OBSERVER.dmbiz 48 jaeger i text="[FVC]: RMS full fit 4.350 um." fimg-fvc1n-0050.fits to fimg-fvc1n-0052.fits 10:26Z Grafana has stopped updating information from the FPS. With the command jaeger status, we see everything is still ok. Grafana is just not getting any of the information and most of the plots are showing NaN or "No Data". We still see the time series line plots for temperatures, pressure, etc. messages. 10:28:04 jaeger configuration reverse 10:29:39 jaeger configuration execute k=1.0, m=8 =========== 10:31:06 OBSERVER.dmbiz 53 jaeger f error="Command 'fvc loop -k 1.0 -m 8' failed because of an uncaught error 'ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 17 is different from 18)'. See traceback in the log for more information." 10:31:06 CmdDone: OBSERVER.dmbiz 53 jaeger fvc loop -k 1.0 -m 8 Failed, so we will retry. This is similar to the previous hiccup in this test. 10:32:10 OBSERVER.dmbiz 54 jaeger i text="[FVC]: RMS full fit 51.540 um." 10:32:52 OBSERVER.dmbiz 54 jaeger i text="[FVC]: RMS full fit 6.110 um." Same error again: 10:33:46 OBSERVER.dmbiz 54 jaeger f error="Command 'fvc loop -k 1.0 -m 8' failed because of an uncaught error 'ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 17 is different from 18)'. See traceback in the log for more information." 10:33:46 CmdDone: OBSERVER.dmbiz 54 jaeger fvc loop -k 1.0 -m 8 Try again: 10:34:52 OBSERVER.dmbiz 55 jaeger i text="[FVC]: RMS full fit 3.820 um." fimg-fvc1n-0054.fits to fimg-fvc1n-0057.fits Took 3 iterations total, but we issued the fvc loop command 3 times. 10:36:14 jaeger configuration reverse 10:37:35 jaeger configuration execute k=0.7, m = 10 =============== 10:38:59 OBSERVER.dmbiz 58 jaeger i text="[FVC]: RMS full fit 51.050 um." 10:39:41 OBSERVER.dmbiz 58 jaeger i text="[FVC]: RMS full fit 17.150 um." 10:40:23 OBSERVER.dmbiz 58 jaeger i text="[FVC]: RMS full fit 7.400 um." 10:41:04 OBSERVER.dmbiz 58 jaeger i text="[FVC]: RMS full fit 4.700 um." fimg-fvc1n-0058.fits to fimg-fvc1n-0061.fits 10:41:16 jaeger configuration reverse 10:42:38 jaeger configuration execute k=0.8, m =10 ============ 10:43:52 OBSERVER.dmbiz 61 jaeger i text="[FVC]: RMS full fit 49.250 um." 10:44:35 OBSERVER.dmbiz 61 jaeger i text="[FVC]: RMS full fit 12.270 um." 10:45:17 OBSERVER.dmbiz 61 jaeger i text="[FVC]: RMS full fit 4.880 um." fimg-fvc1n-0062.fits to fimg-fvc1n-0064.fits 10:45:35 jaeger configuration reverse 10:46:56 jaeger configuration execute k=1.0, m=10 =========== 10:48:09 OBSERVER.dmbiz 64 jaeger i text="[FVC]: RMS full fit 47.640 um." 10:48:51 OBSERVER.dmbiz 64 jaeger i text="[FVC]: RMS full fit 6.040 um." 10:49:32 OBSERVER.dmbiz 64 jaeger i text="[FVC]: RMS full fit 4.330 um." fimg-fvc1n-0065.fits to fimg-fvc1n-0067.fits 10:49:47 jaeger configuration reverse 10:49:54 jaeger.jaeger 72 jaeger f error="Command 'ieb fbi led2 0' failed because of an uncaught error 'DriftError: Failed connecting to the IEB. Disabling it.'. See traceback in the log for more information." 10:49:54 CmdDone: jaeger.jaeger 72 jaeger ieb fbi led2 0 We've lost communication with ieb again. We checked the snapshot and all the robots are back in the home position. For sanity, we reloaded the configuration ID for this test: 10:59:11 jaeger configuration load 61 11:00:35 jaeger configuration execute k= 0.7, m=15 ============ 11:02:08 OBSERVER.dmbiz 71 jaeger i text="[FVC]: RMS full fit 54.870 um." 11:02:51 OBSERVER.dmbiz 71 jaeger i text="[FVC]: RMS full fit 19.590 um." 11:03:35 OBSERVER.dmbiz 71 jaeger i text="[FVC]: RMS full fit 8.880 um." 11:04:16 OBSERVER.dmbiz 71 jaeger i text="[FVC]: RMS full fit 5.190 um." 11:04:57 OBSERVER.dmbiz 71 jaeger i text="[FVC]: RMS full fit 4.680 um." fimg-fvc1n-0068.fits to fimg-fvc1n-0072.fits 11:05:13 jaeger configuration reverse 11:05:18 CmdDone: jaeger.jaeger 11 jaeger ieb fbi led2 0 11:05:33 CmdDone: OBSERVER.dmbiz 73 jaeger ieb status The ieb has disconnected again! This is becoming more frequent! 11:07Z After checking the snapshot (all robots are home), we restarted the jaeger actor again. Starting again because we have 2 more positions to go. 11:07:50 jaeger configuration load 61 11:09:31 jaeger configuration execute k =0.8, m=15 ============ 11:11:23 OBSERVER.dmbiz 77 jaeger i text="[FVC]: RMS full fit 55.440 um." 11:12:04 OBSERVER.dmbiz 77 jaeger i text="[FVC]: RMS full fit 13.720 um." 11:12:47 OBSERVER.dmbiz 77 jaeger i text="[FVC]: RMS full fit 5.680 um." 11:13:31 OBSERVER.dmbiz 77 jaeger i text="[FVC]: RMS full fit 4.270 um." fimg-fvc1n-0073.fits to fimg-fvc1n-0076.fits 11:13:41 jaeger configuration reverse 11:15:03 jaeger configuration execute k = 0.9, m=15 ============= 11:16:24 OBSERVER.dmbiz 80 jaeger i text="[FVC]: RMS full fit 53.980 um." 11:17:09 OBSERVER.dmbiz 80 jaeger i text="[FVC]: RMS full fit 8.840 um." 11:17:52 OBSERVER.dmbiz 80 jaeger i text="[FVC]: RMS full fit 4.270 um." fimg-fvc1n-0077.fits to fimg-fvc1n-0079.fits 11:18:23 jaeger configuration reverse For sanity, and it is the end of the night for us. 11:19:50 CmdStarted: OBSERVER.dmbiz 82 jaeger unwind 11:20:47 CmdDone: OBSERVER.dmbiz 83 jaeger power off We're getting an error with jaeger status: 11:21:34 OBSERVER.dmbiz 86 jaeger f error="Command 'status' failed because of an uncaught error 'KeyError: 3'. See traceback in the log for more information." 11:21:34 CmdDone: OBSERVER.dmbiz 86 jaeger status With jaeger ieb status, we see this output: 11:21:22 OBSERVER.dmbiz 85 jaeger : fbi_led=0.0,0.0,0.0,0.0; flow=5.065,1.21; humidity=22.852,50.685; power_can=T,T,T,T,T,T; power_gfa=F,F,F,F,F,F; power_nuc=T,T,T,T,T,T; power_sextant=F,F,F,F,F,F; power_sync=F; pressure=41.039,23.941; temperature=6.305,2.154,12.25,13.22,13.22,4.22,3.02,3.62,22.42,4.002,1.92,2.71,10.069; voltage=0.063,0.055,0.055,0.055,0.055,0.055 Without confirmation through Grafana, we are relying on this output to make sure the GFAs and the FPS are powered down and off. 11:33Z the sky is completely overcast. We are pretty blind without Grafana, so we decided to close. Calibrations started. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0040 0.0040 Axial B: 0.0020 0.0030 Axial C: -0.1150 -0.1150 Trans D: -0.0341 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 3.72 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6550 -9.6570 Axial B: -9.6680 -9.6710 Axial C: -9.6960 -9.6970 Trans D: 0.1180 0.1140 GALILS Commanded: 4892250 4890800 4906950 0 -3400 Actual: 4859192 4859211 4859230 0 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.99 -0.02 -0.0 -0.0 41.65 ---- Telescope Status ---- 23:52Z Started ventilation 00:34Z encl25m : open 11:33Z encl25m : closed No interlocks bypassed Status at: 11:53Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = FPS Counterweights at: 685/704 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 217 lbs, 22 psi ---- Software Used ---- Get Versions, 12/17/21, 23:25:03 -- STUI software: STUI : 1.7.13 alerts : 2.1.6 apo : 3.0.1 boss : 6.0.1-alpha.0 -boss.specMec : 3.0.4 guider : 4.0.0 ecamera : FAILED fliswarm : FAILED hartmann : 2.0.1-alpha.0 hub : 5.0.0a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 sop : FAILED tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : nosop -- done -- /dev/sdb1 2.0T 239G 1.8T 12% /home 10.25.1.230:/data 21T 17T 4.1T 80% /data ---- APOGEE Data Summary ---- Last morning: 59565 45 12:49 40030045 Dark 60 A 13.0 ------- a-b-c - - 59565 46 12:59 40030046 Dark 60 A 13.0 ------- a-b-c - - 59565 47 13:10 40030047 Dark 60 A 13.0 ------- a-b-c - - 59565 48 13:12 40030048 QuarFlat 10 A 13.0 FPS- 0 a-b-c - 101 59565 49 13:14 40030049 QuarFlat 10 A 13.0 FPS- 0 a-b-c - 102 59565 50 13:15 40030050 QuarFlat 10 A 13.0 FPS- 0 a-b-c - 102 59565 51 13:18 40030051 Arc-Thar 12 A 13.0 FPS- 0 a-b-c -0.07 - 59565 52 13:25 40030052 Arc-Une 40 A 13.0 FPS- 0 a-b-c - - 59565 53 13:27 40030053 Arc-Thar 12 B 13.5 FPS- 0 a-b-c 0.52 - 59565 54 13:34 40030054 Arc-Une 40 B 13.5 FPS- 0 a-b-c - - 59565 55 13:40 40030055 Dark 30 A 13.0 ------- a-b-c - - 59565 56 13:45 40030056 InteFlat 30 A 13.0 ------- a-b-c - 279 59565 57 13:51 40030057 InteFlat 30 A 13.0 ------- a-b-c - 279 59565 58 13:56 40030058 InteFlat 30 A 13.0 ------- a-b-c - 279 59565 59 14:01 40030059 Dark 30 A 13.0 ------- a-b-c - - This evening: (observers39) [observer@sdss5-hub: ~]$ list_ap APOGEE data list, mjd=59566, all raw_data: /data/apogee/utr_cdr/59566/apRaw-*.fits archive: /data/apogee/archive/59566/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59566/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59566 1 23:25 40040001 Dark 3 A 13.0 ------- a-b-c - - 59566 2 23:36 40040002 Dark 60 A 13.0 ------- a-b-c - - 59566 3 23:38 40040003 QuarFlat 10 A 13.0 FPS- 0 a-b-c - 102 59566 4 23:40 40040004 Arc-Thar 12 A 13.0 FPS- 0 a-b-c -0.10 - 59566 5 23:47 40040005 Arc-Une 40 A 13.0 FPS- 0 a-b-c - - 59566 6 23:49 40040006 Arc-Thar 12 B 13.5 FPS- 0 a-b-c 0.52 - 59566 7 23:57 40040007 Arc-Une 40 B 13.5 FPS- 0 a-b-c - - 59566 8 00:08 40040008 Dark 60 A 13.0 ------- a-b-c - - 59566 9 00:14 40040009 Arc---- 20 A 13.0 FPS- 0 a-b-c - - FPI ------------------------------------------------------------------------------ APOGEE data list, mjd=59566, all raw_data: /data/apogee/utr_cdr/59566/apRaw-*.fits archive: /data/apogee/archive/59566/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59566/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59566 1 23:25 40040001 Dark 3 A 13.0 ------- a-b-c - - 59566 2 23:36 40040002 Dark 60 A 13.0 ------- a-b-c - - 59566 3 23:38 40040003 QuarFlat 10 A 13.0 FPS- 0 a-b-c - 102 59566 4 23:40 40040004 Arc-Thar 12 A 13.0 FPS- 0 a-b-c -0.10 - 59566 5 23:47 40040005 Arc-Une 40 A 13.0 FPS- 0 a-b-c - - 59566 6 23:49 40040006 Arc-Thar 12 B 13.5 FPS- 0 a-b-c 0.52 - 59566 7 23:57 40040007 Arc-Une 40 B 13.5 FPS- 0 a-b-c - - 59566 8 00:08 40040008 Dark 60 A 13.0 ------- a-b-c - - 59566 9 00:14 40040009 Arc---- 20 A 13.0 FPS- 0 a-b-c - - FPI 59566 10 06:59 40040010 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59566 11 07:11 40040011 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59566 12 07:17 40040012 Object 30 A 13.0 FPS- 0 a-b-c -0.09 - 59566 13 07:23 40040013 Object 30 A 13.0 FPS- 0 a-b-c -0.09 - 59566 14 07:29 40040014 Object 30 A 13.0 FPS- 0 a-b-c -0.09 - 59566 15 07:35 40040015 Object 30 A 13.0 FPS- 0 a-b-c -0.09 - 59566 16 07:41 40040016 Object 30 A 13.0 FPS- 0 a-b-c -0.09 - 59566 17 07:47 40040017 Object 30 A 13.0 FPS- 0 a-b-c -0.09 - 59566 18 07:53 40040018 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59566 19 07:58 40040019 Object 30 A 13.0 FPS- 0 a-b-c -0.10 - 59566 20 08:01 40040020 DomeFlat 5 A 13.0 FPS- 0 a-b-c - 255 59566 21 08:02 40040021 DomeFlat 5 A 13.0 FPS- 0 a-b-c - 258 59566 22 08:05 40040022 DomeFlat 5 A 13.0 FPS- 0 a-b-c - 0 59566 23 08:06 40040023 DomeFlat 5 A 13.0 FPS- 0 a-b-c - 0 59566 24 08:09 40040024 DomeFlat 5 A 13.0 FPS- 0 a-b-c - 287 59566 25 11:45 40040025 Dark 60 A 13.0 ------- a-b-c - - 59566 26 11:52 40040026 Dark 60 A 13.0 ------- x-x-x - - ------------------------------------------------------------------------------ ---- eBOSS Data Summary ---- Reading FITS from /data/spectro//59566 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename crt plt flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 23:39 b1-00335406 FPS-N None bia Clo 0.0 -5087 -5088 -5103 2 23:39 r1-00335406 FPS-N None bia Clo 0.0 -5087 -5088 -5103 3 23:42 b1-00335407 FPS-N None dar Clo 20.0 -5087 -5088 -5103 4 23:42 r1-00335407 FPS-N None dar Clo 20.0 -5087 -5088 -5103 5 23:44 b1-00335408 FPS-N None arc Lef 4.1 -5087 -5088 -5103 HART 6 23:44 r1-00335408 FPS-N None arc Lef 4.1 -5087 -5088 -5103 HART 7 23:44 r1-00335409 FPS-N None arc Rig 4.1 -5087 -5088 -5103 HART 8 23:44 b1-00335409 FPS-N None arc Rig 4.1 -5087 -5088 -5103 HART 9 23:45 r1-00335410 FPS-N None arc Lef 4.1 -5087 -5088 -5103 HART 10 23:45 b1-00335410 FPS-N None arc Lef 4.1 -5087 -5088 -5103 HART 11 23:46 b1-00335411 FPS-N None arc Rig 4.1 -5087 -5088 -5103 HART 12 23:46 r1-00335411 FPS-N None arc Rig 4.1 -5087 -5088 -5103 HART 13 23:49 b1-00335412 FPS-N None arc Out 4.1 -4742 -4742 -4758 14 23:49 r1-00335412 FPS-N None arc Out 4.1 -4742 -4742 -4758 15 23:51 r1-00335413 FPS-N None fla Out 30.1 -4742 -4742 -4758 16 23:51 b1-00335413 FPS-N None fla Out 30.1 -4742 -4742 -4758 17 02:17 b1-00335414 FPS-N None arc Lef 4.1 -4742 -4742 -4758 HART 18 02:17 r1-00335414 FPS-N None arc Lef 4.1 -4742 -4742 -4758 HART 19 02:17 r1-00335415 FPS-N None arc Rig 4.1 -4742 -4742 -4758 HART 20 02:17 b1-00335415 FPS-N None arc Rig 4.1 -4742 -4742 -4758 HART 21 02:19 b1-00335416 FPS-N None arc Lef 4.1 -4742 -4742 -4758 HART 22 02:19 r1-00335416 FPS-N None arc Lef 4.1 -4742 -4742 -4758 HART 23 02:19 b1-00335417 FPS-N None arc Rig 4.1 -4742 -4742 -4758 HART 24 02:19 r1-00335417 FPS-N None arc Rig 4.1 -4742 -4742 -4758 HART 25 02:21 r1-00335418 FPS-N None arc Lef 4.1 -2745 -2740 -2759 HART 26 02:21 b1-00335418 FPS-N None arc Lef 4.1 -2745 -2740 -2759 HART 27 02:22 b1-00335419 FPS-N None arc Rig 4.1 -2745 -2740 -2759 HART 28 02:22 r1-00335419 FPS-N None arc Rig 4.1 -2745 -2740 -2759 HART 29 02:23 b1-00335420 FPS-N None arc Out 4.1 -3025 -3025 -3043 30 02:23 r1-00335420 FPS-N None arc Out 4.1 -3025 -3025 -3043 31 02:25 r1-00335421 FPS-N None fla Out 30.1 -3025 -3025 -3043 32 02:25 b1-00335421 FPS-N None fla Out 30.1 -3025 -3025 -3043 33 02:42 r1-00335422 FPS-N None sci Out 300.1 -3025 -3025 -3043 34 02:42 b1-00335422 FPS-N None sci Out 300.1 -3025 -3025 -3043 35 02:49 r1-00335423 FPS-N None sci Out 300.1 -3025 -3025 -3043 36 02:49 b1-00335423 FPS-N None sci Out 300.1 -3025 -3025 -3043 37 02:56 b1-00335424 FPS-N None sci Out 300.1 -3025 -3025 -3043 38 02:56 r1-00335424 FPS-N None sci Out 300.1 -3025 -3025 -3043 39 03:03 b1-00335425 FPS-N None sci Out 300.1 -3025 -3025 -3043 40 03:03 r1-00335425 FPS-N None sci Out 300.1 -3025 -3025 -3043 41 03:10 b1-00335426 FPS-N None sci Out 300.1 -3025 -3025 -3043 42 03:10 r1-00335426 FPS-N None sci Out 300.1 -3025 -3025 -3043 43 03:17 r1-00335427 FPS-N None sci Out 300.1 -3025 -3025 -3043 44 03:17 b1-00335427 FPS-N None sci Out 300.1 -3025 -3025 -3043 45 03:24 r1-00335428 FPS-N None sci Out 300.1 -3025 -3025 -3043 46 03:24 b1-00335428 FPS-N None sci Out 300.1 -3025 -3025 -3043 47 03:31 r1-00335429 FPS-N None sci Out 300.1 -3025 -3025 -3043 48 03:31 b1-00335429 FPS-N None sci Out 300.1 -3025 -3025 -3043 49 03:38 b1-00335430 FPS-N None sci Out 300.1 -3025 -3025 -3043 50 03:38 r1-00335430 FPS-N None sci Out 300.1 -3025 -3025 -3043 51 03:45 r1-00335431 FPS-N None sci Out 300.1 -3025 -3025 -3043 52 03:45 b1-00335431 FPS-N None sci Out 300.1 -3025 -3025 -3043 53 05:01 r1-00335432 FPS-N None arc Lef 4.1 -3025 -3025 -3043 HART 54 05:01 b1-00335432 FPS-N None arc Lef 4.1 -3025 -3025 -3043 HART 55 05:02 b1-00335433 FPS-N None arc Rig 4.1 -3025 -3025 -3043 HART 56 05:02 r1-00335433 FPS-N None arc Rig 4.1 -3025 -3025 -3043 HART 57 05:07 r1-00335434 FPS-N None arc Lef 4.1 -4602 -4603 -4620 HART 58 05:07 b1-00335434 FPS-N None arc Lef 4.1 -4602 -4603 -4620 HART 59 05:08 r1-00335435 FPS-N None arc Rig 4.1 -4602 -4603 -4620 HART 60 05:08 b1-00335435 FPS-N None arc Rig 4.1 -4602 -4603 -4620 HART 61 05:09 b1-00335436 FPS-N None arc Out 4.1 -3750 -3749 -3764 62 05:09 r1-00335436 FPS-N None arc Out 4.1 -3750 -3749 -3764 63 05:11 b1-00335437 FPS-N None fla Out 30.1 -3750 -3749 -3764 64 05:11 r1-00335437 FPS-N None fla Out 30.1 -3750 -3749 -3764 65 05:23 r1-00335438 FPS-N None sci Out 300.1 -3750 -3749 -3764 66 05:23 b1-00335438 FPS-N None sci Out 300.1 -3750 -3749 -3764 67 05:30 b1-00335439 FPS-N None sci Out 300.1 -3750 -3749 -3764 68 05:30 r1-00335439 FPS-N None sci Out 300.1 -3750 -3749 -3764 69 05:37 b1-00335440 FPS-N None sci Out 300.1 -3750 -3749 -3764 70 05:37 r1-00335440 FPS-N None sci Out 300.1 -3750 -3749 -3764 71 05:44 b1-00335441 FPS-N None sci Out 300.1 -3750 -3749 -3764 72 05:44 r1-00335441 FPS-N None sci Out 300.1 -3750 -3749 -3764 73 05:51 r1-00335442 FPS-N None sci Out 300.1 -3750 -3749 -3764 74 05:51 b1-00335442 FPS-N None sci Out 300.1 -3750 -3749 -3764 75 05:58 r1-00335443 FPS-N None sci Out 300.1 -3750 -3749 -3764 76 05:58 b1-00335443 FPS-N None sci Out 300.1 -3750 -3749 -3764 77 06:05 b1-00335444 FPS-N None sci Out 300.1 -3750 -3749 -3764 78 06:05 r1-00335444 FPS-N None sci Out 300.1 -3750 -3749 -3764 79 06:11 b1-00335445 FPS-N None sci Out 300.1 -3750 -3749 -3764 80 06:11 r1-00335445 FPS-N None sci Out 300.1 -3750 -3749 -3764 81 06:18 r1-00335446 FPS-N None sci Out 300.1 -3750 -3749 -3764 82 06:18 b1-00335446 FPS-N None sci Out 300.1 -3750 -3749 -3764 83 06:25 r1-00335447 FPS-N None sci Out 300.1 -3750 -3749 -3764 84 06:25 b1-00335447 FPS-N None sci Out 300.1 -3750 -3749 -3764 85 11:35 b1-00335448 FPS-N None bia Clo 0.0 -3750 -3749 -3764 86 11:35 r1-00335448 FPS-N None bia Clo 0.0 -3750 -3749 -3764 87 11:36 b1-00335449 FPS-N None dar Clo 300.0 -3750 -3749 -3764 88 11:36 r1-00335449 FPS-N None dar Clo 300.0 -3750 -3749 -3764 ---- Weather Log ---- Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma 19:00 5.3 -7.3 14.2 262.1 179 19:15 4.0 -6.8 15.3 233.8 211 19:30 4.4 -8.1 12.8 264.5 171 19:45 4.7 -8.1 16.9 247.9 233 20:00 4.8 -7.6 16.4 241.5 176 20:15 3.5 -8.2 15.6 264.0 229 20:30 2.9 -7.7 10.5 223.8 233 20:45 4.6 -6.8 12.5 255.3 352 21:00 4.5 -6.7 19.2 260.7 354 21:15 3.8 -6.5 14.2 262.4 447 21:30 4.4 -6.1 15.4 255.6 453 21:45 4.6 -6.5 17.0 266.7 448 22:00 4.7 -6.6 16.9 265.5 370 22:15 4.9 -6.6 19.6 265.0 414 22:30 4.9 -6.3 16.1 235.1 388 22:45 4.6 -6.6 12.5 260.1 403 23:00 3.9 -6.5 14.2 270.2 553 23:15 3.9 -6.3 14.5 254.8 531 23:30 3.5 -6.3 17.2 262.2 555 23:45 3.2 -6.3 13.9 258.3 543 00:00 2.6 -6.3 18.8 253.9 501 00:15 2.1 -6.4 14.7 242.0 460 00:30 1.9 -6.3 15.6 257.0 459 85 9.70 00:45 1.8 -6.4 13.0 250.8 440 232 16.88 01:00 1.7 -6.4 18.1 246.8 489 181 11.69 01:15 1.8 -6.4 13.1 264.6 545 240 22.97 01:30 1.6 -6.6 14.7 232.0 511 98 19.49 01:45 1.8 -6.6 13.6 240.7 514 118 30.48 02:00 1.8 -6.6 12.1 251.7 520 97 27.33 02:15 1.8 -6.7 18.3 235.7 466 251 47.03 02:30 1.7 -6.9 17.9 240.1 392 175 31.57 02:45 1.4 -7.1 11.9 242.6 360 222 23.32 03:00 1.4 -6.9 13.1 261.0 360 172 31.10 03:15 1.7 -7.2 12.3 258.4 353 191 46.78 03:30 1.7 -7.3 15.9 254.9 331 100 19.51 03:45 1.5 -7.4 15.3 262.1 290 142 31.88 04:00 1.5 -7.4 12.8 262.8 328 151 29.20 04:15 1.2 -7.6 11.2 263.9 284 161 29.10 04:30 1.1 -7.5 11.0 257.5 306 148 13.63 04:45 0.7 -7.4 9.9 262.6 338 237 19.47 05:00 0.9 -7.7 11.4 260.5 322 167 10.59 05:15 0.9 -7.8 12.8 249.1 283 132 10.66 05:30 0.8 -7.9 9.9 259.1 283 104 7.11 05:45 0.5 -8.4 7.1 261.3 263 115 8.36 06:00 1.2 -9.0 11.2 269.1 270 86 16.51 06:15 1.1 -9.5 11.3 265.7 229 85 9.21 06:30 1.2 -9.6 10.5 264.6 206 117 21.74 06:45 1.4 -9.2 11.7 261.8 215 88 20.59 07:00 0.8 -9.3 11.8 261.6 248 167 15.51 07:15 1.4 -9.2 9.9 260.4 270 177 37.35 07:30 1.4 -9.2 11.4 243.3 307 142 44.29 07:45 0.9 -9.1 12.0 260.7 312 201 26.19 08:00 1.1 -8.9 11.0 265.0 308 90 18.58 08:15 1.2 -8.9 6.4 267.3 330 149 16.71 08:30 1.1 -8.9 8.0 261.8 311 88 17.24 08:45 1.1 -8.1 7.8 260.6 340 92 23.64 09:00 1.3 -7.9 3.3 253.7 340 255 31.57 09:15 1.3 -7.7 6.3 262.5 384 201 32.09 09:30 1.2 -7.4 8.4 265.3 422 160 28.39 09:45 1.1 -7.4 7.2 260.0 413 203 25.15 10:00 1.1 -7.4 11.6 267.6 458 143 16.40 10:15 1.1 -7.2 5.4 258.6 442 231 28.22 10:30 1.2 -6.8 6.1 244.3 454 93 16.75 10:45 1.2 -6.8 7.4 263.3 481 191 11.16 11:00 0.8 -6.8 6.6 255.5 451 209 19.63 11:15 0.9 -6.9 6.9 229.4 486 94 12.66 11:30 0.8 -6.9 6.9 217.5 497 92 15.60 11:45 0.8 -7.1 5.1 235.8 524 141 9.82 ---- Problems in Detail ---- Recovering from IEB lost communication and disabled. We checked New Information for Commissioning wiki page for possible jaeger commands to use: We tried: jaeger ieb --help 09:43:16 .jaeger 0 jaeger e text="Failed setting chiller temperature." 09:43:38 OBSERVER.karen 335 jaeger f error="ieb not connected." 09:43:38 CmdDone: OBSERVER.karen 335 jaeger ieb info We issued jaeger ieb enable, but this did not reestablish connection to the ieb. We sent an email to Jose in the hopes he would be still working from Sunspot. 09:51:27 CmdDone: OBSERVER.dmbiz 32 jaeger ieb status 09:51:36 OBSERVER.dmbiz 33 jaeger e error="ieb not connected." 09:52:16 CmdDone: OBSERVER.dmbiz 34 jaeger power off This command apparently did not turn anything off in the FPS. Jose logged on STUI at 9:57Z and suggested we restart the jaeger actor. We did this and we were able to re-establish connection to the IEB. Every thing looked ok on Grafana, and the jaeger status command returned values that were ok. We noticed around 10:15Z, Grafana lost telemetry information and we see "No data" and NaNs. The time-series line plots were still active. jaeger status shows all the CANs and NUCs are powered on and ok. We decided at this point to try to continue with the FVC test since we have a few more to go to finish it. We had another disconnect with the IEB at 10:49Z. jaeger status yielded: 10:53:32 OBSERVER.dmbiz 66 jaeger e error="ieb not connected." We are restaring the jaeger actor again. jaeger status shows everything is still on with the FPS and the IEB is talking. We think this is related to or is similar to the communication problems we had from the previous nights with STUI disconnecting with hub. Although it is hard for the IEB to lose communication, according to Jose, we have experienced 2 instances of this now and it required a restart of the jaeger actor.