Subject: 2.5m obslog 20220919 (SJD 59842) From: kgrabowski@apo.nmsu.edu Date: Tue, 20 Sep 2022 11:01:57 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday September 19, 2022 (SJD 59842) ---- Observing Team ---- Katie Grabowski Dmitry Bizyaev ---- Observing Plan ---- Mon 19 Sep 2022 59842 0.28 01:57 02:11 11:42 19:04 04:36 --- dark ( 9.5 h) plus 0.3 h for 12 deg design ---- Observing Summary ---- Night lost due to humidity and clouds. ---- Observing Log ---- Afternoon: ========== All systems alive and responsive. Started APOGEE and BOSS cals. ---------- motorPosition sp1: 3361, 3514, 2145; ---------- Median spectrographs temperatures: Tsp1= 10.4 Outside air temperature= 16.4; ---------- jaeger power on --gfas 23:13:00 .jaeger 0 jaeger f text="[GET_FIRMWARE_VERSION, [56], 50170]: received a reply from 56 but the command has already timed out." This happened for about 150 robots. 23:13:05 OBSERVER.kgrabow 48 jaeger f error="Command 'power on --gfas' failed because of an uncaught error 'JaegerError: Some positioners failed to initialise: Positioner 56: failed retrieving firmware version.'. See traceback in the log for more information." When I turned it on again, they were fine and I got an FVC loop --no-apply, so I turned it off because the weather was too humid to open. Config Loaded: -999--999-6853 23:31:57 OBSERVER.kgrabow 56 jaeger i text="FVC iteration 1" 23:31:57 OBSERVER.kgrabow 56 jaeger d text="Taking exposure with fliswarm." 23:31:57 OBSERVER.kgrabow 56 jaeger e error="Failed processing image: The exposure succeeded but did not output the filename." 23:31:57 OBSERVER.kgrabow 56 jaeger w text="Cannot write processed image." 23:31:57 OBSERVER.kgrabow 56 jaeger d text="Turning LEDs off." 23:31:57 CmdStarted: jaeger.jaeger 497 jaeger ieb fbi led1 led2 0 23:31:57 jaeger.jaeger 497 jaeger > 23:31:57 jaeger.jaeger 497 jaeger : fbi_led=0.0,0.0,0.0,0.0 23:31:57 CmdDone: jaeger.jaeger 497 jaeger ieb fbi led1 led2 0 23:31:57 OBSERVER.kgrabow 56 jaeger f error="The FVC loop failed." 23:31:57 CmdDone: OBSERVER.kgrabow 56 jaeger fvc loop --no-apply Repeated power on sequence, error didn't return. 23:34:46 OBSERVER.kgrabow 58 jaeger e error="Failed processing image: The exposure succeeded but did not output the filename." 23:34:46 OBSERVER.kgrabow 58 jaeger w text="Cannot write processed image." Had to set GFA temperature again. The first time, it only sent GFA 3 to -20C. 23:50:50 OBSERVER.kgrabow 65 jaeger d fvc_filename=/data/fcam/59842/fimg-fvc1n-0001.fits 23:50:57 OBSERVER.kgrabow 65 jaeger i fvc_rms=52.14 23:50:57 OBSERVER.kgrabow 65 jaeger i fvc_perc_90=69.8 23:50:57 OBSERVER.kgrabow 65 jaeger i fvc_percent_reached=1.2 No light seen in FVC loop or BOSS image. Delayed ventilation due to humidity, thick clouds, and radar. 02:15Z- Started ventilation. No radar, sky clearing, dewpoint difference improving but not stable. Upon powering it on again, I got the different robots unable to get the firmware version, but also this error, possibly for a different group. 02:35:59 .jaeger 0 jaeger f text="[GET_STATUS, [758], 60282]: received a reply from 758 but the command has already timed out." Trying to power it on again, there were about 10 robots that didn't reply fast enough for [GET_STATUS]. Same results powering on and off again, different positioner numbers through. Maybe only half the original group. Plus this error: 02:39:42 .jaeger 0 jaeger w text="Cannot read device 'RTD2'. Low-temperature tracking temporarily disabled: Failed connecting to the IEB." 02:39:42 .jaeger 0 jaeger i fps_status=0x81 02:39:42 OBSERVER.kgrabow 75 jaeger f error="Command 'power on --gfas' failed because of an uncaught error 'JaegerError: Some positioners failed to initialise: Positioner 56: failed retrieving firmware version.'. See traceback in the log for more information." 02:39:42 CmdDone: OBSERVER.kgrabow 75 jaeger power on --gfas Powering on a few different times, same results of firmware errors, with some different positioner numbers. Tried jaeger reload, new error: 02:49:52 .jaeger 0 jaeger f text="[SET_SPEED, [1331], 68311]: received a reply from 1331 but the command has already timed out." 02:49:53 .jaeger 0 jaeger w text="Failed running alerts coroutine _check_robots: Failed connecting to the IEB. Disabling it." 02:49:53 .jaeger 0 jaeger w text="Failed running alerts coroutine _check_ieb: IEB is disabled." 02:49:53 .jaeger 0 jaeger w text="Failed running alerts coroutine _check_flow: IEB is disabled." 02:49:53 .jaeger 0 jaeger w text="Failed running alerts coroutine _check_outside_temperature: IEB is disabled." 02:50:03 .jaeger 0 jaeger w text="Failed running alerts coroutine _check_chiller: IEB is disabled." IEB status returned ok, jaeger status returned ok for 500 positioners, but then "ieb not connected" tried IEB enable, jaeger status ok and managed to take an FVC loop at the folded configuration. Right now the weather took a turn for the worse, but we wondered what the best protocol for this is, because turning it on and off doesn't always make it go away. 15 02:38 r1-00346462 -999--999 6853 arc Lef 4.1 3361 3514 2145 HART 16 02:38 b1-00346462 -999--999 6853 arc Lef 4.1 3361 3514 2145 HART 17 02:39 r1-00346463 -999--999 6853 arc Rig 4.1 3361 3514 2145 HART 18 02:39 b1-00346463 -999--999 6853 arc Rig 4.1 3361 3514 2145 HART Hartmanns ok 02:40:00 9.7 1319 -9.3 2137 -2.6 19 02:40 r1-00346464 -999--999 6853 fla Out 25.1 5497 5649 4281 20 02:40 b1-00346464 -999--999 6853 fla Out 25.1 5497 5649 4281 21 02:42 r1-00346465 -999--999 6853 fla Out 25.1 5497 5649 4281 22 02:42 b1-00346465 -999--999 6853 fla Out 25.1 5497 5649 4281 23 02:44 b1-00346466 -999--999 6853 fla Out 25.1 5497 5649 4281 24 02:44 r1-00346466 -999--999 6853 fla Out 25.1 5497 5649 4281 Took BOSS flats 03:11Z- Dewpoint difference trending back down towards 3. Stopped ventilating 22:58:19 apogee exposure file= apRaw-42800001 3 A Dark 23:00:40 apogee exposure file= apRaw-42800002 60 A Dark 23:02:40 boss exposure 0 sec file=0346455 bias 23:04:20 boss exposure 20 sec file=0346456 dark 23:05:20 boss exposure 4 sec file=0346457 arc 23:06:04 boss exposure 4 sec file=0346458 arc 23:07:44 boss exposure 4 sec file=0346459 arc 23:09:38 boss exposure 25 sec file=0346460 flat 23:11:32 apogee exposure file= apRaw-42800003 10 A QuartzFlat 23:13:27 apogee exposure file= apRaw-42800004 12 B ArcLamp 23:15:40 apogee exposure file= apRaw-42800005 40 B ArcLamp 23:22:57 apogee exposure file= apRaw-42800006 12 A ArcLamp 23:25:10 apogee exposure file= apRaw-42800007 40 A ArcLamp ========================================== 23:31:15 Config Loaded: -999--999-6853 ========================================== 23:32:43 apogee exposure file= apRaw-42800008 60 A Dark 23:43:35 apogee exposure file= apRaw-42800009 30 A ArcLamp 23:51:01 FVC exposure /data/fcam/59842/proc-fimg-fvc1n-0001.fits 23:50:57 fvc_perc_90= 69.8 23:58:40 boss exposure 300 sec file=0346461 science 02:39:10 boss exposure 4 sec file=0346462 arc 02:39:54 boss exposure 4 sec file=0346463 arc 02:42:20 boss exposure 25 sec file=0346464 flat 02:44:08 boss exposure 25 sec file=0346465 flat 02:45:53 boss exposure 25 sec file=0346466 flat ========================================== 02:52:25 Config Loaded: -999--999-6856 ========================================== 02:55:47 FVC exposure /data/fcam/59842/proc-fimg-fvc1n-0003.fits 02:55:42 fvc_perc_90= 69.7 No improvement to humidity. 10:56Z- Started APOGEE cals. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0050 0.0060 Axial B: -0.0030 -0.0030 Axial C: 0.0010 0.0000 Trans D: 0.0130 0.0180 Lateral E: 0.3315 0.3315 Lateral F: 0.5345 0.5345 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 15.89 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -0.0470 -0.0490 Axial B: -0.0410 -0.0430 Axial C: -0.0660 -0.0670 Trans D: -0.2110 -0.2460 GALILS Commanded: 4909500 4883500 4981300 0 -3400 Actual: 4859085 4859211 4859356 0 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3857.01 -0.06 0.05 0.04 41.68 ---- Telescope Status ---- No interlocks bypassed Status at: 11:00Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 695/713 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 262 lbs, 18.7 psi ---- Software Used ---- -- STUI software: STUI : 1.7.17b1 alerts : 2.2.2 boss : 6.0.1 -boss.specMec : 3.0.4 apo : 3.0.1 fliswarm : 0.3.1 jaeger : 1.0.1 kaiju : 1.2.3 coordio : 1.4.2 fps_calibrati : 2022.09.11 cherno : 0.3.1 astrometrynet : 0.91 hal : 0.3.0 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 35 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : FAILED STUIAdditions : 3.1.18 --------------Other Versions-------------- sdss-obstools : 3.9.0 TPM : 1.2-31-gd7eb MCP : 2.1 PLC : V35 Fiducials : v1_148 idlspec2d : v6_0_7 Kronos : 1.4.0a roboscheduler : 1.2.2 /dev/sdb1 2.0T 192G 1.9T 10% /home 10.25.1.230:/data 21T 15T 5.5T 73% /data ---- APOGEE Data Summary ---- Morning cals: APOGEE data list, mjd=59841, all raw_data: /data/apogee/utr_cdr/59841/apRaw-*.fits archive: /data/apogee/archive/59841/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59841/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 59841 10 11:27 42790010 Dark 60 A 13.0 ---------- a-b-c - - 59841 11 11:37 42790011 Dark 60 A 13.0 ---------- a-b-c - - 59841 12 11:48 42790012 Dark 60 A 13.0 ---------- a-b-c - - 59841 13 11:50 42790013 QuarFlat 10 A 13.0 ---------- a-b-c - - 59841 14 11:52 42790014 QuarFlat 10 A 13.0 ---------- a-b-c - - 59841 15 11:54 42790015 QuarFlat 10 A 13.0 ---------- a-b-c - - 59841 16 11:56 42790016 Arc-Thar 12 B 13.5 ---------- a-b-c 3.16 - 59841 17 12:03 42790017 Arc-Une 40 B 13.5 ---------- a-b-c - - 59841 18 12:05 42790018 Arc-Thar 12 A 13.0 ---------- a-b-c 2.66 - 59841 19 12:13 42790019 Arc-Une 40 A 13.0 ---------- a-b-c - - 59841 20 12:18 42790020 Dark 30 A 13.0 ---------- a-b-c - - 59841 21 12:23 42790021 InteFlat 30 A 13.0 ---------- a-b-c - - 59841 22 12:29 42790022 InteFlat 30 A 13.0 ---------- a-b-c - - 59841 23 12:34 42790023 InteFlat 30 A 13.0 ---------- a-b-c - - 59841 24 12:40 42790024 Dark 30 A 13.0 ---------- a-b-c - - 59841 25 12:45 42790025 Arc---- 30 A 13.0 ---------- a-b-c - 96 FPI ------------------------------------------------------------------------------ APOGEE data list, mjd=59842, all raw_data: /data/apogee/utr_cdr/59842/apRaw-*.fits archive: /data/apogee/archive/59842/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59842/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 59842 1 22:58 42800001 Dark 3 A 13.0 ---------- a-b-c - - 59842 2 23:11 42800002 Dark 60 A 13.0 ---------- a-b-c - - 59842 3 23:13 42800003 QuarFlat 10 A 13.0 ---------- a-b-c - - 59842 4 23:15 42800004 Arc-Thar 12 B 13.5 ---------- a-b-c 3.12 - 59842 5 23:22 42800005 Arc-Une 40 B 13.5 ---------- a-b-c - - 59842 6 23:24 42800006 Arc-Thar 12 A 13.0 ---------- a-b-c 2.65 - 59842 7 23:32 42800007 Arc-Une 40 A 13.0 ---------- a-b-c - - 59842 8 23:43 42800008 Dark 60 A 13.0 ---------- a-b-c - - 59842 9 23:48 42800009 Arc---- 30 A 13.0 ---------- a-b-c - 91 FPI ------------------------------------------------------------------------------ ---- eBOSS Data Summary ---- Reading FITS from /data/spectro//59842 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 23:01 b1-00346455 ----------- ---- bia Clo 0.0 2851 3004 1634 2 23:01 r1-00346455 ----------- ---- bia Clo 0.0 2851 3004 1634 3 23:02 b1-00346456 ----------- ---- dar Clo 20.0 2851 3004 1634 4 23:02 r1-00346456 ----------- ---- dar Clo 20.0 2851 3004 1634 5 23:04 b1-00346457 -999--999 6850 arc Lef 4.1 2851 3004 1634 HART 6 23:04 r1-00346457 -999--999 6850 arc Lef 4.1 2851 3004 1634 HART 7 23:05 b1-00346458 -999--999 6850 arc Rig 4.0 2851 3004 1634 HART 8 23:05 r1-00346458 -999--999 6850 arc Rig 4.0 2851 3004 1634 HART 9 23:06 b1-00346459 -999--999 6850 arc Out 4.1 3361 3514 2145 10 23:06 r1-00346459 -999--999 6850 arc Out 4.1 3361 3514 2145 11 23:08 r1-00346460 -999--999 6850 fla Out 25.1 3361 3514 2145 12 23:08 b1-00346460 -999--999 6850 fla Out 25.1 3361 3514 2145 13 23:52 r1-00346461 -999--999 6853 sci Out 300.1 3361 3514 2145 14 23:52 b1-00346461 -999--999 6853 sci Out 300.1 3361 3514 2145 15 02:38 r1-00346462 -999--999 6853 arc Lef 4.1 3361 3514 2145 HART 16 02:38 b1-00346462 -999--999 6853 arc Lef 4.1 3361 3514 2145 HART 17 02:39 r1-00346463 -999--999 6853 arc Rig 4.1 3361 3514 2145 HART 18 02:39 b1-00346463 -999--999 6853 arc Rig 4.1 3361 3514 2145 HART 19 02:40 r1-00346464 -999--999 6853 fla Out 25.1 5497 5649 4281 20 02:40 b1-00346464 -999--999 6853 fla Out 25.1 5497 5649 4281 21 02:42 r1-00346465 -999--999 6853 fla Out 25.1 5497 5649 4281 22 02:42 b1-00346465 -999--999 6853 fla Out 25.1 5497 5649 4281 23 02:44 b1-00346466 -999--999 6853 fla Out 25.1 5497 5649 4281 24 02:44 r1-00346466 -999--999 6853 fla Out 25.1 5497 5649 4281 ---- BOSS Spectrographs Collimation ---- Time TSP1 R1 B1 Move1 B1Resid ============================================= 23:06:10 10.1 -329 -4.2 510 -2.6 02:40:00 9.7 1319 -9.3 2137 -2.6 ---- Weather Log ---- Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma 18:00 16.2 11.1 8.8 190.4 425 18:15 16.4 11.2 7.7 195.0 dpDep 18:30 15.6 11.5 7.7 188.8 539 18:45 15.2 11.6 6.9 205.5 465 19:00 15.4 11.2 10.1 187.4 477 19:15 16.1 11.8 10.9 209.0 486 19:30 16.8 11.8 2.3 188.1 378 19:45 15.6 11.6 7.9 208.7 518 20:00 15.5 12.2 5.3 196.8 419 20:15 17.0 12.2 10.2 201.6 446 20:30 15.6 9.5 5.1 206.2 412 20:45 16.2 11.0 5.7 218.4 354 21:00 17.0 12.2 8.1 189.5 366 21:15 16.6 12.4 5.3 193.9 392 21:30 16.2 10.4 4.8 200.0 351 21:45 17.9 11.1 13.6 219.0 319 22:00 16.1 10.9 10.5 202.1 371 22:15 19.1 12.2 10.2 246.1 386 22:30 17.3 12.0 15.2 212.5 397 22:45 17.1 13.1 15.8 208.1 438 23:00 16.5 12.9 9.3 214.3 480 23:15 15.5 12.2 6.9 189.4 dpDep 23:30 14.9 11.9 11.0 125.8 418 23:45 14.4 11.3 8.7 129.0 540 00:00 14.6 11.4 4.8 119.6 471 00:15 13.7 11.2 9.4 89.9 614 00:30 13.1 11.3 9.4 88.4 dpDep 00:45 12.2 11.2 10.6 87.7 dpDep 01:00 12.4 10.7 8.8 106.5 279 01:15 12.2 10.2 11.8 114.5 dpDep 01:30 12.2 9.9 11.5 109.6 245 01:45 12.4 9.7 9.8 117.8 188 02:00 13.2 9.3 10.6 119.0 dpDep 02:15 13.4 8.8 11.1 119.6 291 02:30 13.1 8.4 18.0 125.8 270 341 2.36 02:45 13.0 9.1 14.2 120.1 dpDep 341 2.31 03:00 13.1 9.7 21.6 125.3 347 341 1.58 03:15 12.8 10.2 17.9 122.7 dpDep 343 18.76 03:30 13.2 10.4 20.0 141.0 448 343 28.27 03:45 13.1 10.7 17.8 133.3 dpDep 341 2.20 04:00 12.9 10.9 19.2 140.7 443 341 1.82 04:15 13.2 10.5 16.5 154.6 447 341 2.29 04:30 12.9 11.2 15.5 155.7 435 327 3.22 04:45 13.1 10.9 17.3 152.5 508 341 2.06 05:00 12.9 11.3 16.5 146.1 524 341 2.48 05:15 12.7 11.4 16.1 148.2 549 341 2.25 05:30 12.8 11.3 16.7 142.8 531 341 2.22 05:45 12.8 10.9 15.9 155.7 554 341 2.36 06:00 13.1 10.6 14.8 159.1 507 341 3.10 06:15 13.1 10.9 15.6 158.8 493 341 3.94 06:30 13.1 10.8 17.4 156.1 499 341 2.18 06:45 12.9 10.8 14.0 153.1 493 341 2.00 07:00 12.8 10.9 13.2 151.6 574 341 6.78 07:15 12.7 10.7 11.5 159.3 540 342 3.73 07:30 12.9 10.5 10.6 180.6 452 343 10.94 07:45 13.3 10.5 10.5 184.6 dpDep 343 24.78 08:00 13.3 10.7 10.1 187.8 424 352 75.69 08:15 13.0 10.7 12.8 189.7 dpDep 360 74.13 08:30 13.3 10.9 13.9 203.3 536 368 67.25 08:45 13.2 10.9 14.0 192.8 530 401 45.62 09:00 13.4 11.1 11.7 191.3 dpDep 375 56.93 09:15 13.3 11.1 11.4 200.5 478 372 58.21 09:30 13.2 11.2 8.4 193.1 478 329 54.36 09:45 13.1 11.3 12.8 212.3 572 351 40.83 10:00 13.0 11.4 10.7 183.1 dpDep 342 14.14 10:15 13.0 11.4 8.2 186.3 540 383 88.70 10:30 12.9 11.4 11.7 182.7 545 362 66.34 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 59842 Fill night time in Hours below *=*=*= Total bright time : 0.3 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 0.3 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 9.5 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 9.5 *=*=*= Dark time lost to technical issues : 0