APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Sunday November 16, 2025 (SJD 60996) ---- Observing Team ---- Dmitry Bizyaev ---- Observing Plan ---- Sun 16 Nov 2025 60996 0.08 00:57 01:12 12:24 21:54 09:08 --- dark (11.2 h) plus 0.2 h for 12 deg design ---- Observing Summary ---- The entire night was lost to cloudy sky and/or high humidity. ---- Observing Log ---- Afternoon ========= All systems found alifve except for Grafana plots for GFA cameras, e,g, their temperature plots. Also, robot 973 gave some problems when configuration was reversed. The BOSS collimation looks, OK, not adjueted: 23:20:47 Hartmann collimate output sp1: offset: r= -329 (In Focus); b= -1.2 (In Focus) sp1: pred. move: spAverageMove= 28 sp1: pred. resid: r= -357, b= -1.1, txt= OK, spTemp = 5.6 wPositions = 35 35 35 35; MCP instr = 10 (mounted); ---------- motorPosition sp1: -2998, -2986, -2987; ---------- Median spectrographs temperatures: Tsp1= 5.6 Outside air temperature= 0.0; ---------- Boss CCD Temp : sp1r0= -126.2 (-126.6); sp1b2= -101.6 (-100.0); sp1LN2: 10.0; ---------- Apogee: vacuum = 4.422e-07 ( < 1.1e-06 ) ln2Level = 93.549 ( > 85 ) ---------- Weather: Humidity = 62.1 % AirTemp = 8.3 C, DpTemp = 1.5 C diff= 6.8 C wind25m = 16.4 mph, direction = 264.8 deg ---------- semOwner = TCC:0:0 stuiVer=(1.7.19 2023-01-12), pid = 730 memory = 593 MB ---------- Jaeger disabled robots: 647, 986 There are 2 disabled robots 22:17:49 apogee exposure file= apRaw-54340001 3 B ArcLamp 22:18:13 boss exposure 0 sec file=0390833 bias 22:19:29 apogee exposure file= apRaw-54340002 60 B Dark 22:19:56 boss exposure 20 sec file=0390834 dark ======================================================= =====>>>>> 22:21:03 Config Loaded: -999--999-22642 ======================================================= 22:21:46 FVC exposure image /data/fcam/60996/proc-fimg-fvc1n-0001.fits 22:21:45 fvc_perc_90= 51.4 fvc_percent_reached= 2.8 fvc_rms= 34.33 22:30:20 apogee exposure file= apRaw-54340003 10 B QuartzFlat 22:32:12 apogee exposure file= apRaw-54340004 12 B ArcLamp 22:34:25 apogee exposure file= apRaw-54340005 40 B ArcLamp 22:41:42 apogee exposure file= apRaw-54340006 12 A ArcLamp 22:43:54 apogee exposure file= apRaw-54340007 40 A ArcLamp 22:51:20 apogee exposure file= apRaw-54340008 60 A Dark 23:02:12 apogee exposure file= apRaw-54340009 30 A ArcLamp 23:07:35 apogee exposure file= apRaw-54340010 30 B ArcLamp 23:18:47 boss exposure 4 sec file=0390835 arc 23:19:27 boss exposure 4 sec file=0390836 arc 23:20:44 Hartmann: offset r= -329 (In Focus) b = -1.2 (In Focus) pred_move = 28 pred_resid r= -357 b= -1.1 cmnt=OK 23:21:10 boss exposure 4 sec file=0390837 arc 23:23:11 boss exposure 25 sec file=0390838 flat Robot 973 got locked when "reverse" was issued. I teid to explode and try again, then to explode it alone, then to move it and then unwind. Nothiong worked. Finally, I exploded nearby robots and was able to fold the array. I decided to leave it enabled until somethign happens with another configuration. See "Problems in detail" section for the errors. Cerebro actor was restarted to estore the Grafana plots. 22:30Z started ventilation The sky is cloudy. ===== Night ===== Opening postponed because of completely cloudy sky. A chiller dew point alert emerged: 06:05:16 .jaeger 0 jaeger w alert_chiller_dew_point=1 06:05:16 .jaeger 0 jaeger w text="Fluid temperature is approaching dew point limit." I observed it and decided to reload jaeger, but the alert suddenly cleared: 06:22:19 .jaeger 0 jaeger i alert_chiller_dew_point=0 Warn jaeger.alert_dew_point at 2025-11-17 06:45:22 UT found 1 06:45:22 .jaeger 2174 jaeger i chiller_temperature_value=5.0; chiller_temperature_setpoint=5.0; chiller_flow_value=4.9; chiller_flow_setpoint=6.0 Altough the outside humidity is 70%, the chiller in Grafana shows 85%. The sky is still cloudy, and the wind gusts are yellow. I decided to stop ventilation. 06:55Z ventilation stopped. Surfaces are somwehere damp, but not wet. Note that the inclosure's humidity meter shows ~90%. 01:29:27 boss exposure 900 sec file=0390839 dark 02:05:42 boss exposure 900 sec file=0390840 dark 02:22:10 boss exposure 900 sec file=0390841 dark 12:18Z humidity gradually goes up, and the sky is mostly cloudy. Calling it a night. Calibrations started. [ (54340001, 3, 3, '/data/apogee/raw/60996/apRaw-54340001-001.fits', 'ARCLAMP', 'None', '', '22641', '755378', '103366', 32.041, '2025-11-16T22:17:49.681', 'Podium', 'Closed', 'Closed', 60996, 'APO', 13.499)] problem with exposure: 54340001 num field design config exptype dithpix nread relflux okay dx dy sigx sigy fpi1 fpi2 -------- ------ ------ ------ ------------ ------- ----- ------- ----- ------- ------- ------ ------ ---- ---- 54340001 103366 755378 22641 ARCLAMP-None 13.5 3 0 False 0.00 0.00 0.00 0.00 0 0 54340002 103366 755378 22641 DARK 13.5 60 -30 True -- -- -- -- -- -- 54340003 -999 -999 22642 QUARTZFLAT 13.5 10 97 True -- 0.59 -- 0.85 -- -- 54340004 -999 -999 22642 ARCLAMP-THAR 13.5 12 97 True 23.34 1.24 1.06 0.88 -- -- 54340005 -999 -999 22642 ARCLAMP-UNE 13.5 40 1631 True 23.15 -5.32 1.09 0.86 -- -- 54340006 -999 -999 22642 ARCLAMP-THAR 13.0 12 102 True 22.82 1.21 1.07 0.88 -- -- 54340007 -999 -999 22642 ARCLAMP-UNE 13.0 40 1546 True 22.62 -5.36 1.08 0.90 -- -- 54340008 -999 -999 22642 DARK 13.0 60 0 True -- -- -- -- -- -- 54340009 -999 -999 22642 FPI 13.0 30 240 True 9.82 -5.21 1.10 0.88 -- -- 54340010 -999 -999 22642 FPI 13.5 30 236 True -2.00 -5.24 1.10 0.90 -- -- num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 mtp10 -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----- 54340003 QUARTZFLAT 10 66 70 80 90 89 85 87 89 89 81 ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.1020 0.1030 Axial B: -0.0270 -0.0250 Axial C: -0.1760 -0.1740 Trans D: -0.0300 -0.0300 Lateral E: 0.3400 0.3400 Lateral F: 0.5470 0.5470 GALILS Commanded: 8450 -1050 8800 22100 -2000 -1700 Actual: 8450 -1050 8800 22100 -2000 -1700 SETMIR VALUES PrimDesOrient: 1.06 2.03 16.67 -410.97 -787.67 PrimOrient: 1.06 2.03 16.67 -410.97 -787.67 SECONDARY: ---------- Focus: 0.0000 Air temp: 0.00 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -7.6390 -7.6390 Axial B: -7.5360 -7.5320 Axial C: -7.5280 -7.5280 Trans D: 0.2010 0.1940 GALILS Commanded: 2250150 2248800 2241650 -1800 -1750 Actual: 2216674 2226979 2200573 -1800 -1750 SETMIR VALUES SecDesOrient: 1757.0 7.9 1.5 -39.3 35.42 SecOrient: 1757.92 7.95 1.63 -39.22 36.32 ---- Telescope Status ---- Never opened. No interlocks bypassed Status at: 12:20Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: n/a; FPS is balanced LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 171 lbs, 19 psi FPS Status ========== CP10 Loop = 7.41 L/min GFA Loop = 2.38 L/min Supply pressure = 45.1 psi Return pressure = 29.4 psi Chiller Status ============== Set temperarature = 4.5 C Flow = 4.9 Gpm ---- Software Used ---- STUI : 1.7.19 alerts : 2.3.0 boss : 6.0.5a1 -boss.specMec : 3.0.4 apo : 4.1.0 fliswarm : 0.5.1a0 jaeger : 1.10.3a1 kaiju : 1.4.1 coordio : 1.17.1a0 fps_calibrati : 2025.02.24+f64593b cherno : 1.1.0 astrometrynet : 0.93 hal : 1.4.1 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.3.1 -mcp.plc : 35 -mcp.azFiduci : v1_152 -mcp.altFiduc : v1_152 -mcp.rotFiduc : v1_152 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : FAILED Boson : 0.5.1 ---- APOGEE Data Summary ---- Last morning's calibraions were OK: 60995 89 13:17 54330089 Dark 60 A 13.0 ---------- a-b-c - - 60995 90 13:27 54330090 Dark 60 A 13.0 ---------- a-b-c - - 60995 91 13:38 54330091 Dark 60 A 13.0 ---------- a-b-c - - 60995 92 13:40 54330092 QuarFlat 10 A 13.0 ---------- a-b-c - - 60995 93 13:42 54330093 QuarFlat 10 A 13.0 ---------- a-b-c - - 60995 94 13:44 54330094 QuarFlat 10 A 13.0 ---------- a-b-c - - 60995 95 13:46 54330095 Arc-Thar 12 B 13.5 ---------- a-b-c 0.32 - 60995 96 13:53 54330096 Arc-Une 40 B 13.5 ---------- a-b-c - - 60995 97 13:55 54330097 Arc-Thar 12 A 13.0 ---------- a-b-c -0.14 - 60995 98 14:03 54330098 Arc-Une 40 A 13.0 ---------- a-b-c - - 60995 99 14:08 54330099 Dark 30 A 13.0 ---------- a-b-c - - 60995 100 14:13 54330100 InteFlat 30 A 13.0 ---------- a-b-c - - 60995 101 14:19 54330101 InteFlat 30 A 13.0 ---------- a-b-c - - 60995 102 14:24 54330102 InteFlat 30 A 13.0 ---------- a-b-c - - 60995 103 14:30 54330103 Dark 30 A 13.0 ---------- a-b-c - - 60995 104 14:35 54330104 Arc---- 30 A 13.0 ---------- a-b-c - 108 FPI A 60995 105 14:41 54330105 Arc---- 30 B 13.5 ---------- a-b-c - 106 FPI B APOGEE data list, mjd=60996, all raw_data: /data/apogee/utr_cdr/60996/apRaw-*.fits archive: /data/apogee/archive/60996/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/60996/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 60996 1 22:18 54340001 Arc---- 3 B 13.5 ---------- a-b-c - 0 test 60996 2 22:29 54340002 Dark 60 B 13.5 ---------- a-b-c - - 60996 3 22:31 54340003 QuarFlat 10 B 13.5 ---------- a-b-c - - 60996 4 22:33 54340004 Arc-Thar 12 B 13.5 ---------- a-b-c 0.34 - 60996 5 22:41 54340005 Arc-Une 40 B 13.5 ---------- a-b-c - - 60996 6 22:43 54340006 Arc-Thar 12 A 13.0 ---------- a-b-c -0.17 - 60996 7 22:50 54340007 Arc-Une 40 A 13.0 ---------- a-b-c - - 60996 8 23:01 54340008 Dark 60 A 13.0 ---------- a-b-c - - 60996 9 23:07 54340009 Arc---- 30 A 13.0 ---------- a-b-c - 108 FPI A 60996 10 23:12 54340010 Arc---- 30 B 13.5 ---------- a-b-c - 106 FPI B ------------------------------------------------------------------------------ ---- BOSS Data Summary ---- Reading FITS from /data/spectro//60996 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 22:18 b1-00390833 ----------- ---- bia Clo 0.0 -3028 -3017 -3015 2 22:18 r1-00390833 ----------- ---- bia Clo 0.0 -3028 -3017 -3015 3 22:19 b1-00390834 ----------- ---- dar Clo 20.0 -3028 -3017 -3015 4 22:19 r1-00390834 ----------- ---- dar Clo 20.0 -3028 -3017 -3015 5 23:19 b1-00390835 -999--999 22642 arc Lef 4.1 -3028 -3017 -3015 HART 6 23:19 r1-00390835 -999--999 22642 arc Lef 4.1 -3028 -3017 -3015 HART 7 23:20 r1-00390836 -999--999 22642 arc Rig 4.1 -3028 -3017 -3015 HART 8 23:20 b1-00390836 -999--999 22642 arc Rig 4.1 -3028 -3017 -3015 HART 9 23:21 b1-00390837 -999--999 22642 arc Out 4.1 -2998 -2986 -2987 10 23:21 r1-00390837 -999--999 22642 arc Out 4.1 -2998 -2986 -2987 11 23:23 b1-00390838 -999--999 22642 fla Out 25.1 -2998 -2986 -2987 12 23:23 r1-00390838 -999--999 22642 fla Out 25.1 -2998 -2986 -2987 13 01:29 b1-00390839 ----------- ---- dar Clo 900.0 -2998 -2986 -2987 14 01:29 r1-00390839 ----------- ---- dar Clo 900.0 -2998 -2986 -2987 15 02:05 r1-00390840 ----------- ---- dar Clo 900.0 -2998 -2986 -2987 16 02:05 b1-00390840 ----------- ---- dar Clo 900.0 -2998 -2986 -2987 17 02:22 b1-00390841 ----------- ---- dar Clo 900.0 -2998 -2986 -2987 18 02:22 r1-00390841 ----------- ---- dar Clo 900.0 -2998 -2986 -2987 ---- BOSS Spectrographs Collimation ---- Time TSP1 R1 B1 Move1 B1Resid ============================================= 23:20:40 5.7 -329 -1.2 28 -1.1 ---- Weather Log ---- Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma 19:00 9.1 2.1 11.5 59.8 445 19:15 9.6 1.2 8.4 70.1 447 19:30 9.9 1.7 14.8 55.7 496 19:45 9.4 1.8 13.1 08.3 571 20:00 9.6 2.1 16.0 33.3 635 20:15 9.3 2.3 15.6 60.6 654 20:30 10.2 1.6 12.7 39.0 572 20:45 11.5 0.6 17.8 35.8 572 21:00 8.9 0.7 13.5 45.7 505 21:15 10.4 1.2 23.2 38.2 579 21:30 10.6 1.0 20.2 39.3 656 21:45 10.5 0.9 23.4 46.1 662 22:00 8.8 1.0 18.9 42.9 609 22:15 9.2 0.8 11.0 62.7 615 22:30 9.8 1.0 18.0 50.2 656 22:45 9.3 0.9 19.3 65.0 585 23:00 8.4 1.1 16.9 44.0 556 23:15 8.4 1.2 15.0 57.0 720 23:30 8.3 1.6 15.2 63.6 640 23:45 8.1 1.8 21.7 42.2 774 00:00 8.1 2.1 22.5 58.9 725 00:15 7.8 2.2 26.2 57.1 714 00:30 7.6 2.3 20.7 62.3 703 366 63.29 00:45 7.5 1.8 19.4 53.0 794 389 66.97 01:00 7.4 2.3 18.1 51.9 dpDep 389 103.87 01:15 7.2 0.7 14.5 51.7 dpDep 372 70.15 01:30 7.3 0.4 14.5 61.5 779 367 90.79 01:45 7.3 0.1 18.8 49.4 802 374 64.58 02:00 7.3 0.3 19.6 58.1 839 372 65.17 02:15 7.3 0.3 16.9 49.9 778 361 77.86 02:30 7.2 0.3 17.1 53.2 803 358 88.11 02:45 7.0 0.3 22.2 44.9 884 343 52.93 03:00 6.9 -2.3 19.0 60.6 886 359 63.99 03:15 6.6 0.2 25.5 42.1 954 341 22.52 03:30 6.6 0.1 24.7 42.5 1170 388 88.58 03:45 6.3 0.2 20.6 47.9 1088 378 97.23 04:00 6.4 -0.4 23.0 59.1 1091 390 85.04 04:15 6.2 -0.8 17.0 58.7 1057 384 99.03 04:30 6.1 -0.1 18.1 43.0 955 393 75.49 04:45 6.1 0.2 21.0 50.5 953 385 58.41 05:00 5.9 0.1 17.9 65.2 859 372 68.49 05:15 5.8 0.1 22.0 66.0 796 395 72.54 05:30 5.8 0.1 18.7 65.8 832 403 69.19 05:45 5.7 -0.2 17.3 63.5 783 397 79.20 06:00 5.4 -0.1 21.9 60.1 816 376 76.31 06:15 5.3 0.1 23.5 64.0 dpDep 395 62.91 06:30 4.8 0.2 20.9 61.7 dpDep 376 77.28 06:45 4.8 -0.1 23.0 65.9 dpDep 378 92.28 07:00 4.9 0.1 25.0 60.4 dpDep 384 77.42 07:15 4.8 0.1 24.6 68.1 dpDep 344 84.31 07:30 4.4 0.1 21.2 67.0 dpDep 357 37.89 07:45 4.4 0.1 22.8 63.1 dpDep 393 85.33 08:00 4.4 0.1 21.7 65.1 dpDep 382 84.00 08:15 4.1 -0.4 24.8 75.6 dpDep 382 66.45 08:30 4.2 -0.2 27.0 70.6 dpDep 392 90.64 08:45 4.1 -0.1 23.7 69.8 dpDep 395 45.14 09:00 4.1 -0.2 21.9 53.9 dpDep 395 105.27 09:15 4.0 -0.1 21.7 65.6 dpDep 493 69.92 09:30 3.9 -0.1 26.3 60.3 dpDep 463 46.85 09:45 3.7 -0.1 23.5 67.2 dpDep 476 43.66 10:00 3.5 -0.1 19.9 63.1 dpDep 375 78.72 10:15 3.3 -0.2 21.2 60.8 dpDep 373 38.92 10:30 3.1 -0.1 22.5 63.5 dpDep 372 82.97 10:45 2.9 -0.4 20.9 61.8 dpDep 350 69.26 11:00 2.9 -0.1 18.9 62.6 dpDep 371 72.25 11:15 2.8 -0.4 18.6 66.1 dpDep 360 40.64 11:30 2.3 -0.2 20.5 62.1 dpDep 365 85.97 11:45 2.3 -0.4 26.3 51.7 dpDep 359 87.48 12:00 2.1 -0.4 27.1 61.3 dpDep 369 101.69 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 60996 Fill night time in Hours below *=*=*= Total bright time : 0.2 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 0.2 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 11.2 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 11.2 *=*=*= Dark time lost to technical issues : 0 ---- Problems in Detail ---- 22:23:55 .jaeger 0 jaeger e text="A collision was detected in positioner 973. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 22:23:55 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 22:23:56 hal.hal 46116 apogee i exposureWroteFile="apRaw-54340002-025.fits"; exposureWroteSummary="apRaw-54340002.fits" 22:23:56 hal.hal 46116 apogee i utrReadState="apRaw-54340002",Done,25,60 22:23:56 .jaeger 0 jaeger e locked=T; locked_by=973; locked_axes=alpha; locked_alpha=283.6885356903076; locked_beta=88.6392080783844 22:23:56 APO.dmbiz 23 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 15.20 seconds after start" [22:25:35 .jaeger 0 jaeger e text="A collision was detected in positioner 973. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 22:25:35 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 22:25:36 .jaeger 0 jaeger e locked=T; locked_by=973; locked_axes=alpha; locked_alpha=140.4480278491974; locked_beta=173.73286843299866 22:25:39 APO.dmbiz 26 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 8.19 seconds after start" 22:25:39 CmdDone: APO.dmbiz 26 jaeger unwind 22:27:04 .jaeger 0 jaeger e text="A collision was detected in positioner 973. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 22:27:04 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 22:27:06 .apogeeql 0 apogeeql w text="guider cartridgeKey is not well defined: KeyVar('guider', cartridgeLoaded=(None, None, None, None, None))" 22:27:06 .apogeeql 0 apogeeql w text="failed to index KeyVar('guider', version=(None,)) by None from version for v_guider: KeyVar('guider', version=(None,)) is unknown" 22:27:06 hal.hal 46116 apogee i utrReadState="apRaw-54340002",Reading,44,60 22:27:06 hal.hal 46116 apogee i utrReadState="apRaw-54340002",Saving,43,60 22:27:06 .jaeger 0 jaeger e locked=T; locked_by=973; locked_axes=alpha; locked_alpha=105.3540587425232; locked_beta=178.46860885620117 22:27:06 APO.dmbiz 29 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 3.86 seconds after start" 22:27:06 CmdDone: APO.dmbiz 29 jaeger unwind 22:28:27 .jaeger 0 jaeger e text="A collision was detected in positioner 973. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 22:28:27 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 22:28:30 APO.dmbiz 32 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 4.50 seconds after start" 22:28:30 CmdDone: APO.dmbiz 32 jaeger unwind --force