APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Saturday October 24, 2026 (SJD 61338) ---- Observing Team ---- Dan Oravetz ---- Observing Plan ---- Sat 24 Oct 2026 61338 bright (10.9 h, w/ 0.2 h for 12 deg design) ---- Observing Summary ---- Tonight was a continuation of the thunderstorms and rain from last night and in-between active waves or rain events everything was soaking wet and humid. I did in-dome test slews to answer some questions we had encountered about PLC status bits during troubleshooting the out-of-closed-loop problem from a few nights ago. Note that positioner complained of a collision 4 times during the checkout testing, so I disabled it for now. ---- Observing Log ---- ============================================================================ AFTERNOON CHECKOUT ============================================================================ All systems alive and responsive. Restarted cherno. Disk space: Filesystem Size Used Avail Use% Mounted on /dev/sdb1 1.0T 435G 590G 43% /home /dev/sdc1 22T 12T 9.8T 54% /data FPS powered on. Positioners are home/folded: 01:14:12 APO.doravetz_2 51 jaeger i folded=T 01:14:12 APO.doravetz_2 51 jaeger : text="All positioners are folded." 01:14:19 APO.doravetz_2 52 jaeger : permanently_disabled=463,647,986; manually_disabled=630,973,996,1126 Loaded a manual folded configuration: 01:14:43 CmdStarted: APO.doravetz_2 53 jaeger configuration load --from-positions 01:14:45 APO.doravetz_2 53 jaeger d text="Resetting cherno offsets." 01:14:45 APO.doravetz_2 53 jaeger d text="Focal plane scale: 1.0." 01:14:45 APO.doravetz_2 53 jaeger d configuration_loaded=28010,-999,-999,-999.0,-999.0,0,-999.0,-999.0,/home/sdss5/software/sdsscore/main/apo/summary_files/028XXX/0280XX/confSummary-28010.par,F 01:14:50 APO.doravetz_2 53 jaeger i configuration_snapshot=/data/logs/jaeger/configuration_snapshots/61338/configuration_snapshot_61338_28010.pdf 01:14:50 APO.doravetz_2 53 jaeger d design_preloaded=-999 01:14:50 APO.doravetz_2 53 jaeger : text="Configuration 28010 loaded." 01:14:50 CmdDone: APO.doravetz_2 53 jaeger configuration load --from-positions Took a '--no-apply' FVC image: FVC loop results: fvc_rms= 36.78 fvc_deltarms= 0.75 fvc_filename=/data/fcam/61338/fimg-fvc1n-0001.fits fvc_perc_90= 52.40 fvc_percent_reached= 2.80 fvc iterations= 1 Scattered-light check: 01:15:56 CmdStarted: APO.doravetz_2 56 fliswarm talk -n fvc expose 50 No evidence of light artifacts. GFA biases, darks, and flat: 22:07:18 CmdStarted: APO.doravetz_3 62 fliswarm talk -c gfa expose --bias --count 10 22:08:39 CmdStarted: APO.doravetz_3 63 fliswarm talk -c gfa expose --dark --count 5 5 22:12:29 CmdStarted: APO.doravetz_3 67 fliswarm talk -c gfa expose --flat 1 Exercised positioners: 01:17:03 CmdStarted: APO.doravetz_2 57 jaeger configuration random --danger 01:18:08 .jaeger 0 jaeger e text="A collision was detected in positioner 828. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 01:18:08 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 01:18:09 .jaeger 0 jaeger i fps_status=0x14 01:18:10 .jaeger 0 jaeger e locked=T; locked_by=828; locked_axes=alpha; locked_alpha=10.003959015011787; locked_beta=169.99354720115662 01:18:10 APO.doravetz_2 57 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/61338/trajectory-61338-0001.json 01:18:10 APO.doravetz_2 57 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 31.93 seconds after start" 01:18:10 CmdDone: APO.doravetz_2 57 jaeger configuration random --danger 01:18:15 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/61338/fps_snapshot_61338_0001.pdf 01:18:15 .jaeger 0 jaeger w text="Snapshot for locked FPS: /data/logs/jaeger/snapshots/61338/fps_snapshot_61338_0001.pdf (JaegerUserWarning)" 01:18:16 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/61338/fps_snapshot_61338_0001.pdf Issued the 'unlock', 'explode 10', and 'unwind', but then another collision during the unwinding: 01:19:45 .jaeger 0 jaeger e text="A collision was detected in positioner 671. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 01:19:45 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 01:19:45 .jaeger 0 jaeger i fps_status=0x14 01:19:47 .jaeger 0 jaeger e locked=T; locked_by=671; locked_axes=alpha; locked_alpha=118.9744234085083; locked_beta=160.02262830734253 01:19:47 APO.doravetz_2 61 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/61338/trajectory-61338-0003.json 01:19:47 APO.doravetz_2 61 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 3.58 seconds after start" Issued the 'unlock', 'explode 10', and 'unwind', but then another collision reported by 671 during the unwinding: 01:20:58 .jaeger 0 jaeger e text="A collision was detected in positioner 671. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 01:20:58 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 01:20:59 APO.doravetz_2 64 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/61338/trajectory-61338-0005.json 01:20:59 APO.doravetz_2 64 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 3.40 seconds after start" 01:20:59 CmdDone: APO.doravetz_2 64 jaeger unwind 01:21:00 .jaeger 0 jaeger e locked=T; locked_by=671; locked_axes=alpha; locked_alpha=83.49131941795349; locked_beta=170.02939224243164 Tried again. Positioner 671 moved closer to home this time, but failed before reaching home. 01:25:41 .jaeger 0 jaeger e text="A collision was detected in positioner 671. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 01:25:41 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 01:25:43 .jaeger 0 jaeger e locked=T; locked_by=671; locked_axes=alpha; locked_alpha=71.97730422019958; locked_beta=158.30640077590942 01:25:43 .jaeger 0 jaeger i fps_status=0x14 01:25:43 APO.doravetz_2 70 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/61338/trajectory-61338-0008.json 01:25:43 APO.doravetz_2 70 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 7.66 seconds after start" This time the 'unwind' finally completed. I disabled 671 for now: 01:27:33 APO.doravetz_2 75 jaeger : permanently_disabled=463,647,986; manually_disabled=630,671,973,996,1126 Trying another '--danger' move with the FPS now that 671 is disabled. 01:28:12 CmdStarted: APO.doravetz_2 76 jaeger configuration random --danger 01:29:27 APO.doravetz_2 76 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/61338/trajectory-61338-0011.json 01:29:28 CmdDone: APO.doravetz_2 76 jaeger configuration random --danger 01:29:32 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/61338/fps_snapshot_61338_0011.pdf 01:29:49 CmdStarted: APO.doravetz_2 77 jaeger configuration reverse BOSS hartmanns, arc, flat: 01:32:51 BOSS expNum=405573, expTime= 4.0s 01:33:31 BOSS expNum=405574, expTime= 4.0s 01:34:12 sp1.motor.move= 22, 19, 21 01:34:12 Hartmann collimate output sp1: offset: r= -659 (In Focus); b= -2.2 (In Focus) sp1: pred. move: spAverageMove= 24 sp1: pred. resid: r= -683, b= -2.1, txt= OK, spTemp = 7.9 01:34:30 BOSS expNum=405575, expTime= 4.0s 01:36:05 BOSS expNum=405576, expTime= 25.0s Did not start ventilation due to approaching thunderstorms. ============================================================================ NIGHT ============================================================================ Did not open at sunset due to ongoing thunderstorms and heavy rain. Testing az_dir_[cw|ccw] bits **************************** 02:33Z - AZ at 121 degrees. az_dir_cw: green; az_dir_ccw: red Moving AZ to -190 degrees. During and through completion of the slew to -190, az_dir_cw: green; az_dir_ccw: red. Moving AZ to 430 degrees. Around 128 degrees (moving positive), the cw and ccw switched: az_dir_cw: red; az_dir_ccw: green Moving AZ to 121 and watching for the switch around 128 degrees again. Verified. For AZ ~127 deg moving positive ==> becomes green az_dir_ccw and red az_dir_cw ~126 deg moving negative ==> becomes green az_dir_cw and red az_dir_ccw 05:39Z - Raining again. Another thunderstorm approaching from the south as well. 08:15Z - Actually some stars visible right now, but it's 100% humid and the pattern of spawning storms to the south and working towards us is continuing. 11:22Z - Called it a night. FPS powered-off. Positioners are home/folded. APOGEE morning cals started. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: -5.4280 -5.4260 Axial B: -5.0380 -5.0390 Axial C: -5.2780 -5.2780 Trans D: -0.8730 -0.8710 Lateral E: 0.3240 0.3245 Lateral F: 0.5335 0.5335 GALILS Commanded: 11450 -1050 5900 26550 -14400 -13850 Actual: 11450 -1050 5900 26550 -14400 -13850 SETMIR VALUES PrimDesOrient: 0.0 -43.05 -8.77 -751.82 -556.2 PrimOrient: -1.06 -43.05 -8.77 -751.82 -556.2 SECONDARY: ---------- Focus: 0.0000 Air temp: 0.00 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -7.9560 -7.9570 Axial B: -6.1130 -6.1140 Axial C: -6.0770 -6.0790 Trans D: 0.2060 0.2050 GALILS Commanded: 360550 383150 340650 -1500 1800 Actual: 321952 350034 299533 -1500 1800 SETMIR VALUES SecDesOrient: 257.0 14.2 5.9 -72.0 -12.76 SecOrient: 257.0 14.22 5.89 -71.79 -12.64 ---- Telescope Status ---- Status at close: Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: manually adjusted for telescope balanced w/ FPS mounted LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 179 lbs, 25 psi FPS Status ========== CP10 Loop = 7.55 L/min GFA Loop = 2.56 L/min Supply pressure = 43.6 psi Return pressure = 28.5 psi Chiller Status ============== Set temperarature = 10.6 C Flow = 5.0 Gpm ---- Software Used ---- STUI : 1.7.21 alerts : 2.3.0 boss : 6.0.5a1 -boss.specMec : 3.0.4 apo : 4.1.0 fliswarm : 0.7.0 jaeger : 1.11.2a1 kaiju : 1.4.1 coordio : 1.18.1 fps_calibrati : 2025.12.04+3e7e2d8 cherno : 1.2.2 astrometrynet : 0.97 hal : 1.4.1 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.3.1 -mcp.plc : 35 -mcp.azFiduci : v1_154 -mcp.altFiduc : v1_154 -mcp.rotFiduc : v1_154 tcc : 1.4.0 apogee : not available apogeecal : not available apogeeql : 2.0.0a0 STUIAdditions : FAILED Boson : 0.7.0 sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : 2.3.1 PLC : V35 Fiducials : v1_154 idlspec2d : 6.3.0dev Kronos : 2.1.0a roboscheduler : 2.1.0 ---- APOGEE Data Summary ---- 61337 morning cals: APOGEE data list, mjd=61337, all raw_data: /data/apogee/utr_cdr/61337/apRaw-*.fits archive: /data/apogee/archive/61337/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/61337/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 61337 10 23:04 57750010 Arc---- 30 B 13.5 ---------- a-b-c - 96 FPI B 61337 11 10:32 57750011 Dark 60 B 13.5 ---------- a-b-c - - 61337 12 10:43 57750012 Dark 60 B 13.5 ---------- a-b-c - - 61337 13 10:54 57750013 Dark 60 B 13.5 ---------- a-b-c - - 61337 14 10:56 57750014 QuarFlat 10 B 13.5 ---------- a-b-c - 100 61337 15 10:58 57750015 QuarFlat 10 B 13.5 ---------- a-b-c - 100 61337 16 10:59 57750016 QuarFlat 10 B 13.5 ---------- a-b-c - 101 61337 17 11:02 57750017 Arc-Thar 12 B 13.5 ---------- a-b-c 3.91 103 61337 18 11:09 57750018 Arc-Une 40 B 13.5 ---------- a-b-c 3.93 95 61337 19 11:11 57750019 Arc-Thar 12 A 13.0 ---------- a-b-c 3.32 103 61337 20 11:18 57750020 Arc-Une 40 A 13.0 ---------- a-b-c 3.40 96 61337 21 11:24 57750021 Dark 30 A 13.0 ---------- a-b-c - - 61337 22 11:29 57750022 InteFlat 30 A 13.0 ---------- a-b-c - - 61337 23 11:34 57750023 InteFlat 30 A 13.0 ---------- a-b-c - - 61337 24 11:40 57750024 InteFlat 30 A 13.0 ---------- a-b-c - - 61337 25 11:45 57750025 Dark 30 A 13.0 ---------- a-b-c - - 61337 26 11:51 57750026 Arc---- 30 A 13.0 ---------- a-b-c - 97 FPI A 61337 27 11:56 57750027 Arc---- 30 B 13.5 ---------- a-b-c - 96 FPI B ------------------------------------------------------------------------------ (observers39) [observer@sdss5-hub: ~]$ list_ap APOGEE data list, mjd=61338, all raw_data: /data/apogee/utr_cdr/61338/apRaw-*.fits archive: /data/apogee/archive/61338/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/61338/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 61338 1 01:13 57760001 Dark 3 B 13.5 ---------- a-b-c - - 61338 2 01:28 57760002 Dark 60 B 13.5 ---------- a-b-c - - 61338 3 01:30 57760003 QuarFlat 10 B 13.5 ---------- a-b-c - 100 61338 4 01:32 57760004 Arc-Thar 12 B 13.5 ---------- a-b-c 3.92 103 61338 5 01:40 57760005 Arc-Une 40 B 13.5 ---------- a-b-c 3.93 95 61338 6 01:42 57760006 Arc-Thar 12 A 13.0 ---------- a-b-c 3.33 102 61338 7 01:49 57760007 Arc-Une 40 A 13.0 ---------- a-b-c 3.39 96 61338 8 02:00 57760008 Dark 60 A 13.0 ---------- a-b-c - - 61338 9 02:06 57760009 Arc---- 30 A 13.0 ---------- a-b-c - 94 FPI A 61338 10 02:11 57760010 Arc---- 30 B 13.5 ---------- a-b-c - 96 FPI B ------------------------------------------------------------------------------ ---- BOSS Data Summary ---- Reading FITS from /data/spectro//61338 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 01:13 r1-00405570 ----------- ---- bia Clo 0.0 -4348 -4344 -4345 2 01:13 b1-00405570 ----------- ---- bia Clo 0.0 -4348 -4344 -4345 3 01:31 r1-00405571 -999--999 28010 arc Lef 4.1 -4348 -4344 -4345 HART 4 01:31 b1-00405571 -999--999 28010 arc Lef 4.1 -4348 -4344 -4345 HART 5 01:31 b1-00405572 -999--999 28010 arc Rig 4.1 -4348 -4344 -4345 HART 6 01:31 r1-00405572 -999--999 28010 arc Rig 4.1 -4348 -4344 -4345 HART 7 01:32 r1-00405573 -999--999 28010 arc Lef 4.1 -3188 -3183 -3186 HART 8 01:32 b1-00405573 -999--999 28010 arc Lef 4.1 -3188 -3183 -3186 HART 9 01:33 r1-00405574 -999--999 28010 arc Rig 4.1 -3188 -3183 -3186 HART 10 01:33 b1-00405574 -999--999 28010 arc Rig 4.1 -3188 -3183 -3186 HART 11 01:34 b1-00405575 -999--999 28010 arc Out 4.1 -3166 -3164 -3165 12 01:34 r1-00405575 -999--999 28010 arc Out 4.1 -3166 -3164 -3165 13 01:36 b1-00405576 -999--999 28010 fla Out 25.1 -3166 -3164 -3165 14 01:36 r1-00405576 -999--999 28010 fla Out 25.1 -3166 -3164 -3165 ---- BOSS Spectrographs Collimation ---- Time TSP1 R1 B1 Move1 B1Resid ============================================= 01:32:20 7.3 329 -6.2 1160 -2.6 01:34:00 7.3 -659 -2.2 24 -2.1 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 61338 Fill night time in Hours below *=*=*= Total bright time : 0.2 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 0.2 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 10.9 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 10.9 *=*=*= Dark time lost to technical issues : 0