Subject: 2.5m obslog 20230305 (SJD 60009) From: doravetz@apo.nmsu.edu Date: Mon, 6 Mar 2023 10:44:08 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Sunday March 05, 2023 (SJD 60009) ---- Observing Team ---- Dan Oravetz Audrey Oravetz (support) ---- Observing Plan ---- Sun 05 Mar 2023 60009 bright (10.3 h, w/ 0.2 h for 12 deg design) BOSS dithers weather permitting ---- Observing Summary ---- Did not open tonight. It was cloudy, but we were also not operational due to a glycol leak - sometime before 04:45Z a glycol leak sprung somewhere on the back side of the telescope, perhaps within the back of the mirror cell area behind the BOSS spectrograph. I cleaned up as much of the gylcol as I could under the telescope, but did not move the telescope from stow so as not to disturb anything not knowing where exactly glyocl may have pooled. ---- Engineering Issues ---- Glycol leak. See Observing summary and Log body. ---- Observing Log ---- ============================================================================ AFTERNOON CHECKOUT ============================================================================ All systems alive and responsive. APOGEE evening cals started. Afternoon disk space: Filesystem Size Used Avail Use% Mounted on /dev/sdb1 2.0T 248G 1.8T 13% /home /dev/sdd1 22T 12T 9.4T 55% /data FPS powered on. Positioners are home/folded: 23:41:47 APO.doravetz 2 jaeger i folded=T 23:41:47 APO.doravetz 2 jaeger : text="All positioners are folded." Loaded a manual folded configuration: 23:41:57 CmdStarted: APO.doravetz 3 jaeger configuration load --from-positions 23:42:04 APO.doravetz 3 jaeger d text="Focal plane scale: 1.0." 23:42:04 APO.doravetz 3 jaeger d configuration_loaded=8954,-999,-999,-999.0,-999.0,0,-999.0,-999.0,/home/sdss5/software/sdsscore/main/apo/summary_files/0089XX/confSummary-8954.par,F 23:42:08 APO.doravetz 3 jaeger i configuration_snapshot=/data/logs/jaeger/configuration_snapshots/60009/configuration_snapshot_60009_8954.pdf 23:42:08 APO.doravetz 3 jaeger d design_preloaded=-999 23:42:08 APO.doravetz 3 jaeger : text="Configuration 8954 loaded." 23:42:08 CmdDone: APO.doravetz 3 jaeger configuration load --from-positions Took a '--no-apply' FVC image: 23:45:09 CmdStarted: APO.doravetz 4 jaeger fvc loop --no-apply 23:45:37 FVC Loop fvc_rms= 39.18 fvc_deltarms= 19.84 fvc_filename=/data/fcam/60009/fimg-fvc1n-0001.fits fvc_perc_90= 51.00 fvc_percent_reached= 2.80 fvc iterations= 1 Scattered-light check: 23:46:46 CmdStarted: APO.doravetz 5 fliswarm talk -n fvc expose 50 23:46:52 CmdStarted: APO.doravetz 6 boss exposure science itime=300 fimg-fvc1n-0002.fits shows no evidence of light artifacts. 00:05Z - Ventilation started. ============================================================================ NIGHT ============================================================================ Cloudy with scattered radar returns, so we aren't opening on time tonight. 04:45Z - Since the clouds seem to only be getting worse, I went down to the telescope to prep for doing FVC rotator scans and found glycol pooled under the telescope. A little while later there was an fps_flow serious alert and a gfa_temp_warning critical alert. I powered-off the FPS and GFAs. The GFAs were ~30C when I powered things off. Note that with the FPS/GFAs powered off there's no temperature telemetry available in the Grafana plots. 06:23Z - I cleaned up the glycol under the telescope as best I could. I didn't see any glycol collected in the counterweight housing (which is what happened after the leak last April) but it looks like the leak is probably in a similar location as last time behind the mirror cell on the BOSS side. I left the telescope at stow with the intention of not disrupting anything until the day staff can look at it and hopefully it'll make tracing the location of the leak easier that way, too. Positioners are home/folded. FPS is powered-off. APOGEE morning cals completed. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0100 0.0090 Axial B: 0.0020 0.0020 Axial C: -0.0300 -0.0290 Trans D: 0.0600 0.0580 Lateral E: 0.3315 0.3315 Lateral F: 0.5355 0.5345 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 0.00 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -0.0300 -0.0310 Axial B: -0.0250 -0.0260 Axial C: -0.0480 -0.0470 Trans D: -0.0180 -0.0270 GALILS Commanded: 4898500 4870400 4964500 -50 -3450 Actual: 4859224 4859192 4859180 -50 -3450 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.99 0.02 -0.01 -0.01 42.86 ---- Telescope Status ---- Status at: 06:41Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 696 696 714 714 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 150 lbs, 19.6 psi Glycol leak, no coolant flow! FPS Status ========== CP10 Loop = 0.10 L/min GFA Loop = 0.01 L/min Supply pressure = 42.0 psi Return pressure = 0.0 psi Chiller Status ============== Set temperarature = 5.5 C Flow = 0.1 Gpm ---- Software Used ---- STUI : 1.7.19 alerts : 2.3.0 boss : 6.0.2-alpha.0 -boss.specMec : 3.0.4 apo : 3.0.1 fliswarm : 0.4.0 jaeger : 1.3.4a0 kaiju : 1.2.4-alpha.0 coordio : 1.6.2a0 fps_calibrati : 2023.01.05 cherno : 0.5.2 astrometrynet : 0.93 hal : 0.6.0b5 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 35 -mcp.azFiduci : v1_147 -mcp.altFiduc : v1_147 -mcp.rotFiduc : v1_147 tcc : 1.4.0 apogee : not available apogeecal : not available apogeeql : v1_7_0dev STUIAdditions : 3.1.18 Boson : 0.3.2 sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : 2.1-1-ge461 PLC : V35 Fiducials : v1_147 idlspec2d : v6_0_9 Kronos : 1.6.1 roboscheduler : 1.3.2 ---- APOGEE Data Summary ---- 60008 morning cals: raw_data: /data/apogee/utr_cdr/60008/apRaw-*.fits archive: /data/apogee/archive/60008/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/60008/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 60008 76 13:02 44460076 Dark 60 B 13.5 ---------- a-b-c - - 60008 77 13:13 44460077 Dark 60 B 13.5 ---------- a-b-c - - 60008 78 13:23 44460078 Dark 60 B 13.5 ---------- a-b-c - - 60008 79 13:25 44460079 QuarFlat 10 B 13.5 ---------- a-b-c - - 60008 80 13:27 44460080 QuarFlat 10 B 13.5 ---------- a-b-c - - 60008 81 13:29 44460081 QuarFlat 10 B 13.5 ---------- a-b-c - - 60008 82 13:31 44460082 Arc-Thar 12 B 13.5 ---------- a-b-c 0.91 - 60008 83 13:38 44460083 Arc-Une 40 B 13.5 ---------- a-b-c - - 60008 84 13:41 44460084 Arc-Thar 12 A 13.0 ---------- a-b-c 0.38 - 60008 85 13:48 44460085 Arc-Une 40 A 13.0 ---------- a-b-c - - 60008 86 13:53 44460086 Dark 30 A 13.0 ---------- a-b-c - - 60008 87 13:59 44460087 InteFlat 30 A 13.0 ---------- a-b-c - - 60008 88 14:04 44460088 InteFlat 30 A 13.0 ---------- a-b-c - - 60008 89 14:09 44460089 InteFlat 30 A 13.0 ---------- a-b-c - - 60008 90 14:15 44460090 Dark 30 A 13.0 ---------- a-b-c - - 60008 91 14:20 44460091 Arc---- 30 A 13.0 ---------- a-b-c - 85 FPI ------------------------------------------------------------------------------ APOGEE data list, mjd=60009, all raw_data: /data/apogee/utr_cdr/60009/apRaw-*.fits archive: /data/apogee/archive/60009/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/60009/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 60009 1 23:14 44470001 Dark 3 A 13.0 ---------- a-b-c - - 60009 2 23:28 44470002 Dark 60 A 13.0 ---------- a-b-c - - 60009 3 23:30 44470003 QuarFlat 10 A 13.0 ---------- a-b-c - - 60009 4 23:32 44470004 Arc-Thar 12 B 13.5 ---------- a-b-c 0.85 - 60009 5 23:39 44470005 Arc-Une 40 B 13.5 ---------- a-b-c - - 60009 6 23:42 44470006 Arc-Thar 12 A 13.0 ---------- a-b-c 0.35 - 60009 7 23:49 44470007 Arc-Une 40 A 13.0 ---------- a-b-c - - 60009 8 00:00 44470008 Dark 60 A 13.0 ---------- a-b-c - - 60009 9 00:05 44470009 Arc---- 30 A 13.0 ---------- a-b-c - 82 FPI 60009 10 06:55 44470010 Dark 60 A 13.0 ---------- a-b-c - - 60009 11 07:06 44470011 Dark 60 A 13.0 ---------- a-b-c - - 60009 12 07:17 44470012 Dark 60 A 13.0 ---------- a-b-c - - 60009 13 07:18 44470013 QuarFlat 10 A 13.0 ---------- a-b-c - - 60009 14 07:20 44470014 QuarFlat 10 A 13.0 ---------- a-b-c - - 60009 15 07:22 44470015 QuarFlat 10 A 13.0 ---------- a-b-c - - 60009 16 07:24 44470016 Arc-Thar 12 B 13.5 ---------- a-b-c 0.89 - 60009 17 07:31 44470017 Arc-Une 40 B 13.5 ---------- a-b-c - - 60009 18 07:34 44470018 Arc-Thar 12 A 13.0 ---------- a-b-c 0.36 - 60009 19 07:41 44470019 Arc-Une 40 A 13.0 ---------- a-b-c - - 60009 20 07:47 44470020 Dark 30 A 13.0 ---------- a-b-c - - 60009 21 07:52 44470021 InteFlat 30 A 13.0 ---------- a-b-c - - 60009 22 07:57 44470022 InteFlat 30 A 13.0 ---------- a-b-c - - 60009 23 08:03 44470023 InteFlat 30 A 13.0 ---------- a-b-c - - 60009 24 08:08 44470024 Dark 30 A 13.0 ---------- a-b-c - - 60009 25 08:14 44470025 Arc---- 30 A 13.0 ---------- a-b-c - 84 FPI ------------------------------------------------------------------------------ num field design config exptype dithpix nread relflux okay dx dy sigx sigy fpi1 fpi2 -------- ------ ------ ------ ------------ ------- ----- ------- ----- ------- ------- ------ ------ ---- ---- 44470001 103229 51305 8953 DARK 13.0 3 0 True -- -- -- -- -- -- 44470002 103229 51305 8953 DARK 13.0 60 -23 True -- -- -- -- -- -- 44470003 103229 51305 8953 QUARTZFLAT 13.0 10 101 True -- -0.36 -- 0.86 -- -- 44470004 103229 51305 8953 ARCLAMP-THAR 13.5 12 103 True 0.81 0.48 1.09 0.88 -- -- 44470005 103229 51305 8953 ARCLAMP-UNE 13.5 40 96 True 0.54 0.31 1.10 0.87 -- -- 44470006 103229 51305 8953 ARCLAMP-THAR 13.0 12 100 True 0.31 0.43 1.06 0.88 -- -- 44470007 -999 -999 8954 ARCLAMP-UNE 13.0 40 97 True 0.08 0.31 1.13 0.91 -- -- 44470008 -999 -999 8954 DARK 13.0 60 5 True -- -- -- -- -- -- 44470009 -999 -999 8954 FPI 13.0 30 153 True 11.92 0.23 1.16 0.87 -- -- 44470010 -999 -999 8954 DARK 13.0 60 -18 True -- -- -- -- -- -- 44470011 -999 -999 8954 DARK 13.0 60 -15 True -- -- -- -- -- -- 44470012 -999 -999 8954 DARK 13.0 60 -8 True -- -- -- -- -- -- 44470013 -999 -999 8954 QUARTZFLAT 13.0 10 101 True -- -0.31 -- 0.87 -- -- 44470014 -999 -999 8954 QUARTZFLAT 13.0 10 101 True -- -0.29 -- 0.86 -- -- 44470015 -999 -999 8954 QUARTZFLAT 13.0 10 101 True -- -0.30 -- 0.86 -- -- 44470016 -999 -999 8954 ARCLAMP-THAR 13.5 12 96 True 0.94 0.52 1.06 0.88 -- -- 44470017 -999 -999 8954 ARCLAMP-UNE 13.5 40 97 True 0.56 0.35 1.11 0.85 -- -- 44470018 -999 -999 8954 ARCLAMP-THAR 13.0 12 90 True 0.30 0.46 1.05 0.89 -- -- 44470019 -999 -999 8954 ARCLAMP-UNE 13.0 40 94 True 0.04 0.32 1.09 0.89 -- -- 44470020 -999 -999 8954 DARK 13.0 30 -3 False -- -- -- -- -- -- 44470021 -999 -999 8954 INTERNALFLAT 13.0 30 99 True -- -- -- -- -- -- 44470022 -999 -999 8954 INTERNALFLAT 13.0 30 100 True -- -- -- -- -- -- 44470023 -999 -999 8954 INTERNALFLAT 13.0 30 100 True -- -- -- -- -- -- 44470024 -999 -999 8954 DARK 13.0 30 247 True -- -- -- -- -- -- 44470025 -999 -999 8954 FPI 13.0 30 153 True 11.90 0.30 1.16 0.87 -- -- num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 mtp10 -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----- 44470003 QUARTZFLAT 10 70 76 83 93 93 88 89 91 90 84 44470013 QUARTZFLAT 10 70 76 83 93 93 87 89 91 90 84 44470014 QUARTZFLAT 10 73 79 83 94 94 88 89 92 91 86 44470015 QUARTZFLAT 10 73 80 84 94 94 88 89 92 91 86 ---- BOSS Data Summary ---- Reading FITS from /data/spectro//60009 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 23:14 b1-00353346 ----------- ---- bia Clo 0.0 881 872 896 2 23:14 r1-00353346 ----------- ---- bia Clo 0.0 881 872 896 3 23:47 r1-00353347 -999--999 8954 sci Out 300.1 881 872 896 4 23:47 b1-00353347 -999--999 8954 sci Out 300.1 881 872 896 5 00:06 b1-00353348 -999--999 8954 arc Lef 4.1 881 872 896 HART 6 00:06 r1-00353348 -999--999 8954 arc Lef 4.1 881 872 896 HART 7 00:07 b1-00353349 -999--999 8954 arc Rig 4.1 881 872 896 HART 8 00:07 r1-00353349 -999--999 8954 arc Rig 4.1 881 872 896 HART 9 00:12 r1-00353350 -999--999 8954 arc Lef 4.1 1065 1054 1080 HART 10 00:12 b1-00353350 -999--999 8954 arc Lef 4.1 1065 1054 1080 HART 11 00:12 r1-00353351 -999--999 8954 arc Rig 4.1 1065 1054 1080 HART 12 00:12 b1-00353351 -999--999 8954 arc Rig 4.1 1065 1054 1080 HART 13 00:16 b1-00353352 -999--999 8954 arc Out 4.1 1249 1238 1264 14 00:16 r1-00353352 -999--999 8954 arc Out 4.1 1249 1238 1264 15 00:20 b1-00353353 -999--999 8954 fla Out 25.1 1249 1238 1264 16 00:20 r1-00353353 -999--999 8954 fla Out 25.1 1249 1238 1264 ---- BOSS Spectrographs Collimation ---- Afternoon focus check ===================== 00:12:12 BOSS expNum=353350, expTime= 4.0s 00:12:53 BOSS expNum=353351, expTime= 4.0s 00:13:34 sp1.motor.move= 184, 184, 184 00:13:34 Hartmann collimate output sp1: offset: r= -659 (In Focus); b= -3.2 (In Focus) sp1: pred. move: spAverageMove= 184 sp1: pred. resid: r= -843, b= -2.6, txt= OK, spTemp = 4.7 00:13:31 APO.doravetz 13 hartmann i r1MeanOffset=-0.10,"In Focus" 00:13:31 APO.doravetz 13 hartmann i b1MeanOffset=0.16,"In Focus" ---- Weather Log ---- Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma 19:00 -100.0 -100.0 10.9 252.3 649 19:15 -100.0 -100.0 4.2 231.0 755 19:30 -100.0 -100.0 22.5 244.3 812 19:45 -100.0 -100.0 11.7 260.6 781 20:00 -100.0 -100.0 25.1 261.5 732 20:15 -100.0 -100.0 18.1 254.1 754 20:30 -100.0 -100.0 15.9 255.6 758 20:45 -100.0 -100.0 24.3 253.5 737 21:00 -100.0 -100.0 23.4 267.8 697 21:15 -100.0 -100.0 19.1 256.3 729 21:30 -100.0 -100.0 24.6 266.8 694 21:45 -100.0 -100.0 20.1 265.9 750 22:00 -100.0 -100.0 18.8 264.8 791 22:15 -100.0 -100.0 25.0 261.0 715 22:30 -100.0 -100.0 17.6 261.0 636 22:45 -100.0 -100.0 23.5 240.8 680 23:00 -100.0 -100.0 22.4 266.1 681 23:15 -100.0 -100.0 21.7 267.3 752 23:30 -100.0 -100.0 23.0 264.8 649 23:45 -100.0 -100.0 19.6 265.7 645 00:00 -100.0 -100.0 18.9 262.3 679 00:15 -100.0 -100.0 22.2 252.1 669 00:30 -100.0 -100.0 15.9 259.3 681 00:45 -100.0 -100.0 15.3 248.6 694 01:00 -100.0 -100.0 17.3 256.9 650 01:15 -100.0 -100.0 22.0 266.7 725 01:30 -100.0 -100.0 21.0 267.1 665 01:45 -100.0 -100.0 23.5 265.8 715 02:00 -100.0 -100.0 22.5 272.5 691 02:15 -100.0 -100.0 24.7 269.2 650 02:30 -100.0 -100.0 22.8 266.5 699 02:45 -100.0 -100.0 19.9 270.1 550 03:00 -100.0 -100.0 23.7 269.1 634 03:15 -100.0 -100.0 21.2 267.5 585 385 80.54 03:30 -100.0 -100.0 24.2 266.8 587 372 48.80 03:45 -100.0 -100.0 20.0 266.3 630 341 54.71 04:00 -100.0 -100.0 16.1 268.3 606 361 82.84 04:15 -100.0 -100.0 17.1 272.8 553 368 70.48 04:30 -100.0 -100.0 23.5 271.2 623 361 52.86 04:45 -100.0 -100.0 14.7 266.3 579 367 58.73 05:00 -100.0 -100.0 14.3 266.7 665 353 39.98 05:15 -100.0 -100.0 15.9 265.6 685 356 72.18 05:30 -100.0 -100.0 12.2 266.0 646 358 83.91 05:45 -100.0 -100.0 17.9 266.3 666 368 51.99 06:00 -100.0 -100.0 16.6 257.3 651 360 61.52 06:15 -100.0 -100.0 13.3 263.8 663 356 54.96 06:30 -100.0 -100.0 13.5 270.1 702 362 64.87 06:45 -100.0 -100.0 16.1 267.9 677 368 44.73 07:00 -100.0 -100.0 17.7 287.8 665 368 66.13 07:15 -100.0 -100.0 15.5 268.8 662 374 50.56 07:30 -100.0 -100.0 15.3 267.1 643 371 60.53 07:45 -100.0 -100.0 15.8 267.3 640 365 72.45 08:00 -100.0 -100.0 13.3 268.4 652 371 33.25 08:15 -100.0 -100.0 12.1 266.0 639 376 78.65 08:30 -100.0 -100.0 15.1 268.6 659 349 69.52 08:45 -100.0 -100.0 15.4 297.1 577 350 68.15 09:00 -100.0 -100.0 15.1 303.8 637 361 48.84 09:15 -100.0 -100.0 12.5 324.3 670 365 57.38 09:30 -100.0 -100.0 16.5 298.6 721 358 62.08 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 60009 Fill night time in Hours below *=*=*= Total bright time : 10.3 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 3.1 *=*=*= Bright time lost to technical issues : 7.2 *=*=*= Total dark time : 0 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 0 *=*=*= Dark time lost to technical issues : 0