Subject: 2.5m obslog 20240108 (SJD 60318) From: dmbiz@apo.nmsu.edu Date: Tue, 9 Jan 2024 12:03:52 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday January 08, 2024 (SJD 60318) ---- Observing Team ---- Dmitry Bizyaev John Wilson (help with FPI troubleshooting in the evening) ---- Observing Plan ---- Mon 08 Jan 2024 60318 0.07 01:10 01:25 12:54 01:35 13:05 --- dark (11.5 h) plus 0.3 h for 12 deg design ---- Observing Summary ---- Clear night with high humidity, temperature below -12C, high wind gusts and blowing snow. The entire night was lost to the weather. ---- Engineering Issues ---- Thank you for cleaning MTPs! ---- Software Issues ---- The TIMS process that manages the FPI shutter failed last morning and made the FPI shutter commands irresponsible. The process was restarted during the evening checkout. New FPI frames look OK. ---- Observing Log ---- Afternoon --------- All systems feel good except the FPI shutter. See below. 00:47:01 Hartmann collimate output sp1: offset: r= -1979 (Out of focus); b= 3.8 (In Focus) sp1: pred. move: spAverageMove= -1600 sp1: pred. resid: r= -379, b= -1.2, txt= OK, spTemp = -0.3 The blue ring is kept untouched. The shutter shows the "?" status. Restarting actor doesn't help. Issuing direct command from the apogee-ics machine selected the following trouble: The response 'Server says "message: Apogeefpi_notconnected"' is indicative of a shutter control problem. Logs on the 192.168.2.160 machine: 2024-01-09T00:38:38.515216 [__main__.TIMSServerFactory] New connection from IPv4Address(type='TCP', host='127.0.0.1', port=60774) 2024-01-09T00:38:38.515574 [TIMSServerProtocol,7325,127.0.0.1] Client Registered: Cli3082370 2024-01-09T00:38:38.515671 [TIMSServerProtocol,7325,127.0.0.1] Received: Sender:Cli3082370, Subsystem:Apogeefpi, Command:['shutters', 'high1'] 2024-01-09T00:38:38.515749 [TIMSServerProtocol,7325,127.0.0.1] Sending Cli3082370: "Server Cli3082370 message:Apogeefpi_notconnected" 2024-01-09T00:38:38.516468 [TIMSServerProtocol,7325,127.0.0.1] Unregistering Client: Cli3082370 Apparently, th ecloent in the screen session was killed: Sending Cli2541355: Apogeefpi Cli2541355 message:shutters:low1:ok Killed Restarted the client: ... OK: Connection established to shutters:/dev/dlpio8 Connected to: gammaspce ConnectionMadeCommand: 01 OK: Connection established to gammaspce:/dev/gammaspce Ventilation postponed due to blowing snow, low temperature (-10.9C), and approaching radar blob. For unknown reason during the first set of GFA biases GFA 1 and 5 didn't expose all images. No error messages and normal respond to "status". Repeated the darks and biaes, and is worked well for all GFA cameras. 23:42:20 boss exposure 0 sec file=0363719 bias 23:43:00 apogee exposure file= apRaw-47560001 3 A ArcLamp ======================================================= =====>>>>> 00:02:26 Config Loaded: -999--999-12368 ======================================================= 00:03:11 FVC exposure /data/fcam/60318/proc-fimg-fvc1n-0001.fits 00:03:03 fvc_perc_90= 71.7 fvc_percent_reached= 1.0 fvc_rms= 45.99 00:05:07 apogee exposure file= apRaw-47560002 60 A Dark 00:16:06 apogee exposure file= apRaw-47560003 10 A QuartzFlat 00:18:01 apogee exposure file= apRaw-47560004 12 B ArcLamp 00:20:14 apogee exposure file= apRaw-47560005 40 B ArcLamp 00:27:31 apogee exposure file= apRaw-47560006 12 A ArcLamp 00:29:44 apogee exposure file= apRaw-47560007 40 A ArcLamp 00:37:09 apogee exposure file= apRaw-47560008 60 A Dark 00:48:43 apogee exposure file= apRaw-47560009 30 A ArcLamp 00:45:00 boss exposure 4 sec file=0363720 arc 00:45:40 boss exposure 4 sec file=0363721 arc 00:46:56 Hartmann: offset r= -1979 (Out of focus) b = 3.8 (In Focus) pred_move = -1600 pred_resid r= -379 b= -1.2 cmnt=OK ----- Night ----- Not opening: low temperatures, low dew point difference, high wind gusts, and blowing snow on site. The sky is clear. 12:01Z no improvements for the air temperature and blowing snow. Finishing night. [ (47560001, 3, 3, '/data/apogee/raw/60318/apRaw-47560001-001.fits', 'ARCLAMP', 'None', '', '12367', '-999', '-999', 32.041, '2024-01-08T23:43:00.945', 'Podium', 'Closed', 'Closed', 60318, 'APO', 12.994)] problem with exposure: 47560001 num field design config exptype dithpix nread relflux okay dx dy sigx sigy fpi1 fpi2 -------- ----- ------ ------ ------------ ------- ----- ------- ----- ------- ------- ------ ------ ---- ---- 47560001 -999 -999 12367 ARCLAMP-None 13.0 3 0 False 0.00 0.00 0.00 0.00 0 0 47560002 -999 -999 12368 DARK 13.0 60 -22 True -- -- -- -- -- -- 47560003 -999 -999 12368 QUARTZFLAT 13.0 10 106 True -- -0.28 -- 0.86 -- -- 47560004 -999 -999 12368 ARCLAMP-THAR 13.5 12 89 True 1.37 0.50 1.06 0.89 -- -- 47560005 -999 -999 12368 ARCLAMP-UNE 13.5 40 101 True 0.99 0.32 1.10 0.88 -- -- 47560006 -999 -999 12368 ARCLAMP-THAR 13.0 12 95 True 0.88 0.55 1.07 0.89 -- -- 47560007 -999 -999 12368 ARCLAMP-UNE 13.0 40 97 True 0.48 0.37 1.10 0.86 -- -- 47560008 -999 -999 12368 DARK 13.0 60 32 True -- -- -- -- -- -- 47560009 -999 -999 12368 FPI 13.0 30 141 True -12.28 0.31 1.16 0.86 -- -- num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 mtp10 -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----- 47560003 QUARTZFLAT 10 67 71 81 90 92 85 87 89 88 82 ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.5030 0.5050 Axial B: -0.2030 -0.1950 Axial C: -0.1220 -0.1100 Trans D: 1.9020 1.9140 Lateral E: 0.3395 0.3395 Lateral F: 0.5465 0.5460 GALILS Commanded: 5650 1750 8800 -4500 -3400 -3100 Actual: 5650 1750 8800 -4500 -3400 -3100 SETMIR VALUES PrimDesOrient: 0.0 44.5 -7.8 -427.0 -2433.9 PrimOrient: 1.07 44.57 -7.9 -426.41 -2433.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 0.00 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -1.0500 -1.0360 Axial B: -1.0640 -1.0720 Axial C: -1.0970 -1.1070 Trans D: -0.0210 -0.0470 GALILS Commanded: 5918000 5843950 5976600 -1450 -7300 Actual: 5883277 5811681 5906269 -1450 -7300 SETMIR VALUES SecDesOrient: 4657.0 -24.5 -17.5 18.0 125.41 SecOrient: 4656.92 -24.58 -17.38 19.01 125.31 ---- Telescope Status ---- Not opened. No interlocks bypassed Status at: 12:02Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 695 / 714 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 151 lbs, 22 psi FPS Status ========== CP10 Loop = 7.37 L/min GFA Loop = 2.23 L/min Supply pressure = 43.1 psi Return pressure = 28.9 psi Chiller Status ============== Set temperarature = 1.1 C Flow = 5.0 Gpm ---- Software Used ---- STUI : 1.7.19 alerts : 2.3.0 boss : 6.0.2 -boss.specMec : 3.0.4 apo : 4.0.0 fliswarm : 0.5.0 jaeger : 1.6.1a0 kaiju : 1.3.1 coordio : 1.9.0 fps_calibrati : 2023.12.08+eebcf2b cherno : 0.6.3 astrometrynet : 0.93 hal : 0.7.3 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1-1-ge461 -mcp.plc : 35 -mcp.azFiduci : v1_149 -mcp.altFiduc : v1_149 -mcp.rotFiduc : v1_149 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.20 Boson : 0.3.2 --------------Other Versions-------------- sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : 2.1-1-ge461 idlspec2d : v6_1_0 Kronos : 1.6.2a roboscheduler : 1.4.0 /dev/sdb1 2.0T 304G 1.8T 15% /home /dev/sdd1 22T 17T 4.7T 78% /data ---- APOGEE Data Summary ---- Last morning all calibrations were OK except the FPI: its shutter's process failed. It was fixed this evening, see log. 60317 13 11:50 47550013 QuarFlat 10 A 13.0 ---------- a-b-c - - 60317 14 11:52 47550014 QuarFlat 10 A 13.0 ---------- a-b-c - - 60317 15 11:54 47550015 QuarFlat 10 A 13.0 ---------- a-b-c - - 60317 16 11:56 47550016 Arc-Thar 12 B 13.5 ---------- a-b-c 1.29 - 60317 17 12:03 47550017 Arc-Une 40 B 13.5 ---------- a-b-c - - 60317 18 12:06 47550018 Arc-Thar 12 A 13.0 ---------- a-b-c 0.82 - 60317 19 12:13 47550019 Arc-Une 40 A 13.0 ---------- a-b-c - - 60317 20 12:18 47550020 Dark 30 A 13.0 ---------- a-b-c - - 60317 21 12:24 47550021 InteFlat 30 A 13.0 ---------- a-b-c - - 60317 22 12:29 47550022 InteFlat 30 A 13.0 ---------- a-b-c - - 60317 23 12:34 47550023 InteFlat 30 A 13.0 ---------- a-b-c - - 60317 24 12:40 47550024 Dark 30 A 13.0 ---------- a-b-c - - 60317 25 12:46 47550025 Arc---- 30 A 13.0 ---------- a-b-c - 3 FPI <- BAD !!! APOGEE data list, mjd=60318, all raw_data: /data/apogee/utr_cdr/60318/apRaw-*.fits archive: /data/apogee/archive/60318/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/60318/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 60318 1 23:43 47560001 Arc---- 3 A 13.0 ---------- a-b-c - -7 test 60318 2 00:15 47560002 Dark 60 A 13.0 ---------- a-b-c - - 60318 3 00:17 47560003 QuarFlat 10 A 13.0 ---------- a-b-c - - 60318 4 00:19 47560004 Arc-Thar 12 B 13.5 ---------- a-b-c 1.28 - 60318 5 00:26 47560005 Arc-Une 40 B 13.5 ---------- a-b-c - - 60318 6 00:29 47560006 Arc-Thar 12 A 13.0 ---------- a-b-c 0.82 - 60318 7 00:36 47560007 Arc-Une 40 A 13.0 ---------- a-b-c - - 60318 8 00:47 47560008 Dark 60 A 13.0 ---------- a-b-c - - 60318 9 00:53 47560009 Arc---- 30 A 13.0 ---------- a-b-c - 75 FPI ------------------------------------------------------------------------------ ---- BOSS Data Summary ---- Reading FITS from /data/spectro//60318 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 23:42 r1-00363719 ----------- ---- bia Clo 0.0 22834 23772 21829 2 23:42 b1-00363719 ----------- ---- bia Clo 0.0 22834 23772 21829 3 00:45 r1-00363720 -999--999 12368 arc Lef 4.1 22834 23772 21829 HART 4 00:45 b1-00363720 -999--999 12368 arc Lef 4.1 22834 23772 21829 HART 5 00:46 r1-00363721 -999--999 12368 arc Rig 4.1 22834 23772 21829 HART 6 00:46 b1-00363721 -999--999 12368 arc Rig 4.1 22834 23772 21829 HART ---- BOSS Spectrographs Collimation ---- Time TSP1 R1 B1 Move1 B1Resid ============================================= 00:46:50 -0.6 -1979 3.8 -1600 -1.2 ---- Weather Log ---- Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma 19:00 -11.4 -12.6 12.0 298.9 -5.4219:15 -11.3 -12.6 12.2 273.8 -5.4419:30 -11.0 -11.8 20.3 299.2 -5.5019:45 -10.9 -11.9 12.2 287.2 -5.5620:00 -10.7 -11.7 13.6 283.0 -5.6120:15 -11.1 -12.2 16.4 323.4 -5.6120:30 -10.7 -11.7 14.4 283.8 -5.6120:45 -10.5 -11.4 17.8 321.4 -5.6421:00 -10.8 -11.9 19.2 304.1 -5.6221:15 -10.8 -11.9 15.6 310.5 -5.6921:30 -10.8 -11.9 12.6 307.2 -5.6621:45 -10.8 -11.8 23.4 314.1 -5.7522:00 -10.6 -11.8 18.8 304.8 -5.7222:15 -10.4 -11.6 22.8 295.3 -5.7822:30 -10.2 -11.2 11.8 340.0 -5.7522:45 -10.3 -11.9 10.2 323.8 -5.7723:00 -10.2 -11.4 2.1 288.3 -5.8023:15 -10.7 -11.9 11.2 331.2 -5.7723:30 -10.6 -11.6 14.7 337.9 -5.8023:45 -10.4 -11.8 11.4 327.5 -5.8100:00 -10.8 -12.6 5.0 315.1 -5.8300:15 -10.9 -12.6 15.1 0.1 -5.8400:30 -11.1 -12.9 7.9 355.5 -5.8800:45 -11.4 -13.1 6.8 355.7 -5.8801:00 -11.3 -13.3 15.1 8.8 -5.9101:15 -11.4 -13.3 6.2 8.5 -5.92 -11.1 16.8 49.4 01:30 -11.3 -13.6 11.2 1.0 -5.98 -10.9 16.8 49.4 01:45 -11.2 -13.6 14.6 16.3 -6.02 -10.9 16.8 49.4 02:00 -11.3 -13.6 12.4 27.0 -6.03 -11.0 16.8 49.4 02:15 -11.3 -13.6 10.2 359.9 -6.05 -11.0 16.8 49.4 02:30 -11.3 -13.7 6.3 11.8 -6.08 -11.2 16.8 49.4 02:45 -11.3 -13.8 14.8 31.0 -6.12 -11.3 16.8 49.4 03:00 -11.5 -13.9 15.9 11.0 -6.17 -11.4 16.8 49.4 03:15 -11.2 -14.2 11.4 42.4 -6.25 -11.3 16.8 49.4 03:30 -11.1 -13.9 19.4 15.2 -6.27 -11.2 16.8 49.4 03:45 -11.4 -14.3 15.5 29.8 -6.30 -11.3 16.8 49.4 04:00 -11.3 -14.3 14.3 0.1 -6.33 -11.4 16.8 49.4 04:15 -11.3 -14.6 10.0 21.1 -6.39 -11.4 16.8 49.4 04:30 -11.8 -14.2 12.1 10.6 -6.39 -11.8 16.8 49.4 04:45 -12.1 -13.9 8.8 28.0 -6.44 -12.0 16.8 49.4 05:00 -12.1 -13.9 11.7 25.0 -6.52 -12.2 16.8 49.4 05:15 -12.1 -13.9 22.1 23.3 -6.56 -12.2 16.8 49.4 05:30 -12.1 -14.0 19.9 4.9 -6.67 -12.2 16.8 49.4 05:45 -12.4 -13.9 20.1 29.3 -6.67 -12.4 16.8 49.4 06:00 -12.6 -13.9 26.8 15.0 -6.73 -12.5 16.8 49.4 06:15 -12.3 -13.9 1.7 21.7 -6.78 -12.3 16.8 49.4 06:30 -12.3 -14.0 18.9 25.3 -6.80 -12.3 16.8 49.4 06:45 -12.5 -14.0 14.6 6.7 -6.86 -12.5 16.8 49.4 07:00 -12.4 -14.3 15.1 7.9 -6.95 -12.4 16.8 49.4 07:15 -12.1 -14.9 25.9 351.0 -6.94 -12.2 16.8 49.4 07:30 -12.3 -14.7 12.4 20.6 -6.98 -12.3 16.8 49.4 07:45 -12.4 -14.8 21.2 2.6 -7.03 -12.4 16.8 49.4 08:00 -12.6 -14.6 25.6 23.8 -7.11 -12.7 16.8 49.4 08:15 -12.6 -14.8 17.2 16.2 -7.14 -12.7 16.8 49.4 08:30 -12.7 -14.9 24.0 21.2 -7.22 -12.7 16.8 49.4 08:45 -12.8 -15.1 17.9 5.3 -7.27 -12.8 16.8 49.4 09:00 -12.6 -15.6 26.3 14.5 -7.28 -12.7 16.8 49.4 09:15 -12.3 -16.1 29.2 19.9 -7.36 -12.7 16.8 49.4 09:30 -12.6 -16.4 18.0 25.7 -7.47 -12.6 16.8 49.4 09:45 -12.6 -16.3 25.6 33.5 -7.47 -12.7 16.8 49.4 10:00 -12.7 -16.3 15.6 13.7 -7.53 -12.9 16.8 49.4 10:15 -13.0 -16.3 26.6 9.1 -7.56 -13.0 16.8 49.4 10:30 -12.4 -16.9 22.8 3.3 -7.64 -12.8 16.8 49.4 10:45 -12.7 -17.2 13.7 358.9 -7.70 -12.9 16.8 49.4 11:00 -12.7 -17.4 27.4 28.2 -7.73 -13.0 16.8 49.4 11:15 -12.7 -17.8 14.9 27.8 -7.77 -13.0 16.8 49.4 11:30 -12.4 -18.5 19.4 2.1 -7.84 -12.6 16.8 49.4 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 60318 Fill night time in Hours below *=*=*= Total bright time : 0.3 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 0.3 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 11.5 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 11.5 *=*=*= Dark time lost to technical issues : 0