Subject: 2.5m obslog 20240627 (SJD 60489) From: dmbiz@apo.nmsu.edu Date: Fri, 28 Jun 2024 10:49:06 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday June 27, 2024 (SJD 60489) ---- Observing Team ---- Dmitry Bizyaev Jose Sanchez-Gallego - fixing NUC3 GFA computer issues in the evening, remotely. ---- Observing Plan ---- Tue 25 Jun 2024 60487 0.78 03:18 03:37 10:35 14:53 21:51 --- dark until 05:34 ( 2.0 h), then bright ( 6.0 h, w/1.0 h for 8 deg design) ---- Observing Summary ---- All night was lost to clouds and accidental rain. Notes ----- - GFA NUC3 computer failed. Jose applied a hot fix. Now GFA3's USB cable is connected to NUC2, apparently until the supper shutdown. - FVC flats were taken in a set of Rot and AZ positions. The AZ ones (Alt = 90) look useless. We should try a sky FVC flat at sunset tomorrow, if weather permits. - I cannot find any morning APOGEE calibrations from yesterday. ---- Engineering Issues ---- During the afternoon checkout I was unable to wake up GFA-3 camera. It complained about NUC-3 computer. Sent email to Jose. Jose suspects that the Nuc-3 computer has died. We have a spare one, but since the summer shutdown shutdown is nearby, it is better to replace it that time. I opened IEB and connected the GFA-3's USB cable to Nuc-2. Jose made reconfigurations. GFA-3 cooled down and took normal biases and darks. ---- Observing Log ---- Afternoon --------- All systems checked, no problems found, except for the GFA 3 camera problem, see below. BOSS collimation looks good, not adjusted: 02:33:59 Hartmann collimate output sp1: offset: r= -1319 (Out of focus); b= -0.2 (In Focus) sp1: pred. move: spAverageMove= -627 sp1: pred. resid: r= -691, b= -2.2, txt= OK, spTemp = 10.5 The jaeger configuration random --danger didn[t work from the first attempt: 02:03:54 .jaeger 0 jaeger e text="A collision was detected in positioner 995. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 02:03:54 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 02:03:56 .jaeger 0 jaeger e locked=T; locked_by=995; locked_axes=alpha; locked_alpha=9.986621215939522; locked_beta=170.00630378723145 unlocked - explode 5 - unwind - repeat: it worked OK for the second time. GFAs 2 and 3 didn't want to connect from the first "reconnect" command. I issued reconnect force for 2 and 3 individually, and #2 got on. GFA 3 doesn't want to wake up: 02:18:39 APO.dmbiz 70 fliswarm f error="Command 'reconnect -n gfa3 --force' failed because of an uncaught error 'ConnectTimeout: HTTPConnectionPool(host='sdss-gfa3', port=2375): Max retries exceeded with url: /v1.41/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22name%22%3A+%5B%22flicamera-gfa3%22%5D%2C+%22status%22%3A+%5B%22running%22%5D%7D (Caused by ConnectTimeoutError(, 'Connection to sdss-gfa3 timed out. (connect timeout=3)'))'. See traceback in the log for more information." 02:18:39 CmdDone: APO.dmbiz 70 fliswarm reconnect -n gfa3 --force Also, 02:19:01 APO.dmbiz 71 fliswarm w text="Node gfa3 is not pinging back or the Docker daemon is not running. Try rebooting the computer." NUC 3 reboot command is issued. Upon checking status it shows that the problem comes back: 02:19:01 APO.dmbiz 71 fliswarm w text="Node gfa3 is not pinging back or the Docker daemon is not running. Try rebooting the computer." 02:22:29 APO.dmbiz 84 fliswarm w text="Failed connecting to gfa3." 02:22:56 APO.dmbiz 85 fliswarm w text="Node gfa3 is not pinging back or the Docker daemon is not running. Try rebooting the computer." and also 03:08:35 APO.dmbiz 116 fliswarm w text="Node sdss-gfa3 is not pinging back." Sent email to Jose. Ventilation postponed because of thick clouds and some rain drops on site. 01:58:17 apogee exposure file= apRaw-49270001 3 A ArcLamp 02:00:00 apogee exposure file= apRaw-49270002 60 A Dark ======================================================= =====>>>>> 02:00:49 Config Loaded: -999--999-15041 ======================================================= 02:01:44 FVC exposure /data/fcam/60489/proc-fimg-fvc1n-0001.fits 02:01:37 fvc_perc_90= 60.1 fvc_percent_reached= 1.6 fvc_rms= 40.48 02:10:59 apogee exposure file= apRaw-49270003 10 A QuartzFlat 02:12:53 apogee exposure file= apRaw-49270004 12 B ArcLamp 02:15:06 apogee exposure file= apRaw-49270005 40 B ArcLamp 02:22:24 apogee exposure file= apRaw-49270006 12 A ArcLamp 02:24:36 apogee exposure file= apRaw-49270007 40 A ArcLamp 02:32:02 apogee exposure file= apRaw-49270008 60 A Dark 02:31:59 boss exposure 4 sec file=0370638 arc 02:32:39 boss exposure 4 sec file=0370639 arc 02:33:56 Hartmann: offset r= -1319 (Out of focus) b = -0.2 (In Focus) pred_move = -627 pred_resid r= -691 b= -2.2 cmnt=OK 02:35:16 boss exposure 4 sec file=0370640 arc 02:37:19 boss exposure 25 sec file=0370641 flat 02:43:01 apogee exposure file= apRaw-49270009 30 A ArcLamp 02:48:24 apogee exposure file= apRaw-49270010 30 B ArcLamp ----- Night ----- Opening delayed because of thick clouds with radars. Jose answered and suspected that the Nuc-3 computer has died. We have a spare one, but since the shutdown is nearby, it is better to replace it that time. At the moment, Jose can connect GFA-3 to another Nuc. I opened IEB and connected the GFA-3's USB cable to Nuc-2. Jose made a reconfiguration. GFA-3 cooled down and took normal biases and darks. By Kaike's request, FVC flat fields are taken at different rotator positions. The command is fliswarm talk -n fvc expose —flat —count 1 600 The enclosure light are on, FF lamps are off. Az=121, Alt=30, Rot = 0 fimg-fvc1n-0002.fits Az=121, Alt=30, Rot = 90 fimg-fvc1n-0003.fits Az=121, Alt=30, Rot = 180 fimg-fvc1n-0004.fits Az=121, Alt=30, Rot = 270 fimg-fvc1n-0005.fits Different AZ positions: Az=301, Alt=90, Rot = 0 fimg-fvc1n-0006.fits Az=211, Alt=90, Rot = 0 fimg-fvc1n-0007.fits Az=121, Alt=90, Rot = 0 fimg-fvc1n-0008.fits Az= 31, Alt=90, Rot = 0 fimg-fvc1n-0009.fits 05:15Z rain started. 10:45Z giving up. The sky is clear in the zenith, but we are still surrounded by clouds with lightning flashes. Calibrations started. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.2020 0.2010 Axial B: -0.0010 -0.0010 Axial C: 0.0650 0.0650 Trans D: -0.0130 -0.0120 Lateral E: 0.3380 0.3380 Lateral F: 0.5455 0.5455 GALILS Commanded: 9800 -300 6750 19150 -3550 -3250 Actual: 9800 -300 6750 19150 -3550 -3250 SETMIR VALUES PrimDesOrient: 0.0 -18.5 -7.8 -427.0 -998.0 PrimOrient: 0.0 -18.23 -7.9 -427.17 -999.0 SECONDARY: ---------- Focus: 0.0000 Air temp: 0.00 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -1.0410 -1.0410 Axial B: 0.2170 0.2170 Axial C: -1.1100 -1.1110 Trans D: -0.0120 -0.0010 GALILS Commanded: 5887300 5813600 5950900 -1500 -7300 Actual: 5883662 5811662 5906238 -1500 -7300 SETMIR VALUES SecDesOrient: 4657.0 -24.5 -17.5 18.0 125.41 SecOrient: 4657.02 -24.52 -17.51 18.31 125.9 ---- Telescope Status ---- Not opened. No interlocks bypassed Status at: 10:47Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 692 / 712 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 215 lbs, 19 psi FPS Status ========== CP10 Loop = 7.60 L/min GFA Loop = 2.29 L/min Supply pressure = 42.6 psi Return pressure = 29.1 psi Chiller Status ============== Set temperarature = 18.0 C Flow = 4.9 Gpm ---- Software Used ---- STUI : 1.7.19 alerts : 2.3.0 boss : 6.0.2 -boss.specMec : 3.0.4 apo : 4.1.0 fliswarm : 0.5.1a0 jaeger : 1.7.5 kaiju : 1.4.0 coordio : 1.11.2 fps_calibrati : 2024.04.01+91998f8 cherno : 1.0.0 astrometrynet : 0.93 hal : 1.3.5 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1-1-ge461 -mcp.plc : 35 -mcp.azFiduci : v1_149 -mcp.altFiduc : v1_149 -mcp.rotFiduc : v1_149 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : FAILED Boson : 0.4.9 --------------Other Versions-------------- sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : 2.1-1-ge461 idlspec2d : v6_1_3 Kronos : 1.7.0 roboscheduler : 1.4.0 /dev/sdb1 2.0T 464G 1.6T 23% /home /dev/sdd1 22T 15T 6.1T 71% /data ---- APOGEE Data Summary ---- There are no morning APOGEE calibrations taken last day. APOGEE data list, mjd=60489, all raw_data: /data/apogee/utr_cdr/60489/apRaw-*.fits archive: /data/apogee/archive/60489/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/60489/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 60489 1 01:58 49270001 Arc---- 3 A 13.0 ---------- a-b-c - -1 test 60489 2 02:10 49270002 Dark 60 A 13.0 ---------- a-b-c - - 60489 3 02:12 49270003 QuarFlat 10 A 13.0 ---------- a-b-c - - 60489 4 02:14 49270004 Arc-Thar 12 B 13.5 ---------- a-b-c 0.98 - 60489 5 02:21 49270005 Arc-Une 40 B 13.5 ---------- a-b-c - - 60489 6 02:24 49270006 Arc-Thar 12 A 13.0 ---------- a-b-c 0.42 - 60489 7 02:31 49270007 Arc-Une 40 A 13.0 ---------- a-b-c - - 60489 8 02:42 49270008 Dark 60 A 13.0 ---------- a-b-c - - 60489 9 02:47 49270009 Arc---- 30 A 13.0 ---------- a-b-c - 79 FPI A 60489 10 02:53 49270010 Arc---- 30 B 13.5 ---------- a-b-c - 80 FPI B ------------------------------------------------------------------------------ ---- BOSS Data Summary ---- Reading FITS from /data/spectro//60489 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 02:32 b1-00370638 -999--999 15041 arc Lef 4.1 6757 6762 6752 HART 2 02:32 r1-00370638 -999--999 15041 arc Lef 4.1 6757 6762 6752 HART 3 02:33 b1-00370639 -999--999 15041 arc Rig 4.1 6757 6762 6752 HART 4 02:33 r1-00370639 -999--999 15041 arc Rig 4.1 6757 6762 6752 HART 5 02:35 r1-00370640 -999--999 15041 arc Out 4.1 6130 6133 6124 6 02:35 b1-00370640 -999--999 15041 arc Out 4.1 6130 6133 6124 7 02:37 b1-00370641 -999--999 15041 fla Out 25.1 6130 6133 6124 8 02:37 r1-00370641 -999--999 15041 fla Out 25.1 6130 6133 6124 ---- BOSS Spectrographs Collimation ---- Time TSP1 R1 B1 Move1 B1Resid ============================================= 02:33:50 9.9 -1319 -0.2 -627 -2.2 ---- Weather Log ---- Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma 18:00 23.0 13.2 8.4 222.3 1642 23.8 16.8 49.4 18:15 21.5 13.3 9.3 251.5 dpDep 24.0 data 16.8 18:30 19.4 13.1 8.4 269.8 1402 23.1 data 16.8 18:45 20.0 13.1 9.7 266.4 1519 22.1 data 16.8 19:00 21.9 13.2 5.2 255.3 1441 22.6 data 16.8 19:15 21.4 13.1 13.4 249.6 1297 23.3 data 16.8 19:30 21.8 12.9 6.4 221.0 1399 23.6 data 16.8 19:45 22.0 12.9 15.7 235.0 1313 23.7 data 16.8 20:00 21.3 12.4 4.6 237.9 1095 24.2 data 16.8 20:15 20.5 11.4 4.9 261.5 1078 23.8 data 16.8 20:30 23.3 9.9 18.3 244.5 960 24.0 data 16.8 20:45 21.6 9.8 13.8 267.3 1066 23.4 data 16.8 21:00 23.5 9.5 13.8 263.1 961 24.5 data 16.8 21:15 22.5 10.1 13.7 241.3 917 23.1 data 16.8 21:30 20.2 11.2 8.7 262.5 980 22.4 data 16.8 21:45 19.8 11.3 11.8 259.5 1026 21.1 data 16.8 22:00 16.7 10.9 11.6 136.5 1140 16.7 data 16.8 22:15 15.4 9.9 14.7 144.0 956 15.3 data 16.8 22:30 15.4 11.1 9.7 134.3 900 15.0 data 16.8 22:45 14.6 11.0 6.9 93.5 1087 14.7 data 16.8 23:00 15.9 9.8 6.1 89.1 1118 14.8 data 16.8 23:15 16.5 9.8 3.8 89.0 976 15.7 data 16.8 23:30 17.4 8.8 0.7 6.1 961 16.4 data 16.8 23:45 16.6 10.1 8.8 351.2 901 16.5 data 16.8 00:00 16.4 10.5 2.2 278.3 948 16.5 data 16.8 00:15 16.4 10.8 4.3 352.5 1051 16.3 data 16.8 00:30 16.2 11.3 1.7 200.0 956 16.1 data 16.8 00:45 16.2 11.5 4.9 21.1 958 16.0 data 16.8 01:00 17.5 10.3 1.0 214.0 1080 16.7 data 16.8 01:15 17.4 10.5 2.5 197.1 1023 16.9 data 16.8 01:30 17.9 9.9 7.1 185.0 1142 17.3 data 16.8 01:45 17.6 10.0 4.3 190.3 1073 17.2 data 16.8 02:00 17.3 9.2 9.2 178.8 1114 16.8 data 16.8 02:15 16.9 9.9 0.9 198.3 1041 16.5 data 16.8 02:30 17.4 8.4 0.5 200.0 753 17.0 data 16.8 02:45 18.3 7.7 0.7 312.2 839 17.3 data 16.8 03:00 17.3 8.6 1.3 255.3 833 17.1 data 16.8 03:15 17.4 8.8 7.5 17.3 871 17.0 data 16.8 03:30 17.4 8.7 7.2 37.3 801 17.0 data 16.8 03:45 16.9 8.8 3.8 358.6 880 16.8 data 16.8 04:00 16.3 9.8 9.0 39.1 712 16.4 data 16.8 04:15 17.4 8.8 2.9 161.9 764 16.9 data 16.8 04:30 17.7 8.9 3.1 189.3 838 17.4 data 16.8 04:45 17.9 9.0 7.1 178.0 885 17.5 data 16.8 05:00 17.4 8.8 0.9 157.6 747 17.0 data 16.8 05:15 17.2 9.5 1.6 193.5 890 17.2 data 16.8 05:30 17.0 9.7 9.6 180.5 989 16.4 data 16.8 05:45 16.8 10.3 2.3 167.8 839 16.7 data 16.8 06:00 16.9 9.7 1.7 194.6 924 16.3 data 16.8 06:15 17.2 8.9 0.9 252.3 772 16.2 data 16.8 06:30 17.5 9.1 1.9 166.6 1029 17.0 data 16.8 06:45 18.0 8.3 2.4 306.4 662 16.7 data 16.8 07:00 17.4 8.9 4.8 265.1 704 16.6 data 16.8 07:15 17.1 8.9 0.8 233.3 688 16.5 data 16.8 07:30 17.2 8.8 1.0 184.9 666 16.7 data 16.8 07:45 16.5 9.4 7.6 84.6 1049 16.3 data 16.8 08:00 16.6 9.6 6.6 89.7 820 16.3 data 16.8 08:15 15.9 9.7 4.4 85.8 818 16.1 data 16.8 08:30 14.9 10.2 5.1 98.7 826 15.0 data 16.8 08:45 16.2 9.5 0.5 194.3 788 15.7 data 16.8 09:00 16.5 9.4 3.2 263.0 735 15.4 data 16.8 09:15 16.5 9.4 2.2 244.9 782 15.5 data 16.8 09:30 16.3 9.4 1.3 206.3 748 15.9 data 16.8 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 60489 Fill night time in Hours below *=*=*= Total bright time : 6.0 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 6.0 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 2.0 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 2.0 *=*=*= Dark time lost to technical issues : 0