Subject: 2.5m obslog 20230226 (SJD 60002) From: dmbiz@apo.nmsu.edu Date: Mon, 27 Feb 2023 12:54:14 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Sunday February 26, 2023 (SJD 60002) ---- Observing Team ---- Dmitry Bizyaev ---- Observing Plan ---- Sun 26 Feb 2023 60002 0.50 01:51 02:05 12:25 05:29 15:50 --- bright until 08:08 ( 6.3 h, w/ 0.2 h for 12 deg design), then dark ( 4.3 h) ---- Observing Summary ---- APO was on the generator power for the whole night. Not opened. Unstable power from UPS in the plug lab makes internet switch go up and down, so we've got critical alerts for apogee and apogeeql for the whole night. Morning APOGEE calibrations skipped because of this. ---- Engineering Issues ---- No commercial power. Suspected bad UPS power in the plug lab. ---- Software Issues ---- No connection to apogee-ics machine because of faulty UPS power. It causes critical alerts for apogee. apogeeql, apogeebpr. ---- Observing Log ---- Afternoon --------- All systems checked. We have a power outage at APO, and the site runs on the main generator. Actually the power outage is in a large area from Carizozo to Timberon. Started around 11:30 am MST and still not repaired. Apogee actor was reported error - critical alert. See below. BOSS collimation looks good:02:42:14 Hartmann collimate output sp1: offset: r= -3958 (Out of focus); b= 6.8 (Out of focus) sp1: pred. move: spAverageMove= -3071 sp1: pred. resid: r= -886, b= -2.8, txt= OK, spTemp = 1.5 The problem with the apogee actor was traced back to the inability to connect to apogee-ics. In turn, John Downey recommended to check connection switch in the plug lab. At the same time, we received a few calls from "Monitoring Center" - hey said that they lost connection to "2.5m plug lab", but what it means for them they cannot explain. UPDATE: they called 7-8 times, every time the connection disappeared. Like robots, they ask my name, read their message and cannot explain why they are calling. The switch was checked. It is labeled as 10.25.1. It had the Power LED on, but didn't show other activity. I power cycle the switch, it did not fix the connection immediately, but it 5-7 minutes suddenly apogee-ics started responding to pings. The apogee actor responded as well, and I started APOGEE evening cals. They failed on the first UNe, and apogee-ics got not responsible again. I powered cycle the switch again. In some time it restored connections. Attempted to resume the calibrations, but they got stuck again on the Th-Ar. I stop trying to push APOGEE calibration until clearing the problem. I think that the problem is in the UPS and not on the switch itself, because: - the UPS makes strange, loud clicking, mechanical noise. Like it is chopping something. f ~ 1Hz. - Two external camera displays in the control room, which shows the apron and telescope, flickering with similar frequency, and cycle between showing pictures and blue screen with black sign "no signal". It may matter because the cameras are mounted on the plug lab building. Either they receive dirty current from the same UPS that confuses them, or the building itself receives not clear current form the site generator. - time to time, seldom, the images on the cameras stabilize, and it corresponds to the time when the apogee-ics gets responsible. I think these events are connected. John Downey has been notified. He's going to replace the UPS tomorrow. 01:16:55 boss exposure 0 sec file=0353108 bias 01:18:23 boss exposure 20 sec file=0353109 dark 01:43:47 apogee exposure file= apRaw-44400001 3 A ArcLamp 01:45:23 apogee exposure file= apRaw-44400002 60 A Dark 01:56:23 apogee exposure file= apRaw-44400003 10 A QuartzFlat 01:58:17 apogee exposure file= apRaw-44400004 12 B ArcLamp 02:00:29 apogee exposure file= apRaw-44400005 40 B ArcLamp 02:40:08 boss exposure 4 sec file=0353110 arc 02:40:49 boss exposure 4 sec file=0353111 arc Night ----- Not opening because APO is on generator power. The sky is clear. After sunset T dropped to -9C and humidity jumped upto the yellow. 02:42:05 Hartmann: offset r= -3958 (Out of focus) b = 6.8 (Out of focus) pred_move = -3071 pred_resid r= -886 b= -2.8 cmnt=OK 02:47:18 apogee exposure file= apRaw-44400006 3 B Dark 02:49:22 apogee exposure file= apRaw-44400007 10 A QuartzFlat 02:51:17 apogee exposure file= apRaw-44400008 12 B ArcLamp The UPS stabilized and I was able to finish evening calibrations 03:31:49 apogee exposure file= apRaw-44400009 12 B ArcLamp 03:34:01 apogee exposure file= apRaw-44400010 40 B ArcLamp 03:41:18 apogee exposure file= apRaw-44400011 12 A ArcLamp 03:43:31 apogee exposure file= apRaw-44400012 40 A ArcLamp 03:50:57 apogee exposure file= apRaw-44400013 60 A Dark 04:01:48 apogee exposure file= apRaw-44400014 30 A ArcLamp Then it got unstable again. The connection was unstable during the whole night. According to telemetry, APOGEE instrument had good temperatures and pressure. During the night the UPS in the plug lab just turned itself off. I turned it back on in the morning, not sure how long it can stay. 12:30Z we are still on the generator. Calling it a night. I skip APOGEE morning cals because of no connection to apogee-ics. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0110 0.0090 Axial B: 0.0030 0.0020 Axial C: -0.0290 -0.0290 Trans D: 0.0650 0.0580 Lateral E: 0.3315 0.3315 Lateral F: 0.5355 0.5345 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 0.00 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -0.0340 -0.0310 Axial B: -0.0320 -0.0260 Axial C: -0.0490 -0.0470 Trans D: -0.0250 -0.0270 GALILS Commanded: 4901250 4877750 4970500 0 -3400 Actual: 4859110 4858997 4859293 0 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.96 -0.08 -0.04 -0.03 41.69 ---- Telescope Status ---- Not opened No interlocks bypassed Status at: 12:41Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 697/715 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 180 lbs, 19 psi FPS Status ========== CP10 Loop = 7.18 L/min GFA Loop = 1.82 L/min Supply pressure = 42.2 psi Return pressure = 29.0 psi Chiller Status ============== Set temperarature = 1.0 C Flow = 5.3 Gpm ---- Software Used ---- STUI : 1.7.19 alerts : 2.3.0 boss : 6.0.2-alpha.0 -boss.specMec : 3.0.4 apo : 3.0.1 fliswarm : 0.4.0 jaeger : 1.3.4a0 kaiju : 1.2.4-alpha.0 coordio : 1.6.2a0 fps_calibrati : 2023.01.05 cherno : 0.5.2 astrometrynet : 0.93 hal : 0.6.0b5 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 35 -mcp.azFiduci : v1_147 -mcp.altFiduc : v1_147 -mcp.rotFiduc : v1_147 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.20 Boson : 0.3.2 --------------Other Versions-------------- sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : 2.1-1-ge461 PLC : V35 Fiducials : v1_147 idlspec2d : v6_0_9 Kronos : 1.6.1 roboscheduler : 1.3.2 /dev/sdb1 2.0T 247G 1.8T 13% /home /dev/sdd1 22T 11T 9.9T 53% /data ---- APOGEE Data Summary ---- APOGEE data list, mjd=60002, all raw_data: /data/apogee/utr_cdr/60002/apRaw-*.fits archive: /data/apogee/archive/60002/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/60002/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 60002 1 01:44 44400001 Arc---- 3 A 13.0 ---------- a-b-c - 1 test 60002 2 01:55 44400002 Dark 60 A 13.0 ---------- a-b-c - - 60002 3 01:57 44400003 QuarFlat 10 A 13.0 ---------- a-b-c - - 60002 4 02:00 44400004 Arc-Thar 12 B 13.5 ---------- a-b-c 0.88 - 60002 5 no file found: /data/apogee/utr_cdr/60002/apRaw-44400005.fits 60002 6 02:47 44400006 Dark 3 B 13.5 ---------- a-b-c - - 60002 7 02:50 44400007 QuarFlat 10 A 13.0 ---------- a-b-c - - cannot read file : /data/apogee/utr_cdr/60002/apRaw-44400008.fits 60002 9 03:33 44400009 Arc-Thar 12 B 13.5 ---------- a-b-c 0.91 - 60002 10 03:40 44400010 Arc-Une 40 B 13.5 ---------- a-b-c - - 60002 11 03:43 44400011 Arc-Thar 12 A 13.0 ---------- a-b-c 0.36 - 60002 12 03:50 44400012 Arc-Une 40 A 13.0 ---------- a-b-c - - 60002 13 04:01 44400013 Dark 60 A 13.0 ---------- a-b-c - - 60002 14 04:06 44400014 Arc---- 30 A 13.0 ---------- a-b-c - 79 FPI ------------------------------------------------------------------------------ ---- BOSS Spectrographs Collimation ---- Time TSP1 R1 B1 Move1 B1Resid ============================================= 02:42:00 1.0 -3958 6.8 -3071 -2.8 ---- Weather Log ---- Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma 19:00 -100.0 -100.0 36.4 225.8 9843 19:15 -100.0 -100.0 64.2 220.6 14680 19:30 -100.0 -100.0 47.8 240.6 23114 19:45 -100.0 -100.0 65.5 202.0 36082 20:00 -100.0 -100.0 52.0 240.3 63100 20:15 -100.0 -100.0 60.3 261.7 66675 20:30 -100.0 -100.0 50.4 263.5 101917 20:45 -100.0 -100.0 39.1 265.3 62849 21:00 -100.0 -100.0 43.9 285.6 58861 21:15 -100.0 -100.0 47.2 284.2 39597 21:30 -100.0 -100.0 39.1 262.6 34250 21:45 -100.0 -100.0 34.3 276.7 dpDep 22:00 -100.0 -100.0 56.6 283.0 dpDep 22:15 -100.0 -100.0 30.1 256.9 dpDep 22:30 -100.0 -100.0 54.3 273.1 dpDep 22:45 -100.0 -100.0 36.7 271.1 dpDep 23:00 -100.0 -100.0 38.9 275.3 dpDep 23:15 -100.0 -100.0 50.8 276.7 18544 23:30 -100.0 -100.0 40.6 254.5 8709 23:45 -100.0 -100.0 34.4 256.8 17229 00:00 -100.0 -100.0 35.5 273.9 dpDep 00:15 -100.0 -100.0 33.5 261.1 dpDep 00:30 -100.0 -100.0 36.7 263.3 dpDep 00:45 -100.0 -100.0 34.8 276.1 7758 01:00 -100.0 -100.0 28.9 268.5 22775 01:15 -100.0 -100.0 23.4 265.5 16854 01:30 -100.0 -100.0 17.9 272.2 2054 01:45 -100.0 -100.0 20.2 280.5 998 02:00 -100.0 -100.0 20.2 280.3 1106 02:15 -100.0 -100.0 18.1 268.1 1104 02:30 -100.0 -100.0 18.7 277.2 681 02:45 -100.0 -100.0 26.4 291.1 870 03:00 -100.0 -100.0 14.6 310.9 1706 03:15 -100.0 -100.0 23.2 290.2 2683 03:30 -100.0 -100.0 19.2 297.4 5077 03:45 -100.0 -100.0 22.7 289.8 3296 04:00 -100.0 -100.0 17.1 303.2 2084 04:15 -100.0 -100.0 13.3 317.1 dpDep 04:30 -100.0 -100.0 16.1 342.8 dpDep 04:45 -100.0 -100.0 12.3 324.2 dpDep 05:00 -100.0 -100.0 11.5 351.8 dpDep 05:15 -100.0 -100.0 6.9 335.2 dpDep 05:30 -100.0 -100.0 12.0 352.9 dpDep 05:45 -100.0 -100.0 13.0 3.6 dpDep 06:00 -100.0 -100.0 14.1 24.5 219 06:15 -100.0 -100.0 17.6 2.0 147 06:30 -100.0 -100.0 20.9 8.5 164 06:45 -100.0 -100.0 19.5 352.5 137 07:00 -100.0 -100.0 25.0 317.5 484 07:15 -100.0 -100.0 27.1 305.8 181 07:30 -100.0 -100.0 24.3 308.3 209 07:45 -100.0 -100.0 25.1 322.6 138 08:00 -100.0 -100.0 12.5 318.0 129 08:15 -100.0 -100.0 19.8 316.3 147 08:30 -100.0 -100.0 26.4 315.6 164 08:45 -100.0 -100.0 13.3 297.8 135 09:00 -100.0 -100.0 21.5 301.6 136 09:15 -100.0 -100.0 14.3 306.1 142 09:30 -100.0 -100.0 17.5 336.1 188 09:45 -100.0 -100.0 23.5 307.8 214 10:00 -100.0 -100.0 26.1 322.6 164 10:15 -100.0 -100.0 11.5 329.7 203 10:30 -100.0 -100.0 18.7 314.9 175 10:45 -100.0 -100.0 26.6 326.1 120 11:00 -100.0 -100.0 19.9 321.8 120 11:15 -100.0 -100.0 11.0 341.1 111 11:30 -100.0 -100.0 19.5 322.5 95 11:45 -100.0 -100.0 15.9 319.2 95 12:00 -100.0 -100.0 17.4 313.2 126 12:15 -100.0 -100.0 23.2 318.9 89 12:30 -100.0 -100.0 27.6 302.0 111 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 60002 Fill night time in Hours below *=*=*= Total bright time : 6.3 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 5.3 *=*=*= Bright time lost to technical issues : 1.0 *=*=*= Total dark time : 4.3 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 0 *=*=*= Dark time lost to technical issues : 4.3