Subject: 2.5m obslog 20250526 (SJD 60822) From: dmbiz@apo.nmsu.edu Date: Tue, 27 May 2025 09:52:22 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday May 26, 2025 (SJD 60822) ---- Observing Team ---- Dmitry Bizyaev ---- Observing Plan ---- Mon 26 May 2025 60822 0.00 03:04 03:22 10:37 12:39 19:55 --- dark ( 7.3 h) plus 1.2 h for 8 deg design ---- Observing Summary ---- The entire night was lost to technical problems with MCP. The mcp computer itself was restored. During the day John and Katie replaced cards in the MCP rack, restarted MCP and TPM, and fixed many conenction problems reported by the MCP scripts. Nevertheless, some of them persist: MCP/TPM cannot manage semaphore, the status of FFS is shown as Error, communication between MCP and tcc is problematic, axes cannot be connected. During the night I tried to restore semaphores via resetting processes at the mcp computer, and by rebooting mco and tpm computers and scripts, but it didn't help. John suspects that the problem is in the PLC card, and will investigate duirng the day time. ---- Engineering Issues ---- MCP problem, see night and day logs. ---- Observing Log ---- Afternoon ========= Most of systems are working as expected. Restoration of MCP is continuing. To this evening, Shane was able to clone new disk from theold MCP one, and the MCP computer is accessible. When mcp scripts were started, they reported numerous warning messages that suggested problems with the PLC card. A plan was to continue the restoration via rebooting TPM. The APOGEE and BOSS evening calibrations look good, but not all calibrations can be done because of the MCP status. 23:41:03 apogee exposure file= apRaw-52600001 3 B ArcLamp 23:41:38 boss exposure 0 sec file=0385927 bias ======================================================= =====>>>>> 23:41:56 Config Loaded: -999--999-20986 ======================================================= 23:43:32 boss exposure 20 sec file=0385928 dark 23:46:00 apogee exposure file= apRaw-52600002 60 B Dark 23:56:52 apogee exposure file= apRaw-52600003 10 B QuartzFlat 23:58:45 apogee exposure file= apRaw-52600004 12 B ArcLamp 00:00:58 apogee exposure file= apRaw-52600005 40 B ArcLamp 00:08:15 apogee exposure file= apRaw-52600006 12 A ArcLamp 00:10:27 apogee exposure file= apRaw-52600007 40 A ArcLamp 00:17:53 apogee exposure file= apRaw-52600008 60 A Dark 00:28:45 apogee exposure file= apRaw-52600009 30 A ArcLamp 00:34:09 apogee exposure file= apRaw-52600010 30 B ArcLamp The system status at sunset: MCP scipts work, and the processes are up. MCP showed semaphore "Sem: None" FFS screen status is Err, and all axes position in MCP/TPM GUI is 0. MCP status shiwn as MCP in the upper left corner is "unavailable". I checked what last mcp version was reported in STUI's Status, and it was mcp : 2.1-1-ge461 Not it reports "2.1", but this STUI script is pretty far away from MCP and can be wrong. I tied to post semaphores MEI and MCAST via "./ipc -sem_post MCAST" , but their status remains "0". But, suddebly, MCP showed that it owns the semaphore. At the same time, it was unable to give" it - no effect. 02:23Z following found on Tips instuction of how the "unavailable" was handled before, I decided to reboot MCP and then TMP, in this order. Before restarting , I checked the status of Xyplex ports and cannot find any locked ones. Not touching them because of it. After restarting MCP computer, the status of MCP is blinking between "unavailable" and "2.1-1-ge461". After restarting TPM processes or TPM cpomputer, the status turns to solid "unavailable". I cannot change semaphore status from "none" to anything else. 09:22Z APOGEE calibrations started. 09:51Z calling it a night. [ (52600001, 3, 3, '/data/apogee/raw/60822/apRaw-52600001-001.fits', 'ARCLAMP', 'None', '', '20985', '734727', '112360', 32.041, '2025-05-26T23:41:03.696', 'Podium', 'Closed', 'Closed', 60822, 'APO', 13.499)] problem with exposure: 52600001 num field design config exptype dithpix nread relflux okay dx dy sigx sigy fpi1 fpi2 -------- ------ ------ ------ ------------ ------- ----- ------- ----- ------- ------- ------ ------ ---- ---- 52600001 112360 734727 20985 ARCLAMP-None 13.5 3 0 False 0.00 0.00 0.00 0.00 0 0 52600002 -999 -999 20986 DARK 13.5 60 -12 True -- -- -- -- -- -- 52600003 -999 -999 20986 QUARTZFLAT 13.5 10 102 True -- -0.26 -- 0.87 -- -- 52600004 -999 -999 20986 ARCLAMP-THAR 13.5 12 99 True 3.07 0.53 1.05 0.87 -- -- 52600005 -999 -999 20986 ARCLAMP-UNE 13.5 40 106 True 2.67 0.44 1.13 0.89 -- -- 52600006 -999 -999 20986 ARCLAMP-THAR 13.0 12 94 True 2.58 0.59 1.04 0.86 -- -- 52600007 -999 -999 20986 ARCLAMP-UNE 13.0 40 98 True 2.21 0.37 1.09 0.90 -- -- 52600008 -999 -999 20986 DARK 13.0 60 2 True -- -- -- -- -- -- 52600009 -999 -999 20986 FPI 13.0 30 202 True 14.01 0.33 1.19 0.89 -- -- 52600010 -999 -999 20986 FPI 13.5 30 202 True -10.10 0.31 1.16 0.87 -- -- 52600011 -999 -999 20986 DARK 13.5 60 -7 True -- -- -- -- -- -- 52600012 -999 -999 20986 DARK 13.5 60 -28 True -- -- -- -- -- -- num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 mtp10 -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----- 52600003 QUARTZFLAT 10 67 72 82 91 91 86 88 90 90 82 ---- Telescope Status ---- Never opened A monitor is connected to the MCP computer in the lower enclosure for the further mcp works. Status at: 09:50Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 651 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 164 lbs, 24 psi FPS Status ========== CP10 Loop = 7.48 L/min GFA Loop = 2.24 L/min Supply pressure = 43.5 psi Return pressure = 29.8 psi Chiller Status ============== Set temperarature = 14.5 C Flow = 4.9 Gpm ---- Software Used ---- -- STUI software: STUI : 1.7.19 alerts : 2.3.0 boss : 6.0.2 -boss.specMec : 3.0.4 apo : 4.1.0 fliswarm : 0.5.1a0 jaeger : 1.9.2a0 kaiju : 1.4.0 coordio : 1.13.0 fps_calibrati : 2025.02.24 cherno : 1.1.0 astrometrynet : 0.93 hal : 1.4.1 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_150 -mcp.altFiduc : v1_150 -mcp.rotFiduc : v1_150 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : FAILED Boson : 0.4.9 --------------Other Versions-------------- sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : unavailable idlspec2d : v6_2_1 Kronos : 2.0.0 roboscheduler : 2.1.0 /dev/sdb1 2.0T 459G 1.6T 23% /home /dev/sdd1 22T 13T 7.9T 63% /data ---- APOGEE Data Summary ---- APOGEE data list, mjd=60822, all raw_data: /data/apogee/utr_cdr/60822/apRaw-*.fits archive: /data/apogee/archive/60822/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/60822/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 60822 1 23:41 52600001 Arc---- 3 B 13.5 ---------- a-b-c - -13 test 60822 2 23:56 52600002 Dark 60 B 13.5 ---------- a-b-c - - 60822 3 23:58 52600003 QuarFlat 10 B 13.5 ---------- a-b-c - - 60822 4 00:00 52600004 Arc-Thar 12 B 13.5 ---------- a-b-c 3.00 - 60822 5 00:07 52600005 Arc-Une 40 B 13.5 ---------- a-b-c - - 60822 6 00:10 52600006 Arc-Thar 12 A 13.0 ---------- a-b-c 2.45 - 60822 7 00:17 52600007 Arc-Une 40 A 13.0 ---------- a-b-c - - 60822 8 00:28 52600008 Dark 60 A 13.0 ---------- a-b-c - - 60822 9 00:33 52600009 Arc---- 30 A 13.0 ---------- a-b-c - 111 FPI A 60822 10 00:39 52600010 Arc---- 30 B 13.5 ---------- a-b-c - 111 FPI B ------------------------------------------------------------------------------ ---- BOSS Data Summary ---- Reading FITS from /data/spectro//60822 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 23:41 r1-00385927 ----------- ---- bia Clo 0.0 5933 6073 6055 2 23:41 b1-00385927 ----------- ---- bia Clo 0.0 5933 6073 6055 3 23:43 r1-00385928 ----------- ---- dar Clo 20.0 5933 6073 6055 4 23:43 b1-00385928 ----------- ---- dar Clo 20.0 5933 6073 6055 ---- Weather Log ---- Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma 18:00 15.8 -13.1 10.3 46.5 1913 18:15 17.4 -10.7 12.9 40.1 1724 18:30 15.5 -11.3 11.4 66.9 1928 18:45 14.7 -12.0 7.2 45.1 1846 19:00 14.8 -12.4 14.8 61.5 2182 19:15 15.6 -12.6 13.3 10.8 2005 19:30 16.3 -10.1 20.1 46.6 1902 19:45 17.2 -11.0 15.8 60.5 1779 20:00 18.9 -10.7 13.7 36.8 1934 20:15 17.8 -9.7 11.7 83.8 1877 20:30 16.7 -9.5 15.8 68.0 2171 20:45 15.8 -10.8 12.6 75.4 1917 21:00 17.0 -10.4 11.0 93.1 1956 21:15 16.5 -11.1 15.3 55.3 2000 21:30 16.5 -9.8 15.2 74.5 2218 21:45 16.1 -10.8 11.0 63.7 2092 22:00 16.9 -10.4 9.7 51.9 2653 22:15 19.6 -9.9 12.2 69.8 2000 22:30 16.8 -9.9 17.2 57.8 2362 22:45 16.3 -9.5 18.4 53.1 2231 23:00 18.3 -10.0 20.0 35.5 2040 23:15 16.8 -10.2 16.8 62.8 2243 23:30 17.9 -10.2 15.3 88.3 1936 23:45 18.5 -9.8 13.8 66.7 1931 00:00 17.5 -10.1 16.8 45.4 1909 00:15 16.7 -10.4 19.0 64.3 1880 00:30 16.8 -10.8 17.3 58.5 2146 00:45 15.7 -11.3 15.6 66.0 2095 01:00 16.1 -11.2 14.8 59.6 2100 01:15 16.0 -11.6 14.8 52.8 2032 01:30 15.2 -12.2 11.1 45.5 2081 01:45 14.7 -13.1 9.6 62.8 2086 02:00 13.9 -13.8 10.9 46.0 2115 02:15 13.2 -14.8 13.9 36.2 2334 02:30 13.9 -14.9 9.9 41.1 2189 02:45 13.7 -14.9 15.6 40.8 2200 03:00 13.7 -14.8 13.5 40.0 2774 03:15 12.7 -14.9 12.0 36.1 2668 03:30 13.1 -14.8 18.3 37.0 2763 03:45 13.0 -14.9 14.3 50.3 3054 345 2.36 04:00 12.8 -14.9 13.3 40.0 3235 346 1.94 04:15 11.9 -15.4 13.7 55.3 3407 346 1.97 04:30 12.4 -15.4 7.6 22.3 3111 345 1.82 04:45 12.6 -15.3 11.2 28.1 3019 345 2.02 05:00 13.6 -15.1 10.3 28.1 2944 345 1.99 05:15 13.6 -14.8 12.4 36.1 2885 345 1.84 05:30 13.5 -14.8 9.9 39.9 2640 346 2.06 05:45 12.3 -14.8 10.9 28.0 2640 345 2.03 06:00 12.4 -14.7 10.3 34.6 2377 346 2.27 06:15 12.8 -14.5 7.9 08.5 2102 345 1.95 06:30 12.9 -14.2 4.9 99.2 2040 345 2.15 06:45 13.1 -14.2 8.5 94.2 2060 346 2.51 07:00 13.1 -13.7 10.2 88.3 1999 345 2.29 07:15 13.4 -13.6 7.0 40.5 2028 345 2.35 07:30 13.1 -12.9 9.3 00.3 2021 345 2.37 07:45 13.1 -12.6 9.5 82.5 1984 345 2.11 08:00 13.3 -12.1 9.9 00.0 1999 345 2.22 08:15 13.1 -12.6 10.2 85.8 1882 345 2.33 08:30 12.6 -12.9 8.9 92.0 1988 345 3.16 08:45 12.9 -12.1 9.2 09.1 2033 345 2.26 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 60822 Fill night time in Hours below *=*=*= Total bright time : 1.2 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 0 *=*=*= Bright time lost to technical issues : 1.2 *=*=*= Total dark time : 7.3 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 0 *=*=*= Dark time lost to technical issues : 7.3