Subject: 2.5m obslog 20201024 (SJD 59147) From: kinemuchi@apo.nmsu.edu Date: Sun, 25 Oct 2020 12:40:17 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Saturday October 24, 2020 (SJD 59147) ---- Observing Team ---- Dylan Gatlin Karen Kinemuchi Jose Sanchez-Gallego (for apogee-ql fix) Fred Mrozek (reposition the telescope) ---- Observing Plan ---- Sat 24 OCT 2020 59147.0 0.67 01:15 12:08 20:27 07:22 ---- KK APOGEE-2 until 07:50 ( 6.6 h), then BHM ( 4.3 h) ---- Observing Summary ---- APOGEE-2N (SDSS-IV) OBSERVING: ============================== Cart 4 (Plate 11270; Field=K19_076+07_btx) - GSOGTF; 2xDAB --> not done Missing fibers: [20, 39, 62, 151, 186, 193, 211, 241, 273, 278] Faint fibers: ['121 - 129', 133, 136, '152 - 162', '166 - 167', '169 - 170', 175, 177, '179 - 183', 185, '187 - 190', 192, '194 - 195', 197, 204] Cart 3 (Plate 12765; Field=FL_20200 - GSOGTF; 4xAB --> not done Missing fibers: [8, 211, 242, 245, 256, 273] Faint fibers: ['151 - 173', '175 - 180', '271 - 272', '274 - 276', '278 - 300'] Cart 6 (Plate 12759; Field=N752_btx)- GSOGTF; 2xDAB --> DONE! Missing fibers: [181, 211, 273] Faint fibers: [15, 34, 105, '151 - 173', '175 - 180', 289] Cart 5 (Plate 12760; Field=N752_btx) - GSOGTF; 3xDAB --> not done. Missing fibers: [211, 273] Faint fibers: ['121 - 137', '139 - 140', 142, '147 - 164', '166 - 173', '175 - 210', '212 - 213', 216, 220, '222 - 223', '225 - 226', '230 - 231', 234, '236 - 240'] Cart 7 (Plate 9081; Field=N188) - GSOGTF; 4xAB --> done? Missing fibers: [172, 211, 273] Faint fibers: ['151 - 163', '166 - 171', 173, '175 - 180', '212 - 240'] BHM-Lead (SDSS-V) OBSERVING: =================== Cart 1 (Plate 15002) - No data. See Problems in Detail. Cart 9 (Plate 15017) - GSOGTF; 2x900s --> not done See Problems in Detail. Notes for day staff and developers: ==================================== - /data disk usage is at 92% and needs to be cleaned * Cart 5: acquisition fiber #3 is crumpled. We can still see part of the guide star, but is no longer circular. * MCP server crashed(?) 3 times tonight and we lost control of the telescope while slewing to field and also while taking data on Cart 9. We closed to protect the telescope. We are uncertain as to why the MCP crashed. Called Shane, JeffH, and Fred. See Problems in Detail section. Telescope is currently at stow position in the enclosure. * Dust counts: 800. * MCP crashed as we are writing this log. We are getting a critical alerts now. We will try to fix this if we can, but we cannot log onto the server. ---- Observing Log ---- Afternoon: Short dark okay, started APOGEE evening cals. BOSS cals okay Mirror numbers match yesterday but not nominals. We cannot take new nominals, which may be a LUI bug involving the new TPM. 22:50:10 Hartmann collimate output on cart #9 sp1: offset: r= -989 (In Focus); b= -9.3 (Out of focus) sp1: pred. move: spAverageMove= 983 sp1: pred. resid: r= -1972, b= -6.2, txt= Move blue ring -12.4 degrees., spTemp = 8.2 Adjusted SP1 by +5 (whoops, I made a sign error!) 23:09:56 Hartmann collimate output on cart #9 sp1: offset: r= -1649 (Out of focus); b= -13.3 (Out of focus) sp1: pred. move: spAverageMove= 1296 sp1: pred. resid: r= -2945, b= -9.2, txt= Move blue ring -18.5 degrees., spTemp = 8.3 Adjusted SP1 by -15 23:15:22 Hartmann collimate output on cart #9 sp1: offset: r= -2968 (Out of focus); b= 6.8 (Out of focus) sp1: pred. move: spAverageMove= -2576 sp1: pred. resid: r= -391, b= -1.2, txt= OK, spTemp = 8.3 Boss arc and flats completed, SOS alive, but the temperature is still NAN Disk usage is quite high on hub. Home Disk Hogs (GB): manga 104.743 eboss 88.9687 sdss3 27.1813 apogee 18.7952 observer 14.974 logging 13.7661 Filesystem Size Used Avail Use% Mounted on /dev/vdd 688G 580G 108G 85% /home /dev/vdc1 11T 9.9T 936G 92% /data 23:26Z Started ventilation. It was mostly cloudy, but there was no radar or humidity. Loaded cart 4 for the night, taped down that loose cable (the interlock cable for the LN2 fill control box), and ran test slews. Test slews were fine, but once I returned to stow and reengaged a stop button, it began to drift to ~-3deg. This rotator movement is related to the small drift when the telescope is at 5 degrees. This drift in rotator is due to it not being completely balanced between the spectrograph and the dummy weight on the sp2 (north) side. These drifts will spook the observers. Confirmed with JohnD that this is ok. !! Does not affect cart changes since telescope is at 90, but it is good to check the MCP that the rotator is indeed at 0 degrees before doing a cart change. 00:20Z Enclosure open NIGHT: ====== 23:50:22 loadCart: Cart 4, Plate 11270; 23:50:22 survey: APOGEE-2&MaNGA; lead: APOGEE lead 00:51:51 apogee.expose=apRaw-35850009, 10, A, Dark 00:53:38 apogee.expose=apRaw-35850010, 10, A, Dark Forgot to set focus to 3000. Focused now. GSOGTF; partly cloudy skies; seeing is 1.4" 59147 8 23:12 35850008 Dark 60 A 13.0 9-15017 a-b-c - - BHM&MWM 59147 9 00:51 35850009 Dark 10 A 13.0 4-11270 a-b-c - - APOGEE-2 59147 10 00:53 35850010 Dark 10 A 13.0 4-11270 a-b-c - - APOGEE-2 59147 11 01:20 35850011 Object 94 A 13.0 4-11270 a-b-c -5.67 - APOGEE-2 59147 12 01:37 35850012 Object 94 B 13.5 4-11270 a-b-c -5.13 - APOGEE-2 59147 13 01:53 35850013 Object 94 B 13.5 4-11270 a-b-c -5.14 - APOGEE-2 59147 14 02:10 35850014 Object 94 A 13.0 4-11270 a-b-c -5.68 - APOGEE-2 59147 15 02:27 35850015 DomeFlat 5 A 13.0 4-11270 a-b-c - 53 APOGEE-2 59147 16 02:30 35850016 Dark 10 A 13.0 4-11270 a-b-c - - APOGEE-2 59147 17 02:32 35850017 Dark 10 A 13.0 4-11270 a-b-c - - APOGEE-2 aptest 59147 35850015 flat= /data/apogee/utr_cdr/59147/apRaw-35850015.fits master= apRaw-21180043.fits nfibers= 300 Missing fibers: [20, 39, 62, 151, 186, 193, 211, 241, 273, 278] Faint fibers: ['121 - 129', 133, 136, '152 - 162', '166 - 167', '169 - 170', 175, 177, '179 - 183', 185, '187 - 190', 192, '194 - 195', 197, 204] Plate 11270 is not complete ================================================================================ 02:33:48 loadCart: Cart 3, Plate 12765; 02:33:48 survey: APOGEE-2&MaNGA; lead: APOGEE lead GSOGTF; partly cloud; seeing is 1.3" 59147 18 02:43 35850018 Object 47 A 13.0 3-12765 a-b-c -5.67 - APOGEE-2 59147 19 02:51 35850019 Object 47 B 13.5 3-12765 a-b-c -5.14 - APOGEE-2 59147 20 03:00 35850020 Object 47 B 13.5 3-12765 a-b-c -5.14 - APOGEE-2 59147 21 03:08 35850021 Object 47 A 13.0 3-12765 a-b-c -5.67 - APOGEE-2 59147 22 03:16 35850022 Object 47 A 13.0 3-12765 a-b-c -5.67 - APOGEE-2 59147 23 03:25 35850023 Object 47 B 13.5 3-12765 a-b-c -5.15 - APOGEE-2 59147 24 03:33 35850024 Object 47 B 13.5 3-12765 a-b-c -5.13 - APOGEE-2 59147 25 03:42 35850025 Object 47 A 13.0 3-12765 a-b-c -5.66 - APOGEE-2 59147 26 03:50 35850026 DomeFlat 5 A 13.0 3-12765 a-b-c - 50 APOGEE-2 59147 27 03:52 35850027 Dark 10 A 13.0 3-12765 a-b-c - - APOGEE-2 59147 28 03:54 35850028 Dark 10 A 13.0 3-12765 a-b-c - - APOGEE-2 aptest 59147 35850026 flat= /data/apogee/utr_cdr/59147/apRaw-35850026.fits master= apRaw-21180043.fits nfibers= 300 Missing fibers: [8, 211, 242, 245, 256, 273] Faint fibers: ['151 - 173', '175 - 180', '271 - 272', '274 - 276', '278 - 300'] Plate 12765 is not complete. ================================================================================ ---------------------------------------- 03:56:40 loadCart: Cart 6, Plate 12759; 03:56:40 survey: APOGEE-2&MaNGA; lead: APOGEE lead GSOGTF; partly cloudy skies; seeing is 2.0" 59147 29 04:05 35850029 Object 94 A 13.0 6-12759 a-b-c -5.67 - APOGEE-2 59147 30 04:21 35850030 Object 94 B 13.5 6-12759 a-b-c -5.14 - APOGEE-2 59147 31 04:38 35850031 Object 94 B 13.5 6-12759 a-b-c -5.12 - APOGEE-2 59147 32 04:42 35850032 DomeFlat 5 B 13.5 6-12759 a-b-c - 54 APOGEE-2 59147 33 04:46 35850033 Dark 10 B 13.5 6-12759 a-b-c - - APOGEE-2 59147 34 04:48 35850034 Dark 10 B 13.5 6-12759 a-b-c - - APOGEE-2 aptest 59147 35850032 flat= /data/apogee/utr_cdr/59147/apRaw-35850032.fits master= apRaw-21180043.fits nfibers= 300 Missing fibers: [181, 211, 273] Faint fibers: [15, 34, 105, '151 - 173', '175 - 180', 289] Plate 12759 is COMPLETE! --------------------------------------- 04:51:14 loadCart: Cart 5, Plate 12760; 04:51:14 survey: APOGEE-2&MaNGA; lead: APOGEE lead GSOGTF; clearing skies, but clouds to the east and north, wind is increasing; seeing is 2.0" 59147 35 04:57 35850035 Object 94 B 13.5 5-12760 a-b-c -5.13 - APOGEE-2 59147 36 05:14 35850036 Object 94 A 13.0 5-12760 a-b-c -5.64 - APOGEE-2 59147 37 05:30 35850037 Object 94 A 13.0 5-12760 a-b-c -5.64 - APOGEE-2 59147 38 05:47 35850038 Object 94 B 13.5 5-12760 a-b-c -5.12 - APOGEE-2 59147 39 06:04 35850039 Object 94 B 13.5 5-12760 a-b-c -5.11 - APOGEE-2 59147 40 06:37 35850040 Object 94 A 13.0 ------- a-b-x - - None 59147 41 06:37 35850041 DomeFlat 5 A 13.0 5-12760 a-b-c - 34 APOGEE-2 59147 42 06:40 35850042 Dark 10 A 13.0 5-12760 a-b-c - - APOGEE-2 59147 43 06:41 35850043 Dark 10 A 13.0 5-12760 a-b-c - - APOGEE-2 aptest 59147 35850041 flat= /data/apogee/utr_cdr/59147/apRaw-35850041.fits master= apRaw-21180043.fits nfibers= 300 Missing fibers: [211, 273] Faint fibers: ['121 - 137', '139 - 140', 142, '147 - 164', '166 - 173', '175 - 210', '212 - 213', 216, 220, '222 - 223', '225 - 226', '230 - 231', 234, '236 - 240'] Plate 12760 is not complete. ================================================================================ 06:43:00 loadCart: Cart 7, Plate 9081; 06:43:00 survey: APOGEE-2&MaNGA; lead: APOGEE lead GSOGTF; partly cloudy; seeing is 1.8" 59147 44 06:48 35850044 Object 47 A 13.0 7-9081 a-b-c -5.67 - APOGEE-2 59147 45 06:56 35850045 Object 47 B 13.5 7-9081 a-b-c -5.15 - APOGEE-2 59147 46 07:04 35850046 Object 47 B 13.5 7-9081 a-b-c -5.15 - APOGEE-2 59147 47 07:13 35850047 Object 47 A 13.0 7-9081 a-b-c -5.66 - APOGEE-2 59147 48 07:21 35850048 Object 47 A 13.0 7-9081 a-b-c -5.66 - APOGEE-2 59147 49 07:30 35850049 Object 47 B 13.5 7-9081 a-b-c -5.14 - APOGEE-2 59147 50 07:38 35850050 Object 47 B 13.5 7-9081 a-b-c -5.15 - APOGEE-2 59147 51 07:46 35850051 Object 47 A 13.0 7-9081 a-b-c -5.67 - APOGEE-2 59147 52 07:55 35850052 DomeFlat 5 A 13.0 7-9081 a-b-c - 56 APOGEE-2 59147 53 07:57 35850053 Dark 10 A 13.0 7-9081 a-b-c - - APOGEE-2 59147 54 08:01 35850054 Dark 10 A 13.0 ------- a-b-x - - None aptest 59147 35850052 flat= /data/apogee/utr_cdr/59147/apRaw-35850052.fits master= apRaw-21180043.fits nfibers= 300 Missing fibers: [172, 211, 273] Faint fibers: ['151 - 163', '166 - 171', 173, '175 - 180', '212 - 240'] Plate 9081 is not complete? ================================================================================ 08:01:12 loadCart: Cart 1, Plate 15002; 08:01:12 survey: BHM&MWM; lead: BHM lead HARTMANN: 08:05:04 boss exposure 4.0, file=318734 08:05:42 boss exposure 4.0, file=318735 08:05:44 apogee.expose=apRaw-35850055, 10, A, Dark 08:06:27 sp1.motor.move= -124, -123, -124 08:06:27 Hartmann collimate output on cart #1 sp1: offset: r= 329 (In Focus); b= 1.8 (In Focus) sp1: pred. move: spAverageMove= -124 sp1: pred. resid: r= 453, b= 1.4, txt= OK, spTemp = 6.7 CALIBS: 08:06:46 boss exposure 4.0, file=318736 08:07:47 apogee.expose=apRaw-35850056, 10, A, Dark 08:08:13 boss exposure 25.0, file=318737 08:21Z MCP appears to have died. Cannot connect to mcp@sdss-mcp. Telescope has lost it's position. See Problems in Detail. Plate 15002 is not complete. ================================================================================ 10:20:46 loadCart: Cart 9, Plate 15017; 10:20:46 survey: BHM&MWM; lead: BHM lead HARTMANN: 10:23:20 boss exposure 4.0, file=318738 10:23:57 boss exposure 4.0, file=318739 10:24:45 sp1.motor.move= -940, -941, -941 10:24:45 Hartmann collimate output on cart #9 sp1: offset: r= -659 (In Focus); b= 3.8 (In Focus) sp1: pred. move: spAverageMove= -940 sp1: pred. resid: r= 281, b= 0.9, txt= OK, spTemp = 6.3 CALIBS: 10:25:04 boss exposure 4.0, file=318740 10:26:31 boss exposure 25.0, file=318741 GSOGTF; clearing skies with thin clouds; seeing is 1.7" 10:34:27 boss exposure 900.0, file=318742 10:51:07 boss exposure 900.0, file=318743 11:07:47 boss exposure 900.0, file=318744 600) 08:02:42 Alt: ind=3, deg= 46.2, err= 337144 error is too large to be corrected ( > 600) 08:06:46 Alt: ind=3, deg= 46.2, err= 337144 error is too large to be corrected ( > 600) 09:30:40 Alt: ind=6, deg= 86.8, err= -522424 error is too large to be corrected ( > 600) 09:30:45 Alt: ind=6, deg= 86.8, err= -522291 error is too large to be corrected ( > 600) 09:33:56 Alt: ind=6, deg= 86.8, err= -522487 error is too large to be corrected ( > 600) 09:34:44 Alt: ind=5, deg= 73.5, err= -522512 error is too large to be corrected ( > 600) We accepted this large fiducial and did more test slews in altitude. Now the telescope altitude position is know. Next we reset the Azimuth. We had a guess from STUI Status window that AZ was around -37. We set in the MCP the Azimuth position with the following command: AZ set_pos_deg -37 We did incremental slews of 0, 50, 100, and 121. We were still off and manually put the AZ back at 121, reset the position in the MCP, and then did test slews. We moved the AZ to 151 (pointing towards ARCSAT) and back to 121, crossing 2 fiducials. Fiducial errors were: 09:43:02 Az: ind=11, deg=-36.0, err= 1744313 error is too large to be corrected ( > 600) 09:43:42 Az: ind=0, deg= -6.0, err= -1532787 error is too large to be corrected ( > 600) 09:44:44 Az: ind=15, deg= 24.1, err= 1750817 error is too large to be corrected ( > 600) 09:45:04 Az: ind=16, deg= 39.1, err= 1750268 error is too large to be corrected ( > 600) 09:46:18 Az: ind=17, deg= 54.1, err= 1742156 error is too large to be corrected ( > 600) 09:46:37 Az: ind=18, deg= 69.1, err= 1754722 error is too large to be corrected ( > 600) 09:46:56 Az: ind=19, deg= 84.1, err= 1742759 error is too large to be corrected ( > 600) 09:47:17 Az: ind=20, deg= 99.1, err= 1752384 error is too large to be corrected ( > 600) 09:48:18 Az: ind=21, deg=114.1, err= 1745934 error is too large to be corrected ( > 600) We accepted this fiducial error and did test slews. The errors were smaller and we were able to return to the stow 121 position. Next for the Rotator. We knew from STUI rotator was at 150. We set in the MCP with the following command: ROT set_pos_deg 150. We moved the rotator back to 0 and checked with the markings on the rotator to make sure we were moving in the right directions. After making sure we were really at ROT=0, we set in the MCP, ROT set_pos_deg 0. We did test slews to /rota=40, and checked the fiducial errors: 10:06:14 Rot: ind=77, deg= 5.8, err= -4394 error is too large to be corrected ( > 600) 10:06:18 Rot: ind=76, deg= 10.6, err= -4378 error is too large to be corrected ( > 600) 10:06:22 Rot: ind=75, deg= 15.3, err= -4367 error is too large to be corrected ( > 600) 10:06:26 Rot: ind=74, deg= 20.1, err= -4384 error is too large to be corrected ( > 600) 10:06:30 Rot: ind=73, deg= 24.8, err= -4361 error is too large to be corrected ( > 600) 10:06:34 Rot: ind=72, deg= 29.5, err= -4366 error is too large to be corrected ( > 600) 10:06:38 Rot: ind=71, deg= 34.3, err= -4359 error is too large to be corrected ( > 600) 10:06:44 Rot: ind=70, deg= 39.0, err= -4364 error is too large to be corrected ( > 600) We accepted the fiducials and did a couple of test slews to stow and gang change position. We were unable to observe Cart 1, so we did a cart change. We noticed in the interlocks gui the instrument lift interlock was red (the interlocks window through MCP), but the STUI interlocks window did not show this was on. We double checked to make sure the instrument lift was down, and it appeared to be fine - we moved it up and down with the pendant without any trouble. Cart change to Cart 9 was fine with no trouble with the pendant in automatic mode. We slewed to field fine from the podium and resumed operations. Sorry BHM-RM group. We did not finish your plate. We don't know why mcp crashed in the first place. 11:05Z MCP was lost again. We received a "serious alert" that the mcp heartbeat was dead. We noticed this time the guide stars streak away. When we went down the telescope, it was at a low altitude. We determined the telescope should have been at 71 degrees in altitude but according to the inclinomator, it was at 18 degrees altitude. We moved the telescope with the manual pendant again, but did not accept fiducials, since we forgot to hub startNubs mcp and could not get the fiducials except on the MCP gui. At this point, we are unsure about the mcp and these random slews of the telescope. We have closed the telescope at 12:00Z. The telescope is stored at stow with the LN2 line on. The TCC is reporting a wrong position for the telescope, but the MCP is close. We are within 0.5 degree or less of the stow position. 12:34Z MCP crashed again as we are writing up the night log and plug request. The telescope is stop buttoned so it should be stowed and ok. MCP gui shows that the MCP is unavailable. The verbal serious alert is going on.