Subject: 2.5m obslog 20210616 (SJD 59382) From: kgrabowski@apo.nmsu.edu Date: Thu, 17 Jun 2021 11:08:32 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Wednesday June 16, 2021 (SJD 59382) ---- Observing Team ---- Karen Kinemuchi Katie Grabowski Remote help for hub crash: Shane Thomas Jose Sanchez-Gallego ---- Observing Plan ---- Wed 16 Jun 2021 59382 0.41 02:53 10:32 13:32 21:13 --- KK MWM until 07:02 ( 4.1 h), then BHM ( 3.5 h) ---- Observing Summary ---- Lost 3 hours of BHM for hardware when the data disk crashed. Details in log section. MWM: ====== Cart.1, plate 15199,RV_K2EB_C06_314+48_btx, GSGOTF, 8xA+4xB, complete! Missing fibers: [211, 273] Faint fibers: ['4 - 5', 10, '31 - 60', '121 - 150', '152 - 154', 156, 158, '161 - 162', 169, '176 - 180', 227, '271 - 272', 276, '279 - 281', 283, '286 - 287', 289, '291 - 295', 299] ----------------- Cart.2, plate 15272, RV_011+47_btx, Dec+10", 6xA+3xB, not done Missing fibers: [38, 57, 59, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 142', '145 - 150', 153, 156, '161 - 164', '166 - 173', '175 - 177', '179 - 180', 197] ----------------- Cart.3, plate 15357, RV_071+62_btx, GSOGTF, 6xA+3xB, not done Missing fibers: [211, 273] Faint fibers: ['121 - 150', '153 - 154', 156, '158 - 159', '161 - 162', 173, '176 - 179'] ----------------- Cart.4, plate 15370, TESS_019, (-10",-10"), 4xA+2xB, complete! Missing fibers: [20, 39, 151, 186, 193, 211, 241, 273, 278] Faint fibers: ['121 - 129', '131 - 142', '145 - 150', '152 - 156', '158 - 173', '175 - 183', 185, '187 - 192', '194 - 197', '199 - 202', '205 - 208', 210] ----------------- BHM: ======= Cart.9, plate 15403- mounted but not observed due to data disk crash. Cart.8, plate 15029, AQM_334.95_+00.90, GSOGTF, 6xA+3xB, >92% complete! Missing fibers: [35, 184, 211, 227, 273] Faint fibers: ['121 - 180', '271 - 272', '274 - 276', '279 - 300'] ----------------- NOTES TO DAY STAFF AND DEVELOPERS: ================================= * Dust=7k * /home disk is VERY FULL. Please clean: Filesystem Size Used Avail Use% Mounted on /dev/vdd 688G 654G 34G 96% /home 10.25.1.230:/data 21T 13T 7.7T 62% /data Hub crashed after mounting cart.9 on the telescope. Shane is working to fix it remotely. Our new SDSS5 server suffered a temperature event and shut itself off, the host that was running the VM SDSS5_Data. This is the VM that we mount from for the sdss4-hub. I was able to restart that machine remotely via the Dell IDRAC and then bring up the services on that. This brought back /data on the sdss4-hub *without* a reboot! We are not sure how to resurrect STUI. Contacted Jose. May need to restart hub. They are working on this. Got everything back. More details in log section. Jose attempted to restart APOGEE QL. Not sure whether this is the version Nathan had restarted recently. APOGEE QL isn't updating. ---- Observing Log ---- 22:42Z Cloudy on site. AFTERNOON CHECKOUT: =================== 23:34Z Discovered a netburner problem upon logging into STUI. Reloaded controllers and boss actor pinged back ok. See error message in the Problems in detail section. 23:40Z Restarted podium STUI. Started APOGEE evening cals. Grabbed mirror numbers - ok; guider/gcamera prep - done BOSS SPECTROGRAPH CALS: 23:54:34 boss exposure 0.0, file=333189 23:55:57 boss exposure 20.0, file=333190 SoS is alive. BOSS COLLIMATION: 23:58:28 boss exposure 4.0, file=333191 23:59:39 boss exposure 4.0, file=333192 00:01:05 sp1.motor.move= 1159, 1159, 1159 00:01:05 Hartmann collimate output on cart #8 sp1: offset: r= 329 (In Focus); b= -6.2 (Out of focus) sp1: pred. move: spAverageMove= 1160 sp1: pred. resid: r= -831, b= -2.6, txt= OK, spTemp = 11.0 Blue ring was not adjusted. It is in spec. motorPosition sp1: -2984, -2991, -2992; ---------- Median spectrographs temperatures: Tsp1= 11.0 Outside air temperature= 24.4; BOSS CALS: 00:02:27 boss exposure 25.0, file=333193 00:05:14 boss exposure 4.0, file=333194 00:23Z Evening cals finished. 00:45Z Ventilation begins. 00:50Z Cart bay ready. Telescope ready. Disk check: df -kh Filesystem Size Used Avail Use% Mounted on /dev/vda2 7.7G 6.3G 1.1G 86% / tmpfs 3.4G 0 3.4G 0% /dev/shm /dev/vda1 194M 133M 52M 73% /boot /dev/vdb 19G 8.8G 9.1G 49% /exports/hub-local /dev/vdd 688G 654G 34G 96% /home /dev/vdc1 21T 13T 7.7T 62% /data 10.25.1.230:/data 21T 13T 7.7T 62% /data /dev/loop2 7.7G 6.4G 914M 88% /mnt/backup /dev/loop1 194M 133M 52M 73% /mnt/backup/boot /dev/loop0 19G 8.8G 9.1G 49% /mnt/backup/exports/hub-local NIGHT: ====== 00:41:25 loadCart: Cart 1, Plate 15199; 00:41:25 survey: BHM&MWM; lead: MWM lead 02:17:51 encl25m : open; clear 02:22:57 apogee.expose=apRaw-38200012, 10, A, Dark 02:24:46 apogee.expose=apRaw-38200013, 10, A, Dark 02:37:30 boss exposure 4.0, file=333195 02:38:08 boss exposure 4.0, file=333196 02:38:52 sp1.motor.move= 2480, 2480, 2480 02:38:52 Hartmann collimate output on cart #1 sp1: offset: r= 2968 (Out of focus); b= -6.2 (Out of focus) sp1: pred. move: spAverageMove= 2480 sp1: pred. resid: r= 488, b= 1.5, txt= OK, spTemp = 10.7 02:39:12 boss exposure 4.0, file=333197 02:40:39 boss exposure 25.0, file=333198 02:40:51 OBSERVER.karen 100 sop e text="Failed to take guider flat" repeated guider dark and flat. GSOGTF, seeing ~1.7", clear. 02:51:21 apogee.expose=apRaw-38200014, 46, A, Object 02:51:28 boss exposure 900.0, file=333199 02:59:31 apogee.expose=apRaw-38200015, 46, A, Object 03:07:42 apogee.expose=apRaw-38200016, 46, A, Object 03:07:48 boss exposure 900.0, file=333200 03:15:51 apogee.expose=apRaw-38200017, 46, A, Object 03:24:02 apogee.expose=apRaw-38200018, 46, A, Object 03:24:09 boss exposure 900.0, file=333201 03:32:12 apogee.expose=apRaw-38200019, 46, A, Object 03:40:22 apogee.expose=apRaw-38200020, 46, A, Object 03:40:29 boss exposure 900.0, file=333202 03:48:32 apogee.expose=apRaw-38200021, 46, A, Object 03:56:55 apogee.expose=apRaw-38200022, 5, A, DomeFlat 03:58:45 apogee.expose=apRaw-38200023, 10, A, Dark 04:00:33 apogee.expose=apRaw-38200024, 10, A, Dark ---------------------------------------- flat= /data/apogee/utr_cdr/59382/apRaw-38200022.fits master= apRaw-21180043.fits nfibers= 300 Missing fibers: [211, 273] Faint fibers: ['4 - 5', 10, '31 - 60', '121 - 150', '152 - 154', 156, 158, '161 - 162', 169, '176 - 180', 227, '271 - 272', 276, '279 - 281', 283, '286 - 287', 289, '291 - 295', 299] ----------------- 04:02:33 loadCart: Cart 2, Plate 15272; 04:02:33 survey: BHM&MWM; lead: MWM lead 04:05:17 boss exposure 4.0, file=333203 04:05:58 boss exposure 4.0, file=333204 04:06:36 sp1.motor.move= -119, -120, -120 04:06:36 Hartmann collimate output on cart #2 sp1: offset: r= 659 (In Focus); b= 2.8 (In Focus) sp1: pred. move: spAverageMove= -120 sp1: pred. resid: r= 779, b= 2.4, txt= OK, spTemp = 9.5 04:06:58 boss exposure 4.0, file=333205 04:08:25 boss exposure 25.0, file=333206 No guide stars. Dec+10". Seeing ~1.6". Clear. 04:14:58 apogee.expose=apRaw-38200025, 46, A, Object 04:15:05 boss exposure 900.0, file=333207 04:23:07 apogee.expose=apRaw-38200026, 46, A, Object 04:31:19 apogee.expose=apRaw-38200027, 46, A, Object 04:31:26 boss exposure 900.0, file=333208 04:39:28 apogee.expose=apRaw-38200028, 46, A, Object 04:47:40 apogee.expose=apRaw-38200029, 46, A, Object 04:47:47 boss exposure 900.0, file=333209 04:55:49 apogee.expose=apRaw-38200030, 46, A, Object 05:04:14 apogee.expose=apRaw-38200031, 5, A, DomeFlat 05:05:56 apogee.expose=apRaw-38200032, 10, A, Dark 05:07:44 apogee.expose=apRaw-38200033, 10, A, Dark ---------------------------------------- flat= /data/apogee/utr_cdr/59382/apRaw-38200031.fits master= apRaw-21180043.fits nfibers= 300 Missing fibers: [38, 57, 59, 190, 198, 211, 259, 273] Faint fibers: ['61 - 90', '121 - 142', '145 - 150', 153, 156, '161 - 164', '166 - 173', '175 - 177', '179 - 180', 197] ----------------- 05:09:46 loadCart: Cart 3, Plate 15357; 05:09:46 survey: BHM&MWM; lead: MWM lead 05:12:52 boss exposure 4.0, file=333210 05:13:30 boss exposure 4.0, file=333211 05:14:13 sp1.motor.move= -2411, -2411, -2411 05:14:13 Hartmann collimate output on cart #3 sp1: offset: r= -2638 (Out of focus); b= 6.8 (Out of focus) sp1: pred. move: spAverageMove= -2411 sp1: pred. resid: r= -226, b= -0.7, txt= OK, spTemp = 9.0 05:14:31 boss exposure 4.0, file=333212 05:15:58 boss exposure 25.0, file=333213 GSOGTF, seeing ~1.5", clear sky. 05:18:16 apogee.expose=apRaw-38200034, 46, A, Object 05:18:22 boss exposure 900.0, file=333214 05:26:25 apogee.expose=apRaw-38200035, 46, A, Object 05:34:37 apogee.expose=apRaw-38200036, 46, A, Object 05:34:43 boss exposure 900.0, file=333215 05:42:46 apogee.expose=apRaw-38200037, 46, A, Object 05:50:57 apogee.expose=apRaw-38200038, 46, A, Object 05:51:03 boss exposure 900.0, file=333216 05:59:06 apogee.expose=apRaw-38200039, 46, A, Object 06:07:30 apogee.expose=apRaw-38200040, 5, A, DomeFlat 06:09:20 apogee.expose=apRaw-38200041, 10, A, Dark 06:11:08 apogee.expose=apRaw-38200042, 10, A, Dark ---------------------------------------- flat= /data/apogee/utr_cdr/59382/apRaw-38200040.fits master= apRaw-21180043.fits nfibers= 300 Missing fibers: [211, 273] Faint fibers: ['121 - 150', '153 - 154', 156, '158 - 159', '161 - 162', 173, '176 - 179'] ----------------- 06:14:25 boss exposure 4.0, file=333217 06:15:03 boss exposure 4.0, file=333218 06:15:42 sp1.motor.move= 350, 349, 350 06:15:42 Hartmann collimate output on cart #4 sp1: offset: r= -329 (In Focus); b= -3.2 (In Focus) sp1: pred. move: spAverageMove= 349 sp1: pred. resid: r= -678, b= -2.1, txt= OK, spTemp = 8.4 06:16:01 boss exposure 4.0, file=333219 06:17:28 boss exposure 25.0, file=333220 No guide stars. (-10",-10"). Seeing ~2". Clear. 06:20:47 apogee.expose=apRaw-38200043, 46, A, Object 06:20:54 boss exposure 900.0, file=333221 06:28:57 apogee.expose=apRaw-38200044, 46, A, Object 06:37:08 apogee.expose=apRaw-38200045, 46, A, Object 06:37:14 boss exposure 900.0, file=333222 06:45:17 apogee.expose=apRaw-38200046, 46, A, Object 06:53:43 apogee.expose=apRaw-38200047, 5, A, DomeFlat 06:55:25 apogee.expose=apRaw-38200048, 10, A, Dark 06:57:12 apogee.expose=apRaw-38200049, 10, A, Dark ---------------------------------------- flat= /data/apogee/utr_cdr/59382/apRaw-38200047.fits master= apRaw-21180043.fits nfibers= 300 Missing fibers: [20, 39, 151, 186, 193, 211, 241, 273, 278] Faint fibers: ['121 - 129', '131 - 142', '145 - 150', '152 - 156', '158 - 173', '175 - 183', 185, '187 - 192', '194 - 197', '199 - 202', '205 - 208', 210] ----------------- 06:57:43 loadCart: Cart 9, Plate 15403; 06:57:43 survey: BHM&MWM; lead: BHM lead 07:00:41 boss exposure 4.0, file=333223 07:01:19 boss exposure 4.0, file=333224 07:02:00 sp1.motor.move= -1925, -1925, -1925 07:02:00 Hartmann collimate output on cart #9 sp1: offset: r= -2308 (Out of focus); b= 4.8 (Out of focus) sp1: pred. move: spAverageMove= -1925 sp1: pred. resid: r= -383, b= -1.2, txt= OK, spTemp = 8.3 STUI disconnected before cals completed. Hub crashed. 07:03:25 OBSERVER.kgrabow 176 stui F Timeout; Actor='hub'; Cmd="version" 07:03:25 .stui 0 stui Failing; Text='Connection is dead' 07:03:25 .stui 0 stui Failed; Text='Connection is dead' Closed safely and normally. MCP/TPM still working normally. One of the computers on the second rack in the computer room didn't have a green power button on the right hand side and showed a small orange thermometer bulb on the left hand side instead of a blue light. Shane is checking the status of things remotely. Sent an email regarding the temperature event. Temperature bulb disappeared, showing a blue light. Called Jose, who said that there is a tron instance running but I cannot find it (a ps aux of the sdss4-hub machine doesn't show it running but I can telnet to port 9877) Shane has to unmount on every VM and remount the /data drive. Jose thinks that /data going down took with it all the processes that write to it. He's been trying to kill all the actors in an attempt to let the hub connections close and try to reopen them, but now all the processes are defunct. May have to restart the hub. They are working on this. 9:36Z- Petunia and SOS back. Reconnected TCC to STUI. Jose will attempt to restart APOGEE QL. Not sure whether this is the version Nathan had restarted recently. Reloaded BOSS controllers to get the DAQ back. 09:26:24 apogee.expose=apRaw-38200050, 3, A, Dark 09:32:48 apogee.expose=apRaw-38200051, 10, A, Dark BOSS bias ok ---------------------------------------- 09:33:11 loadCart: Cart 8, Plate 15029; 09:33:11 survey: BHM&MWM; lead: BHM lead 09:34:36 apogee.expose=apRaw-38200052, 10, A, Dark 09:38:33 encl25m : open; 09:42:05 boss exposure 4.0, file=333227 09:42:43 boss exposure 4.0, file=333228 09:43:24 sp1.motor.move= 1343, 1341, 1343 09:43:24 Hartmann collimate output on cart #8 sp1: offset: r= 1979 (Out of focus); b= -2.2 (In Focus) sp1: pred. move: spAverageMove= 1343 sp1: pred. resid: r= 636, b= 2.0, txt= OK, spTemp = 8.8 09:43:43 boss exposure 4.0, file=333229 09:45:10 boss exposure 25.0, file=333230 GSOGTF, seeing ~1.8", clear. 09:52:54 apogee.expose=apRaw-38200053, 46, A, Object 09:53:01 boss exposure 900.0, file=333231 10:01:05 apogee.expose=apRaw-38200054, 46, A, Object 10:09:14 apogee.expose=apRaw-38200055, 46, A, Object 10:09:21 boss exposure 900.0, file=333232 0:17:24 apogee.expose=apRaw-38200056, 46, A, Object 10:25:34 apogee.expose=apRaw-38200057, 46, A, Object 10:25:41 boss exposure 900.0, file=333233 10:33:44 apogee.expose=apRaw-38200058, 46, A, Object 10:42:06 apogee.expose=apRaw-38200059, 5, A, DomeFlat 10:43:42 apogee.expose=apRaw-38200060, 10, A, Dark 10:45:31 apogee.expose=apRaw-38200061, 10, A, Dark ---------------------------------------- [observer@sdss4-hub ~]$ aptest 59382 38200059 flat= /data/apogee/utr_cdr/59382/apRaw-38200059.fits master= apRaw-21180043.fits nfibers= 300 Missing fibers: [35, 184, 211, 227, 273] Faint fibers: ['121 - 180', '271 - 272', '274 - 276', '279 - 300'] ----------------- 10:48:48 loadCart: Cart 9, Plate 15403; 10:48:48 survey: BHM&MWM; lead: BHM lead 10:52:34 encl25m : closed; 10:55Z- Started APOGEE and BOSS cals. ---- BOSS Spectrographs Collimation Summary ---- --- Hartmann --- Time Cart TSP1 R1 B1 Move1 B1Resid ================================================================================ 00:01 8-15029A 11.0 329 -6.2 1160 -2.6 02:39 1-15199A 10.7 2968 -6.2 2480 1.5 04:07 2-15272A 9.5 659 2.8 -120 2.4 05:14 3-15357A 9.0 -2638 6.8 -2411 -0.7 06:16 4-15370A 8.4 -329 -3.2 349 -2.1 07:02 9-15403A 8.3 -2308 4.8 -1925 -1.2 09:44 8-15029A 8.8 1979 -2.2 1343 2.0 ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: -4.9250 -4.9240 Axial B: -4.6880 -4.6800 Axial C: -5.7750 -5.7660 Trans D: -0.0003 -0.0009 Lateral E: 0.3745 0.3745 Lateral F: 0.5710 0.5745 GALILS Commanded: 5850 -4500 14850 28200 25200 24900 Actual: 5850 -4500 14850 28200 25200 24900 SETMIR VALUES PrimDesOrient: 0.0 69.89 100.0 375.3 622.11 PrimOrient: 0.04 69.89 100.0 375.3 622.11 SECONDARY: ---------- Focus: 0.0000 Air temp: 24.72 Altitude: 30.00005840078799 MIGS TONIGHT NOMINAL Axial A: -9.4510 -9.4480 Axial B: -9.4150 -9.4120 Axial C: -9.5030 -9.5050 Trans D: 0.0470 0.0390 GALILS Commanded: 4624900 4607200 4675700 -2000 -6600 Actual: 4604680 4587269 4633076 -2000 -6600 SETMIR VALUES SecDesOrient: 3657.0 -14.01 -2.9 12.5 114.22 SecOrient: 3657.82 -13.82 -2.66 12.85 114.09 ---- Telescope Offsets and Scale ---- --- Offsets --- (arcsec) Time Cart Az Alt Rot objOff guideRot calibOff guideRMS ================================================================================ 02:51 1-15199A -2.0 43.4 178.1 (4.9,17.4) -1.9 ( 0, 0, 0) 0.584 03:08 1-15199A -7.5 43.1 173.4 (6.4,15.9) -34.2 ( 0, 0, 0) 0.461 03:24 1-15199A -12.9 42.5 168.8 (7.4,15.1) -27.9 ( 0, 0, 0) 0.346 03:34 1-15199A -16.2 42.0 165.9 (7.9,13.9) -18.0 ( 0, 0, 0) 0.446 03:40 1-15199A -18.1 41.6 164.3 (8.3,13.5) -17.2 ( 0, 0, 0) 0.300 03:57 1-15199A -23.1 40.4 160.0 (8.9,12.4) -11.4 ( 0, 0, 0) 0.333 04:15 2-15272A 19.7 62.7 196.4 (-8.2,18.5) -82.7 ( 0, 0, 0) 0.410 04:31 2-15272A 10.9 63.6 189.0 (-5.6,18.3) -107.1 ( 0, 0, 0) 0.378 04:48 2-15272A 1.7 64.0 181.3 (-2.9,17.3) -99.9 ( 0, 0, 0) 0.385 05:00 2-15272A -5.2 63.9 175.4 (-1.1,16.0) -98.5 ( 0, 0, 0) 0.417 05:18 3-15357A 234.8 73.6 65.4 (2.9,-13.9) 27.9 ( 0, 0, 0) 0.619 05:35 3-15357A 238.5 70.7 71.6 (5.6,-13.4) 29.5 ( 0, 0, 0) 0.385 05:51 3-15357A 240.8 67.7 76.5 (7.0,-12.5) 33.4 ( 0, 0, 0) 0.344 06:07 3-15357A 242.3 64.7 80.6 (7.4,-11.8) 34.3 ( 0, 0, 0) 0.352 06:21 4-15370A 178.1 45.9 -7.5 (-10.2,-16.5) 38.8 ( 0, 0, 0) 0.564 06:37 4-15370A 179.4 46.0 -2.5 (-9.2,-17.3) 56.8 ( 0, 0, 0) 0.276 06:53 4-15370A 180.8 46.0 2.5 (-7.8,-17.9) 62.6 ( 0, 0, 0) 0.410 07:04 9-15403A 999.0 99.0 999.0 (0.0,0.0) 0.0 ( 0, 0, 0) 99.999 09:09 99-9999Z 999.0 99.0 999.0 (0.0,0.0) 0.0 ( 0, 0, 0) 99.999 09:53 8-15029A 44.3 49.2 215.9 (-2.7,10.8) 6.0 ( 0, 0, 0) 0.587 10:09 8-15029A 38.9 51.6 211.8 (-1.4,10.4) -19.6 ( 0, 0, 0) 0.485 10:26 8-15029A 33.1 53.6 207.2 (-0.3,10.4) -29.5 ( 0, 0, 0) 0.394 ---- Focus Log ---- --- Focus --- Time Cart Scale M1 M2 Focus Az Alt Temp Wind Dir FWHM ================================================================================ 02:51 1-15199A 400.0 -3950 237 -96 -2.0 43.4 18.0 20 111 1.8 03:08 1-15199A 403.0 -3975 258 -53 -7.5 43.1 17.7 17 111 2.7 03:24 1-15199A 409.0 -4042 246 -14 -12.9 42.5 17.6 13 87 2.0 03:34 1-15199A 404.0 -3985 302 -7 -16.2 42.0 17.3 17 81 1.9 03:40 1-15199A 409.0 -4036 270 1 -18.1 41.6 17.2 11 92 1.8 03:57 1-15199A 415.0 -4093 242 16 -23.1 40.4 17.0 15 95 1.8 04:15 2-15272A 373.0 -3682 588 91 19.7 62.7 16.6 14 94 1.8 04:31 2-15272A 374.0 -3688 586 97 10.9 63.6 16.3 14 110 1.7 04:48 2-15272A 368.0 -3631 644 108 1.7 64.0 16.0 22 112 1.7 05:00 2-15272A 360.0 -3548 711 107 -5.2 63.9 15.9 17 111 1.8 05:18 3-15357A 364.0 -3593 618 68 234.8 73.6 15.6 23 110 1.4 05:35 3-15357A 392.0 -3867 390 62 238.5 70.7 15.5 22 119 1.4 05:51 3-15357A 401.0 -3956 325 65 240.8 67.7 15.3 21 114 1.4 06:07 3-15357A 401.0 -3959 320 57 242.3 64.7 15.2 23 111 1.9 06:21 4-15370A 331.0 -3268 946 57 178.1 45.9 15.2 19 112 2.0 06:37 4-15370A 332.0 -3277 911 31 179.4 46.0 15.1 29 124 2.0 06:53 4-15370A 332.0 -3274 896 14 180.8 46.0 14.9 27 124 2.0 07:04 9-15403A 0.0 9999 9999 9999 999.0 99.0 -99.0 99 124 99.9 09:09 99-9999Z 0.0 9999 9999 9999 999.0 99.0 -99.0 99 -99 99.9 09:53 8-15029A 230.0 -2271 1667 -36 44.3 49.2 14.8 7 88 2.5 10:09 8-15029A 180.0 -1780 2027 -73 38.9 51.6 14.3 5 105 1.9 10:26 8-15029A 169.0 -1669 2124 -62 33.1 53.6 13.9 8 92 1.7 ---- Telescope Status ---- 00:45Z Started ventilation 02:15Z encl25m : open 10:52Z encl25m : closed No interlocks bypassed Status at:11:02Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: cartridge = 9 Counterweights at: 481 484 481 480 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 117 lbs, 21.1 psi ---- Software Used ---- Get Versions, 06/16/21, 23:38:57 -- STUI software: STUI : 1.7.12 alerts : trunk apo : trunk boss : unknown -boss.specMec : 3.0.4 gcamera : v2_2 guider : 3.8.1 hartmann : 1.8.0 hub : Tag: v4.2 mcp : 2.0-7-g8799 -mcp.plc : 34 -mcp.azFiduci : v1_145 -mcp.altFiduc : v1_145 -mcp.rotFiduc : v1_145 platedb : 2.8.0 sop : 4.0.8 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : unknown STUIAdditions : 3.1.6 -- done -- Home Disk Hogs (GB): muna 132.068 eboss 83.1765 manga 66.812 sdss3 27.1813 apogee 17.4802 logging 13.6955 Filesystem Size Used Avail Use% Mounted on /dev/vdd 688G 654G 34G 96% /home 10.25.1.230:/data 21T 13T 7.7T 62% /data Setting up: idlspec2d Flavor: Linux64 Version: v5_7_12 Setting up: |elodie Flavor: Linux64 Version: v1_3 Setting up: |spinspect Flavor: Linux64 Version: trunk Setting up: |specflat Flavor: Linux64 Version: v1_20 Setting up: |speclog Flavor: Linux64 Version: trunk Setting up: |platelist Flavor: Linux64 Version: trunk Setting up: |sdss_python_module Flavor: Linux64 Version: v1_6_7 Setting up: |sqlalchemy Flavor: Linux64 Version: 0.9.7 Setting up: |psycopg2 Flavor: Linux64 Version: 2.5.3 Setting up: |idlutils Flavor: Linux64 Version: v5_5_16 Setting up: |idl Flavor: Linux64 Version: 7.1 Setting up: |dust Flavor: Linux64 Version: v0_1 Setting up: |photolog Flavor: Linux64 Version: trunk Setting up: actorcore Flavor: Linux64 Version: v4_1_6 Setting up: |opscore Flavor: Linux64 Version: v2_5 Setting up: |external Flavor: Linux64 Version: v1_2_0 Setting up: |actorkeys Flavor: Linux64 Version: trunk Setting up: |plc Flavor: Linux64 Version: v30_2 Setting up: |sdsstools Flavor: Linux64 Version: v4_33 Setting up: |sdss3tools Flavor: Linux64 Version: v1_7_13 Setting up: |RO Flavor: Linux64 Version: 3.6.9 -- Other versions ---- idlspec2d : v5_7_12 plate-mapper3 : NOCVS:v7_0_9 petunia : 2.2.3 autoscheduler : git sdss_python_module: trunk mangadrp : trunk exported ---- APOGEE Data Summary ---- [observer@sdss4-hub ~]$ list_ap -m 59381 -morning APOGEE data list, mjd=59381, morning raw_data: /data/apogee/utr_cdr/59381/apRaw-*.fits archive: /data/apogee/archive/59381/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59381/ap2D-a-*.fits.fz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 59381 1 10:58 38190080 Dark 60 A 13.0 8-15029 a-b-c - - 59381 2 11:08 38190081 Dark 60 A 13.0 8-15029 a-b-c - - 59381 3 11:19 38190082 Dark 60 A 13.0 8-15029 a-b-c - - 59381 4 11:30 38190083 QuarFlat 10 A 13.0 8-15029 a-b-c - 49 59381 5 11:32 38190084 QuarFlat 10 A 13.0 8-15029 a-b-c - 49 59381 6 11:34 38190085 QuarFlat 10 A 13.0 8-15029 a-b-c - 49 59381 7 11:35 38190086 Arc-Thar 12 A 13.0 8-15029 a-b-c -1.36 - 59381 8 11:38 38190087 Arc-Thar 12 A 13.0 8-15029 a-b-c -1.36 - 59381 9 11:40 38190088 Arc-Une 40 A 13.0 8-15029 a-b-c - - 59381 10 11:47 38190089 Arc-Une 40 A 13.0 8-15029 a-b-c - - 59381 11 11:54 38190090 Dark 30 A 13.0 8-15029 a-b-c - - 59381 12 11:59 38190091 InteFlat 30 A 13.0 8-15029 a-b-c - 0 59381 13 12:05 38190092 InteFlat 30 A 13.0 8-15029 a-b-c - 0 59381 14 12:10 38190093 InteFlat 30 A 13.0 8-15029 a-b-c - 0 59381 15 12:15 38190094 Dark 30 A 13.0 8-15029 a-b-c - - -------------------------------------------------------------------------------- APOGEE data list, mjd=59382, all raw_data: /data/apogee/utr_cdr/59382/apRaw-*.fits archive: /data/apogee/archive/59382/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59382/ap2D-a-*.fits.fz -------------------------------------------------------------------------------- mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm -------------------------------------------------------------------------------- 59382 1 21:57 38200001 QuarFlat 3 A 13.0 8-15029 a-b-c - 20 59382 2 22:12 38200002 QuarFlat 3 A 13.0 8-15029 a-b-c - 20 59382 3 23:02 38200003 QuarFlat 3 A 13.0 8-15029 a-b-c - 26 59382 4 23:39 38200004 Dark 3 A 13.0 8-15029 a-b-c - - 59382 5 23:40 38200005 Dark 60 A 13.0 8-15029 a-b-c - - 59382 6 23:51 38200006 QuarFlat 10 A 13.0 8-15029 a-b-c - 55 59382 7 23:53 38200007 Arc-Thar 12 A 13.0 8-15029 a-b-c -1.38 - 59382 8 23:55 38200008 Arc-Thar 12 A 13.0 8-15029 a-b-c -1.39 - 59382 9 23:57 38200009 Arc-Une 40 A 13.0 8-15029 a-b-c - - 59382 10 00:04 38200010 Arc-Une 40 A 13.0 8-15029 a-b-c - - 59382 11 00:12 38200011 Dark 60 A 13.0 8-15029 a-b-c - - 59382 12 02:22 38200012 Dark 10 A 13.0 1-15199 a-b-c - - 59382 13 02:24 38200013 Dark 10 A 13.0 1-15199 a-b-c - - 59382 14 02:51 38200014 Object 46 A 13.0 1-15199 a-b-c -1.33 - 59382 15 02:59 38200015 Object 46 A 13.0 1-15199 a-b-c -1.34 - 59382 16 03:07 38200016 Object 46 A 13.0 1-15199 a-b-c -1.35 - 59382 17 03:15 38200017 Object 46 A 13.0 1-15199 a-b-c -1.33 - 59382 18 03:23 38200018 Object 46 A 13.0 1-15199 a-b-c -1.35 - 59382 19 03:31 38200019 Object 46 A 13.0 1-15199 a-b-c -1.34 - 59382 20 03:40 38200020 Object 46 A 13.0 1-15199 a-b-c -1.33 - 59382 21 03:48 38200021 Object 46 A 13.0 1-15199 a-b-c -1.34 - 59382 22 03:56 38200022 DomeFlat 5 A 13.0 1-15199 a-b-c - 31 59382 23 03:58 38200023 Dark 10 A 13.0 1-15199 a-b-c - - 59382 24 04:00 38200024 Dark 10 A 13.0 1-15199 a-b-c - - 59382 25 04:14 38200025 Object 46 A 13.0 2-15272 a-b-c -1.32 - 59382 26 04:22 38200026 Object 46 A 13.0 2-15272 a-b-c -1.32 - 59382 27 04:31 38200027 Object 46 A 13.0 2-15272 a-b-c -1.34 - 59382 28 04:39 38200028 Object 46 A 13.0 2-15272 a-b-c -1.33 - 59382 29 04:47 38200029 Object 46 A 13.0 2-15272 a-b-c -1.34 - 59382 30 04:55 38200030 Object 46 A 13.0 2-15272 a-b-c -1.33 - 59382 31 05:04 38200031 DomeFlat 5 A 13.0 2-15272 a-b-c - 46 59382 32 05:05 38200032 Dark 10 A 13.0 2-15272 a-b-c - - 59382 33 05:07 38200033 Dark 10 A 13.0 2-15272 a-b-c - - 59382 34 05:18 38200034 Object 46 A 13.0 3-15357 a-b-c -1.34 - 59382 35 05:26 38200035 Object 46 A 13.0 3-15357 a-b-c -1.34 - 59382 36 05:34 38200036 Object 46 A 13.0 3-15357 a-b-c -1.35 - 59382 37 05:42 38200037 Object 46 A 13.0 3-15357 a-b-c -1.34 - 59382 38 05:50 38200038 Object 46 A 13.0 3-15357 a-b-c -1.33 - 59382 39 05:58 38200039 Object 46 A 13.0 3-15357 a-b-c -1.32 - 59382 40 06:07 38200040 DomeFlat 5 A 13.0 3-15357 a-b-c - 42 59382 41 06:09 38200041 Dark 10 A 13.0 3-15357 a-b-c - - 59382 42 06:10 38200042 Dark 10 A 13.0 3-15357 a-b-c - - 59382 43 06:20 38200043 Object 46 A 13.0 4-15370 a-b-c -1.33 - 59382 44 06:28 38200044 Object 46 A 13.0 4-15370 a-b-c -1.33 - 59382 45 06:36 38200045 Object 46 A 13.0 4-15370 a-b-c -1.34 - 59382 46 06:45 38200046 Object 46 A 13.0 4-15370 a-b-c -1.34 - 59382 47 06:53 38200047 DomeFlat 5 A 13.0 4-15370 a-b-c - 45 59382 48 06:55 38200048 Dark 10 A 13.0 4-15370 a-b-c - - 59382 49 06:57 38200049 Dark 10 A 13.0 4-15370 a-b-c - - 59382 50 09:26 38200050 Dark 3 A 13.0 9-15403 a-b-c - - 59382 51 09:32 38200051 Dark 10 A 13.0 9-15403 a-b-c - - 59382 52 09:34 38200052 Dark 10 A 13.0 8-15029 a-b-c - - 59382 53 09:52 38200053 Object 46 A 13.0 8-15029 a-b-c -1.36 - 59382 54 10:00 38200054 Object 46 A 13.0 8-15029 a-b-c -1.38 - 59382 55 10:09 38200055 Object 46 A 13.0 8-15029 a-b-c -1.38 - 59382 56 10:17 38200056 Object 46 A 13.0 8-15029 a-b-c -1.38 - 59382 57 10:25 38200057 Object 46 A 13.0 8-15029 a-b-c -1.39 - 59382 58 10:33 38200058 Object 46 A 13.0 8-15029 a-b-c -1.37 - 59382 59 10:41 38200059 DomeFlat 5 A 13.0 8-15029 a-b-c - 27 59382 60 10:43 38200060 Dark 10 A 13.0 8-15029 a-b-c - - 59382 61 10:45 38200061 Dark 10 A 13.0 8-15029 a-b-c - - 59382 62 11:00 38200062 Dark 60 A 13.0 ------- x-x-x - - -------------------------------------------------------------------------------- ---- MaNGA Data Summary ---- Reading FITS from /data/spectro/59382/ 1 02:37 b1-00333195 1 15199 arc Lef C 4.1 -2984 -2991 -2992 HART 02:37 r1-00333195 1 15199 arc Lef C 4.1 -2984 -2991 -2992 HART 2 02:38 b1-00333196 1 15199 arc Rig C 4.1 -2984 -2991 -2992 HART 02:38 r1-00333196 1 15199 arc Rig C 4.1 -2984 -2991 -2992 HART 3 02:39 b1-00333197 1 15199 arc Out C 4.1 -504 -511 -512 02:39 r1-00333197 1 15199 arc Out C 4.1 -504 -511 -512 4 02:40 b1-00333198 1 15199 fla Out C 25.1 -504 -511 -512 02:40 r1-00333198 1 15199 fla Out C 25.1 -504 -511 -512 5 02:51 b1-00333199 1 15199 sci Out C 900.1 -504 -511 -512 02:51 r1-00333199 1 15199 sci Out C 900.1 -504 -511 -512 6 03:07 b1-00333200 1 15199 sci Out C 900.1 -504 -511 -512 03:07 r1-00333200 1 15199 sci Out C 900.1 -504 -511 -512 7 03:24 b1-00333201 1 15199 sci Out C 900.1 -504 -511 -512 03:24 r1-00333201 1 15199 sci Out C 900.1 -504 -511 -512 8 03:40 b1-00333202 1 15199 sci Out C 900.1 -504 -511 -512 03:40 r1-00333202 1 15199 sci Out C 900.1 -504 -511 -512 9 04:05 b1-00333203 2 15272 arc Lef C 4.1 -504 -511 -512 HART 04:05 r1-00333203 2 15272 arc Lef C 4.1 -504 -511 -512 HART 10 04:05 b1-00333204 2 15272 arc Rig C 4.1 -504 -511 -512 HART 04:05 r1-00333204 2 15272 arc Rig C 4.1 -504 -511 -512 HART 11 04:06 b1-00333205 2 15272 arc Out C 4.1 -623 -631 -632 04:06 r1-00333205 2 15272 arc Out C 4.1 -623 -631 -632 12 04:08 b1-00333206 2 15272 fla Out C 25.1 -623 -631 -632 04:08 r1-00333206 2 15272 fla Out C 25.1 -623 -631 -632 13 04:15 b1-00333207 2 15272 sci Out C 900.1 -623 -631 -632 04:15 r1-00333207 2 15272 sci Out C 900.1 -623 -631 -632 14 04:31 b1-00333208 2 15272 sci Out C 900.1 -623 -631 -632 04:31 r1-00333208 2 15272 sci Out C 900.1 -623 -631 -632 15 04:47 b1-00333209 2 15272 sci Out C 900.1 -623 -631 -632 04:47 r1-00333209 2 15272 sci Out C 900.1 -623 -631 -632 16 05:12 b1-00333210 3 15357 arc Lef C 4.1 -623 -631 -632 HART 05:12 r1-00333210 3 15357 arc Lef C 4.1 -623 -631 -632 HART 17 05:13 b1-00333211 3 15357 arc Rig C 4.1 -623 -631 -632 HART 05:13 r1-00333211 3 15357 arc Rig C 4.1 -623 -631 -632 HART 18 05:14 b1-00333212 3 15357 arc Out C 4.1 -3034 -3042 -3043 05:14 r1-00333212 3 15357 arc Out C 4.1 -3034 -3042 -3043 19 05:15 b1-00333213 3 15357 fla Out C 25.1 -3034 -3042 -3043 05:15 r1-00333213 3 15357 fla Out C 25.1 -3034 -3042 -3043 20 05:18 b1-00333214 3 15357 sci Out C 900.1 -3034 -3042 -3043 05:18 r1-00333214 3 15357 sci Out C 900.1 -3034 -3042 -3043 21 05:34 b1-00333215 3 15357 sci Out C 900.1 -3034 -3042 -3043 05:34 r1-00333215 3 15357 sci Out C 900.1 -3034 -3042 -3043 22 05:51 b1-00333216 3 15357 sci Out C 900.1 -3034 -3042 -3043 05:51 r1-00333216 3 15357 sci Out C 900.1 -3034 -3042 -3043 23 06:14 b1-00333217 4 15370 arc Lef C 4.1 -3034 -3042 -3043 HART 06:14 r1-00333217 4 15370 arc Lef C 4.1 -3034 -3042 -3043 HART 24 06:15 b1-00333218 4 15370 arc Rig C 4.1 -3034 -3042 -3043 HART 06:15 r1-00333218 4 15370 arc Rig C 4.1 -3034 -3042 -3043 HART 25 06:16 b1-00333219 4 15370 arc Out C 4.1 -2684 -2693 -2693 06:16 r1-00333219 4 15370 arc Out C 4.1 -2684 -2693 -2693 26 06:17 b1-00333220 4 15370 fla Out C 25.1 -2684 -2693 -2693 06:17 r1-00333220 4 15370 fla Out C 25.1 -2684 -2693 -2693 27 06:20 b1-00333221 4 15370 sci Out C 900.1 -2684 -2693 -2693 06:20 r1-00333221 4 15370 sci Out C 900.1 -2684 -2693 -2693 28 06:37 b1-00333222 4 15370 sci Out C 900.1 -2684 -2693 -2693 06:37 r1-00333222 4 15370 sci Out C 900.1 -2684 -2693 -2693 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --- OK --- ---- Weather Log ---- --- Weather --- Time Cart Temp DP Diff Humid Wind Dir 1umDst IRSCSig IRSCm ================================================================================ 02:51 1-15199A 18.0 1.0 17.0 31 20 111 1545 4.6 329.0 03:08 1-15199A 17.7 0.7 17.0 31 17 111 1621 7.7 329.0 03:24 1-15199A 17.6 0.5 17.1 31 13 87 1599 10.4 329.0 03:34 1-15199A 17.3 0.5 16.8 32 17 81 1540 5.8 328.0 03:40 1-15199A 17.2 0.4 16.8 32 11 92 1698 5.8 328.0 03:57 1-15199A 17.0 0.5 16.5 32 15 95 1669 3.3 328.0 04:15 2-15272A 16.6 0.5 16.1 33 14 94 1681 4.1 329.0 04:31 2-15272A 16.3 0.7 15.6 34 14 110 1649 4.2 305.0 04:48 2-15272A 16.0 0.7 15.3 35 22 112 1562 2.3 329.0 05:00 2-15272A 15.9 0.9 15.0 36 17 111 1533 2.4 329.0 05:18 3-15357A 15.6 0.9 14.7 36 23 110 1587 2.1 329.0 05:35 3-15357A 15.5 0.7 14.8 36 22 119 1378 2.9 329.0 05:51 3-15357A 15.3 0.7 14.6 37 21 114 1510 2.6 329.0 06:07 3-15357A 15.2 0.9 14.3 37 23 111 1354 2.7 329.0 06:21 4-15370A 15.2 1.2 14.0 38 19 112 1369 2.5 329.0 06:37 4-15370A 15.1 1.5 13.6 39 29 124 1391 2.6 329.0 06:53 4-15370A 14.9 2.0 12.9 41 27 124 1308 2.8 329.0 07:04 9-15403A -99.0 -99.0 0.0 999 99 124 9999 999.0 329.0 09:09 99-9999Z -99.0 -99.0 0.0 999 99 -99 9999 999.0 999.0 09:53 8-15029A 14.8 1.5 13.3 40 7 88 1085 2.6 329.0 10:09 8-15029A 14.3 1.6 12.7 42 5 105 1019 2.1 329.0 10:26 8-15029A 13.9 2.2 11.7 44 8 92 1030 2.3 329.0 ---- Problems in Detail ---- 23:34:09 OBSERVER.karen 8 hub f NoTarget="the target named ecamera is not connected" 23:34:10 OBSERVER.karen 7 boss f text="command failed: error('Invalid response from Netburner.',) at /home/sdss4/products/Linux64/bossICC/5.0.1/python/bossICC/DAQInterface.py:1443" 23:34:10 CmdDone: OBSERVER.karen 7 boss ping