Subject: 2.5m obslog 20211208 (SJD 59557) From: kinemuchi@apo.nmsu.edu Date: Thu, 9 Dec 2021 12:36:21 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Wednesday December 08, 2021 (SJD 59557) ---- Observing Team ---- Katie Grabowski Karen Kinemuchi FPS Commissioning Team ====================== Rick Poggee Conor Sayres Jose Sanchez-Gallego Juna Kollmeier ---- Observing Plan ---- FPS Commissioning ---- Observing Summary ---- We remained closed due to clouds, but we completed a bunch of FPS tests: Kaike's flat fields altitude vs rotator positions Rick's test with incremental rotator moves Jose's configuration test, includes a few Holtzman fields Conor's robotcalib test We had a tricky robot collision recovery during Conor's test. See log for details and the process we used to get the robots to fold home. ---- Observing Log ---- AFTERNOON: ========== Mostly cloudy. All systems alive and responsive. Telescope currently at 121,30, 135.4 Jose was working with FVC and illuminating fibers 22:29:29 boss exposure 0.0, file=335266 22:31:16 boss exposure 20.0, file=335267 22:32Z- Started APOGEE cals 03:59:48 boss exposure 4.0, file=335270 00:00:29 boss exposure 4.0, file=335271 00:01:13 sp1.motor.move= 2641, 2640, 2639 00:01:13 Hartmann collimate output on cart #19 sp1: offset: r= 2968 (Out of focus); b= -7.3 (Out of focus) sp1: pred. move: spAverageMove= 2640 sp1: pred. resid: r= 327, b= 1.0, txt= OK, spTemp = 5.0 00:02:15 boss exposure 4.0, file=335272 00:04:30 boss exposure 25.0, file=335273 Fiducials and APOGEE cals ok. Delayed ventilation due to clouds. NIGHT: ======= 02:03Z Running Kaike's flat field test for altitude vs rotator. At altitude = 35: Rotator=0 02:09:06 boss exposure 30.0, file=335274 02:09:17 apogee.expose=apRaw-39950011, 5, A, DomeFlat Rotator=60 02:11:11 apogee.expose=apRaw-39950012, 5, A, DomeFlat 02:11:15 boss exposure 30.0, file=335275 Rotator = 120 02:13:19 apogee.expose=apRaw-39950013, 5, A, DomeFlat 02:13:29 boss exposure 30.0, file=335276 Rotator = 180 02:15:38 apogee.expose=apRaw-39950014, 5, A, DomeFlat 02:15:48 boss exposure 30.0, file=335277 Rotator = 240 02:18:07 apogee.expose=apRaw-39950015, 5, A, DomeFlat 02:18:23 boss exposure 30.0, file=335278 Rotator = 300 02:20:27 apogee.expose=apRaw-39950016, 5, A, DomeFlat 02:20:37 boss exposure 30.0, file=335279 Altitude = 45 Rotator = 300 02:22:26 apogee.expose=apRaw-39950017, 5, A, DomeFlat 02:22:35 boss exposure 30.0, file=335280 Rotator=240 02:24:34 apogee.expose=apRaw-39950018, 5, A, DomeFlat 02:24:44 boss exposure 30.0, file=335281 Rotator=180 02:28:03 apogee.expose=apRaw-39950019, 5, A, DomeFlat 02:28:15 boss exposure 30.0, file=335282 Rotator=120 02:30:18 apogee.expose=apRaw-39950020, 5, A, DomeFlat 02:30:33 boss exposure 30.0, file=335283 Rotator=60 02:32:36 apogee.expose=apRaw-39950021, 5, A, DomeFlat 02:32:45 boss exposure 30.0, file=335284 Rotator=0 02:34:41 apogee.expose=apRaw-39950022, 5, A, DomeFlat 02:34:52 boss exposure 30.0, file=335285 Altitude=55 Rotator=0 02:36:32 apogee.expose=apRaw-39950023, 5, A, DomeFlat 02:36:47 boss exposure 30.0, file=335286 Rotator=60 02:38:36 apogee.expose=apRaw-39950024, 5, A, DomeFlat 02:38:49 boss exposure 30.0, file=335287 Rotator=120 02:40:37 apogee.expose=apRaw-39950025, 5, A, DomeFlat 02:40:52 boss exposure 30.0, file=335288 Rotator=180 02:42:40 apogee.expose=apRaw-39950026, 5, A, DomeFlat 02:42:50 boss exposure 30.0, file=335289 Rotator=240 02:44:40 apogee.expose=apRaw-39950027, 5, A, DomeFlat 02:44:57 boss exposure 30.0, file=335290 Rotator=300 02:47:02 apogee.expose=apRaw-39950028, 5, A, DomeFlat 02:47:15 boss exposure 30.0, file=335291 Altitude=65 Rotator=300 02:36:32 apogee.expose=apRaw-39950029, 5, A, DomeFlat 02:36:47 boss exposure 30.0, file=335292 Rotator=240 02:38:36 apogee.expose=apRaw-39950030, 5, A, DomeFlat 02:38:49 boss exposure 30.0, file=335293 Rotator=180 02:40:37 apogee.expose=apRaw-39950031, 5, A, DomeFlat 02:40:52 boss exposure 30.0, file=335294 Rotator=120 02:42:40 apogee.expose=apRaw-39950032, 5, A, DomeFlat 02:42:50 boss exposure 30.0, file=335295 Rotator=60 02:44:40 apogee.expose=apRaw-39950033, 5, A, DomeFlat 02:44:57 boss exposure 30.0, file=335296 Rotator=0 02:47:02 apogee.expose=apRaw-39950034, 5, A, DomeFlat 02:47:15 boss exposure 30.0, file=335297 Altitude=75 Rotator=0 02:36:32 apogee.expose=apRaw-39950035, 5, A, DomeFlat 02:36:47 boss exposure 30.0, file=335298 Rotator=60 02:38:36 apogee.expose=apRaw-39950036, 5, A, DomeFlat 02:38:49 boss exposure 30.0, file=335299 Rotator=120 02:40:37 apogee.expose=apRaw-39950037, 5, A, DomeFlat 02:40:52 boss exposure 30.0, file=335300 Rotator=180 02:42:40 apogee.expose=apRaw-39950038, 5, A, DomeFlat 02:42:50 boss exposure 30.0, file=335301 Rotator=240 02:44:40 apogee.expose=apRaw-39950039, 5, A, DomeFlat 02:44:57 boss exposure 30.0, file=335302 Rotator=300 02:47:02 apogee.expose=apRaw-39950040, 5, A, DomeFlat 02:47:15 boss exposure 30.0, file=335303 Altitude=85 Rotator=300 02:36:32 apogee.expose=apRaw-39950041, 5, A, DomeFlat 02:36:47 boss exposure 30.0, file=335304 Rotator=240 02:38:36 apogee.expose=apRaw-39950042, 5, A, DomeFlat 02:38:49 boss exposure 30.0, file=335305 Rotator=180 02:40:37 apogee.expose=apRaw-39950043, 5, A, DomeFlat 02:40:52 boss exposure 30.0, file=335306 Rotator=120 02:42:40 apogee.expose=apRaw-39950044, 5, A, DomeFlat 02:42:50 boss exposure 30.0, file=335307 Rotator=60 02:44:40 apogee.expose=apRaw-39950045, 5, A, DomeFlat 02:44:57 boss exposure 30.0, file=335308 Rotator=0 02:47:02 apogee.expose=apRaw-39950046, 5, A, DomeFlat 02:47:15 boss exposure 30.0, file=335309 Moved telescope to 121,30,135 for more FVC tests At this position, data file #0001 Rick's test: 03:41Z Moved telescope to stow position (121,30, rot=0). Starting test. file #0002 03:44:16 tcc track 121,30 mount /rota=5 /rotty=mount file #0003 03:45:05 tcc track 121,30 mount /rota=10 /rotty=mount file #0004 03:46:01 tcc track 121,30 mount /rota=15 /rotty=mount file #0005 03:47:02 tcc track 121,30 mount /rota=20 /rotty=mount file #0006 03:47:50 tcc track 121,30 mount /rota=25 /rotty=mount file #0007 03:48:31 tcc track 121,30 mount /rota=30 /rotty=mount file #0008 03:49:32 tcc track 121,30 mount /rota=45 /rotty=mount file #0009 03:50:40 tcc track 121,30 mount /rota=60 /rotty=mount file #0010 03:51:39 tcc track 121,30 mount /rota=90 /rotty=mount file #0011 03:52:42 tcc track 121,30 mount /rota=135 /rotty=mount file #0012 03:53:51 tcc track 121,30 mount /rota=180 /rotty=mount file #0013 03:55:05 tcc track 121,30 mount /rota=225 /rotty=mount file #0014 03:56:14 tcc track 121,30 mount /rota=270 /rotty=mount file #0015 03:57:21 tcc track 121,30 mount /rota=315 /rotty=mount file #0016 03:58:31 tcc track 121,30 mount /rota=345 /rotty=mount file #0017 03:59:20 tcc track 121,30 mount /rota=355 /rotty=mount file #0018 APOGEE auxiliary back illuminator was installed in the gang connector port on the FPS. Metrology fibers illuminated. 5 sec exposure. file #0019. File #0020: LED #4 - APOGEE only fibers 6% intensity. 04:21Z LED #1,2,4 on at 6% intensity. File #0021 Metrology + APOGEE. 04:24Z File #0022, increase to 16% for BOSS. Turn off LED #4 and adjust. File #0023 - APOGEE only, 5 second exptime, 16% brightness. File #0024 - APOGEE+Metrology, exptime=5s, 6% and 16% brightness in LEDs. 04:32Z Stack 10, 5 second dark. File #0025. 06:07Z Conor test with FVC (running script robotcalib on jaeger). Scripted stopped around 06:57Z with a collision of #1136 on iteration 15 of the script. Visually inspected the snapshot, was ok. Restarted jaeger actor, jaeger reload, jaeger unwind and robots were ok. 07:00Z Restarting robotcalib script on sdss5@sdss5-fps machine. 07:26Z Stopped script at iteration 9 to prepare for Jose's test. 07:39Z Jose tests on configurations Checking a configuration and clearance of the robots, identify if the RMS values are increasing or not. Using Design 5. Assigned targets 296 07:51:50 jaeger configuration load 5 07:53:48 jaeger unwind 07:55:20 jaeger configuration execute 07:57:36 jaeger fvc loop Looking in the output for RMS values of between 40 and 80. Will stop iterating when RMS is less than 5 or delta is less than 1. If RMS starts increasing, the loop will stop. The goal is to see if the RMS values decrease with each iteration, and ideally be less than 10. 08:01:27 jaeger configuration reverse 08:01:27 OBSERVER.kgrabow 103 jaeger f error="Trajectory validation failed." 08:01:27 CmdDone: OBSERVER.kgrabow 103 jaeger configuration reverse If this error occurs, issue: 08:02:29 jaeger unwind 08:04:07 jaeger configuration load 840 08:05:57 jaeger configuration execute 08:06:50 jaeger fvc loop 08:10:54 jaeger unwind Restarting jaeger actor Checking jaeger version: 08:12:54 jaeger version Output: 08:12:54 OBSERVER.kgrabow 113 jaeger i version=0.12.1-alpha.0 08:12:54 OBSERVER.kgrabow 113 jaeger i kaiju_version=1.2.0-alpha.2 08:12:54 OBSERVER.kgrabow 113 jaeger i coordio_version=1.1.4-alpha.2 08:12:54 OBSERVER.kgrabow 113 jaeger i fps_calibrations_version=APO.2021.12.04+1310ea8 08:13:38 jaeger configuration load 999 08:15:57 jaeger configuration execute 08:17:03 jaeger fvc loop 08:19:34 OBSERVER.kgrabow 117 jaeger f error="Failed processing image: Cannot apply corrections. 2 robots are collided." To recover from this, use the jaeger explode command: 08:20:07 jaeger explode 5 Watch the snapshot PDF image to see which robots were collided. The explode will gently offset the robots to a comfortable position to uncollide themselves. Sending all the robots back to home position and will try another configuration. We will repeat this test. 08:21:22 jaeger unwind Trying another new APO configuration: 08:22:41 jaeger configuration load 120 08:24:04 jaeger configuration execute 08:25:09 jaeger fvc loop Note that this loop will iterate at least 5 times to see if there is a convergence of the RMS values. Now repeating this process - watching the RMS values converge to a small value. These data will be analyzed by Jose to assess how well the robots are arriving to their design positions. For success, look for this comment: 08:28:25 OBSERVER.kgrabow 123 jaeger : text="FVC target RMS reached." Repeat: 08:28:45 jaeger unwind 08:30:16 jaeger configuration load 8 (one of Holtzman's fields) 08:31:21 jaeger configuration execute 08:32:26 jaeger fvc loop 08:34:47 jaeger unwind 08:36:22 jaeger configuration load 9 08:38:10 jaeger configuration execute 08:39:02 jaeger fvc loop 08:42:32 jaeger unwind RMS values: 49.409, 16.945,7.794,5.083, 5.155 08:43:32 jaeger configuration load 92 08:45:23 CmdStarted: OBSERVER.kgrabow 135 jaeger configuration execute 08:46:24 CmdStarted: OBSERVER.kgrabow 136 jaeger fvc loop 08:49:58 CmdStarted: OBSERVER.kgrabow 137 jaeger unwind RMS values: 74.028, 41.318, 16.624, 13.778, 12.338 08:51:18 CmdStarted: OBSERVER.kgrabow 138 jaeger configuration load 666 08:52:13 CmdStarted: OBSERVER.kgrabow 139 jaeger configuration execute 08:52:58 CmdStarted: OBSERVER.kgrabow 140 jaeger fvc loop 08:56:34 CmdStarted: OBSERVER.kgrabow 141 jaeger unwind RMS values:52.553, 18.682, 10.568, 8.926 08:58:10 CmdStarted: OBSERVER.kgrabow 143 jaeger configuration load 13 08:59:24 CmdStarted: OBSERVER.kgrabow 144 jaeger configuration execute 09:00:25 CmdStarted: OBSERVER.kgrabow 145 jaeger fvc loop 09:01:47 CmdStarted: OBSERVER.kgrabow 146 jaeger unwind This is one of Holtzman's fields. RMS values: 48.375, 56.148 Jose: The FVC loop aborted early. 09:03:01 CmdStarted: OBSERVER.kgrabow 147 jaeger configuration load 88 09:04:48 CmdStarted: OBSERVER.kgrabow 148 jaeger configuration execute 09:05:38 CmdStarted: OBSERVER.kgrabow 149 jaeger fvc loop 09:08:08 CmdStarted: OBSERVER.kgrabow 150 jaeger unwind RMS values: 53.412, 35.063, 34.358 09:08:09 OBSERVER.kgrabow 150 jaeger f error="Command 'unwind' failed because of an uncaught error 'RuntimeError: Grid is Kaiju-collided, cannot generate paths'. See traceback in the log for more information." Checking snapshot - fps_snapshot_59557_0111.pdf We will attempt an explode command: 09:10:56 CmdStarted: OBSERVER.kgrabow 153 jaeger explode 5 09:11:23 CmdStarted: OBSERVER.kgrabow 154 jaeger unwind 09:11:45 OBSERVER.kgrabow 154 jaeger f error="Failed calculating trajectory: Failed generating a valid unwind trajectory. 6 deadlocks were found" 09:12:22 CmdStarted: OBSERVER.kgrabow 155 jaeger explode 5 09:13:05 OBSERVER.kgrabow 156 jaeger f error="Failed calculating trajectory: Failed generating a valid unwind trajectory. 8 deadlocks were found" Going to try to do an unwind force 09:13:53 CmdStarted: OBSERVER.kgrabow 157 jaeger unwind --force 09:15:29 CmdStarted: OBSERVER.kgrabow 158 jaeger unwind 09:15:29 OBSERVER.kgrabow 158 jaeger f error="Command 'unwind' failed because of an uncaught error 'RuntimeError: Grid is Kaiju-collided, cannot generate paths'. See traceback in the log for more information." 09:15:38 CmdStarted: OBSERVER.kgrabow 159 jaeger explode 5 09:17:33 CmdStarted: OBSERVER.kgrabow 160 jaeger unwind We see in the snapshot PDFs the deadlocked robots. See PDF #114 and #115. Went looking for Conor. The commands we are using are not allowing us to send the robots back to home position. See problems in detail on how we got 8 deadlocked robots out of deadlock. Now we continue with Jose's test. 09:42:03 CmdStarted: OBSERVER.kgrabow 174 jaeger configuration load 311 09:43:56 CmdStarted: OBSERVER.kgrabow 175 jaeger configuration execute 09:44:44 CmdStarted: OBSERVER.kgrabow 176 jaeger fvc loop 09:48:24 CmdStarted: OBSERVER.kgrabow 177 jaeger unwind RMS values: 59.977, 21.226, 10.109, 7.831,6.618 09:50Z We decided to get Conor's test done tonight, so stopping the jaeger actor and starting the script (robotcalib --niter 100 --danger --met) on sdss5@sdss-fps. To watch the output, when logged into sdss5@sdss-fps, type: tmux a -t robotloop 12:07Z Script died at Iteration #54. Last image was under iteration #54, file proc-fimg-fvc1n-0157.fits Run commands on sdss5@sdss-fps: jaeger unwind - unable to do so, so now: jaeger unlock - no error messages. jaeger unwind. Checked snapshot image, and all robots homed fine. Snapshot file numbers were #131-213. We have collected roughly 73 iterations (non consecutively, and through 3 runnings of this script). During this last running of the script, the first file number is 117 and the last file number is 157. We are stopping this script at this point and take morning cals, as requested by Jose. BOSS bias: 12:14:42 boss exposure 0.0, file=335310 12:17Z jaeger power off Command didn't work on sdss5-fps command line. Restarted jaeger and turned everything off through STUI. 12:28Z Giving up. Started morning cals. Telescope back at stow position. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0040 0.0040 Axial B: 0.0040 0.0030 Axial C: -0.1160 -0.1150 Trans D: -0.0340 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5345 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 4.61 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6560 -9.6570 Axial B: -9.6680 -9.6710 Axial C: -9.6960 -9.6970 Trans D: 0.1190 0.1140 GALILS Commanded: 4891000 4889350 4905650 -50 -3450 Actual: 4859419 4859091 4859180 -50 -3450 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3857.01 0.01 -0.11 -0.07 42.86 ---- Telescope Status ---- Never opened No interlocks bypassed Status at: 12:29Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 201 lbs, 20.3 psi ---- Software Used ---- Filesystem Size Used Avail Use% Mounted on devtmpfs 16G 0 16G 0% /dev tmpfs 16G 8.0K 16G 1% /dev/shm tmpfs 16G 1.3M 16G 1% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/mapper/centos-root 50G 40G 11G 79% / /dev/sda1 1014M 238M 777M 24% /boot /dev/sdb1 2.0T 272G 1.8T 14% /home tmpfs 3.2G 0 3.2G 0% /run/user/2005 10.25.1.230:/data 21T 17T 3.2T 85% /data overlay 50G 40G 11G 79% /var/lib/docker/overlay2/2fc82a212d7b05767e9dcf9122c0af446f9144f66730227739ab2d409681017f/merged tmpfs 3.2G 4.0K 3.2G 1% /run/user/3021 tmpfs 3.2G 0 3.2G 0% /run/user/1002 -- STUI software: STUI : 1.7.13 alerts : 2.1.6 apo : 3.0.1 boss : 6.0.1-alpha.0 -boss.specMec : 3.0.4 guider : 4.0.0 ecamera : FAILED fliswarm : FAILED hartmann : 2.0.1-alpha.0 hub : 5.0.0a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 sop : 0.1.0-alpha.0 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.9 ---- APOGEE Data Summary ---- APOGEE data list, mjd=59557, all raw_data: /data/apogee/utr_cdr/59557/apRaw-*.fits archive: /data/apogee/archive/59557/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59557/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59557 1 22:23 39950001 Dark 3 A 13.0 ------- a-b-c - - 59557 2 22:42 39950002 Dark 60 A 13.0 ------- a-b-c - - 59557 3 22:44 39950003 QuarFlat 10 A 13.0 19- 0 a-b-c - 102 59557 4 22:47 39950004 Arc-Thar 12 A 13.0 19- 0 a-b-c 4.77 - 59557 5 22:54 39950005 Arc-Une 40 A 13.0 19- 0 a-b-c - - 59557 6 22:56 39950006 Arc-Thar 12 B 13.5 19- 0 a-b-c 5.39 - 59557 7 23:03 39950007 Arc-Une 40 B 13.5 19- 0 a-b-c - - 59557 8 23:14 39950008 Dark 60 A 13.0 ------- a-b-c - - 59557 9 23:26 39950009 Arc---- 20 A 13.0 19- 0 a-b-c - - FPI 59557 10 23:32 39950010 Arc---- 30 A 13.0 19- 0 a-b-c - - FPI 59557 11 02:09 39950011 DomeFlat 5 A 13.0 19- 0 a-b-c - 974 59557 12 02:11 39950012 DomeFlat 5 A 13.0 19- 0 a-b-c - 980 59557 13 02:13 39950013 DomeFlat 5 A 13.0 19- 0 a-b-c - 981 59557 14 02:16 39950014 DomeFlat 5 A 13.0 19- 0 a-b-c - 982 59557 15 02:18 39950015 DomeFlat 5 A 13.0 19- 0 a-b-c - 981 59557 16 02:20 39950016 DomeFlat 5 A 13.0 19- 0 a-b-c - 980 59557 17 02:22 39950017 DomeFlat 5 A 13.0 19- 0 a-b-c - 979 59557 18 02:25 39950018 DomeFlat 5 A 13.0 19- 0 a-b-c - 978 59557 19 02:28 39950019 DomeFlat 5 A 13.0 19- 0 a-b-c - 976 59557 20 02:30 39950020 DomeFlat 5 A 13.0 19- 0 a-b-c - 977 59557 21 02:33 39950021 DomeFlat 5 A 13.0 19- 0 a-b-c - 977 59557 22 02:35 39950022 DomeFlat 5 A 13.0 19- 0 a-b-c - 976 59557 23 02:37 39950023 DomeFlat 5 A 13.0 19- 0 a-b-c - 974 59557 24 02:39 39950024 DomeFlat 5 A 13.0 19- 0 a-b-c - 977 59557 25 02:41 39950025 DomeFlat 5 A 13.0 19- 0 a-b-c - 977 59557 26 02:43 39950026 DomeFlat 5 A 13.0 19- 0 a-b-c - 975 59557 27 02:45 39950027 DomeFlat 5 A 13.0 19- 0 a-b-c - 978 59557 28 02:47 39950028 DomeFlat 5 A 13.0 19- 0 a-b-c - 975 59557 29 02:53 39950029 DomeFlat 5 A 13.0 19- 0 a-b-c - 982 59557 30 02:55 39950030 DomeFlat 5 A 13.0 19- 0 a-b-c - 976 59557 31 02:57 39950031 DomeFlat 5 A 13.0 19- 0 a-b-c - 977 59557 32 02:59 39950032 DomeFlat 5 A 13.0 19- 0 a-b-c - 977 59557 33 03:02 39950033 DomeFlat 5 A 13.0 19- 0 a-b-c - 977 59557 34 03:04 39950034 DomeFlat 5 A 13.0 19- 0 a-b-c - 977 59557 35 03:05 39950035 DomeFlat 5 A 13.0 19- 0 a-b-c - 970 59557 36 03:08 39950036 DomeFlat 5 A 13.0 19- 0 a-b-c - 974 59557 37 03:10 39950037 DomeFlat 5 A 13.0 19- 0 a-b-c - 978 59557 38 03:12 39950038 DomeFlat 5 A 13.0 19- 0 a-b-c - 977 59557 39 03:14 39950039 DomeFlat 5 A 13.0 19- 0 a-b-c - 976 59557 40 03:16 39950040 DomeFlat 5 A 13.0 19- 0 a-b-c - 980 59557 41 03:19 39950041 DomeFlat 5 A 13.0 19- 0 a-b-c - 979 59557 42 03:21 39950042 DomeFlat 5 A 13.0 19- 0 a-b-c - 983 59557 43 03:23 39950043 DomeFlat 5 A 13.0 19- 0 a-b-c - 981 59557 44 03:25 39950044 DomeFlat 5 A 13.0 19- 0 a-b-c - 981 59557 45 03:27 39950045 DomeFlat 5 A 13.0 19- 0 a-b-c - 982 59557 46 03:29 39950046 DomeFlat 5 A 13.0 19- 0 a-b-c - 978 59557 47 12:29 39950047 Dark 60 A 13.0 ------- x-x-x - - ------------------------------------------------------------------------------ ---- eBOSS Data Summary ---- Reading FITS from /data/spectro//59557 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename crt plt flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 19:43 b1-00335261 FPS-N None bia Clo 0.0 -3187 -3186 -3200 2 19:43 r1-00335261 FPS-N None bia Clo 0.0 -3187 -3186 -3200 3 19:48 b1-00335262 FPS-N None bia Clo 0.0 -3187 -3186 -3200 4 19:48 r1-00335262 FPS-N None bia Clo 0.0 -3187 -3186 -3200 5 20:12 b1-00335263 FPS-N None bia Clo 0.0 -3187 -3186 -3200 6 20:12 r1-00335263 FPS-N None bia Clo 0.0 -3187 -3186 -3200 7 20:16 b1-00335264 FPS-N None bia Clo 0.0 -3187 -3186 -3200 8 20:16 r1-00335264 FPS-N None bia Clo 0.0 -3187 -3186 -3200 9 20:34 b1-00335265 FPS-N None bia Clo 0.0 -3187 -3186 -3200 10 20:34 r1-00335265 FPS-N None bia Clo 0.0 -3187 -3186 -3200 11 22:29 b1-00335266 FPS-N None bia Clo 0.0 -3187 -3186 -3200 12 22:29 r1-00335266 FPS-N None bia Clo 0.0 -3187 -3186 -3200 13 22:31 r1-00335267 FPS-N None dar Clo 20.0 -3187 -3186 -3200 14 22:31 b1-00335267 FPS-N None dar Clo 20.0 -3187 -3186 -3200 15 23:58 r1-00335268 FPS-N None arc Lef 4.1 -3187 -3186 -3200 HART 16 23:58 b1-00335268 FPS-N None arc Lef 4.1 -3187 -3186 -3200 HART 17 23:58 b1-00335269 FPS-N None arc Rig 4.0 -3187 -3186 -3200 HART 18 23:58 r1-00335269 FPS-N None arc Rig 4.0 -3187 -3186 -3200 HART 19 23:59 b1-00335270 FPS-N None arc Lef 4.1 -3187 -3186 -3200 HART 20 23:59 r1-00335270 FPS-N None arc Lef 4.1 -3187 -3186 -3200 HART 21 00:00 b1-00335271 FPS-N None arc Rig 4.1 -3187 -3186 -3200 HART 22 00:00 r1-00335271 FPS-N None arc Rig 4.1 -3187 -3186 -3200 HART 23 00:02 b1-00335272 FPS-N None arc Out 4.1 -546 -546 -561 24 00:02 r1-00335272 FPS-N None arc Out 4.1 -546 -546 -561 25 00:04 b1-00335273 FPS-N None fla Out 25.1 -546 -546 -561 26 00:04 r1-00335273 FPS-N None fla Out 25.1 -546 -546 -561 27 02:09 b1-00335274 FPS-N None fla Out 30.1 -546 -546 -561 28 02:09 r1-00335274 FPS-N None fla Out 30.1 -546 -546 -561 29 02:11 r1-00335275 FPS-N None fla Out 30.1 -546 -546 -561 30 02:11 b1-00335275 FPS-N None fla Out 30.1 -546 -546 -561 31 02:13 b1-00335276 FPS-N None fla Out 30.1 -546 -546 -561 32 02:13 r1-00335276 FPS-N None fla Out 30.1 -546 -546 -561 33 02:15 b1-00335277 FPS-N None fla Out 30.1 -546 -546 -561 34 02:15 r1-00335277 FPS-N None fla Out 30.1 -546 -546 -561 35 02:18 b1-00335278 FPS-N None fla Out 30.1 -546 -546 -561 36 02:18 r1-00335278 FPS-N None fla Out 30.1 -546 -546 -561 37 02:20 b1-00335279 FPS-N None fla Out 30.1 -546 -546 -561 38 02:20 r1-00335279 FPS-N None fla Out 30.1 -546 -546 -561 39 02:22 b1-00335280 FPS-N None fla Out 30.1 -546 -546 -561 40 02:22 r1-00335280 FPS-N None fla Out 30.1 -546 -546 -561 41 02:24 r1-00335281 FPS-N None fla Out 30.1 -546 -546 -561 42 02:24 b1-00335281 FPS-N None fla Out 30.1 -546 -546 -561 43 02:28 b1-00335282 FPS-N None fla Out 30.1 -546 -546 -561 44 02:28 r1-00335282 FPS-N None fla Out 30.1 -546 -546 -561 45 02:30 r1-00335283 FPS-N None fla Out 30.1 -546 -546 -561 46 02:30 b1-00335283 FPS-N None fla Out 30.1 -546 -546 -561 47 02:32 b1-00335284 FPS-N None fla Out 30.1 -546 -546 -561 48 02:32 r1-00335284 FPS-N None fla Out 30.1 -546 -546 -561 49 02:34 b1-00335285 FPS-N None fla Out 30.1 -546 -546 -561 50 02:34 r1-00335285 FPS-N None fla Out 30.1 -546 -546 -561 51 02:36 r1-00335286 FPS-N None fla Out 30.1 -546 -546 -561 52 02:36 b1-00335286 FPS-N None fla Out 30.1 -546 -546 -561 53 02:38 r1-00335287 FPS-N None fla Out 30.1 -546 -546 -561 54 02:38 b1-00335287 FPS-N None fla Out 30.1 -546 -546 -561 55 02:40 r1-00335288 FPS-N None fla Out 30.1 -546 -546 -561 56 02:40 b1-00335288 FPS-N None fla Out 30.1 -546 -546 -561 57 02:42 r1-00335289 FPS-N None fla Out 30.1 -546 -546 -561 58 02:42 b1-00335289 FPS-N None fla Out 30.1 -546 -546 -561 59 02:44 r1-00335290 FPS-N None fla Out 30.1 -546 -546 -561 60 02:44 b1-00335290 FPS-N None fla Out 30.1 -546 -546 -561 61 02:47 r1-00335291 FPS-N None fla Out 30.1 -546 -546 -561 62 02:47 b1-00335291 FPS-N None fla Out 30.1 -546 -546 -561 63 02:52 r1-00335292 FPS-N None fla Out 30.1 -546 -546 -561 64 02:52 b1-00335292 FPS-N None fla Out 30.1 -546 -546 -561 65 02:54 r1-00335293 FPS-N None fla Out 30.1 -546 -546 -561 66 02:54 b1-00335293 FPS-N None fla Out 30.1 -546 -546 -561 67 02:57 r1-00335294 FPS-N None fla Out 30.1 -546 -546 -561 68 02:57 b1-00335294 FPS-N None fla Out 30.1 -546 -546 -561 69 02:59 b1-00335295 FPS-N None fla Out 30.1 -546 -546 -561 70 02:59 r1-00335295 FPS-N None fla Out 30.1 -546 -546 -561 71 03:01 b1-00335296 FPS-N None fla Out 30.1 -546 -546 -561 72 03:01 r1-00335296 FPS-N None fla Out 30.1 -546 -546 -561 73 03:03 b1-00335297 FPS-N None fla Out 30.1 -546 -546 -561 74 03:03 r1-00335297 FPS-N None fla Out 30.1 -546 -546 -561 75 03:05 b1-00335298 FPS-N None fla Out 30.1 -546 -546 -561 76 03:05 r1-00335298 FPS-N None fla Out 30.1 -546 -546 -561 77 03:07 r1-00335299 FPS-N None fla Out 30.1 -546 -546 -561 78 03:07 b1-00335299 FPS-N None fla Out 30.1 -546 -546 -561 79 03:10 r1-00335300 FPS-N None fla Out 30.1 -546 -546 -561 80 03:10 b1-00335300 FPS-N None fla Out 30.1 -546 -546 -561 81 03:12 r1-00335301 FPS-N None fla Out 30.1 -546 -546 -561 82 03:12 b1-00335301 FPS-N None fla Out 30.1 -546 -546 -561 83 03:14 r1-00335302 FPS-N None fla Out 30.1 -546 -546 -561 84 03:14 b1-00335302 FPS-N None fla Out 30.1 -546 -546 -561 85 03:16 r1-00335303 FPS-N None fla Out 30.1 -546 -546 -561 86 03:16 b1-00335303 FPS-N None fla Out 30.1 -546 -546 -561 87 03:18 r1-00335304 FPS-N None fla Out 30.1 -546 -546 -561 88 03:18 b1-00335304 FPS-N None fla Out 30.1 -546 -546 -561 89 03:20 r1-00335305 FPS-N None fla Out 30.1 -546 -546 -561 90 03:20 b1-00335305 FPS-N None fla Out 30.1 -546 -546 -561 91 03:22 b1-00335306 FPS-N None fla Out 30.1 -546 -546 -561 92 03:22 r1-00335306 FPS-N None fla Out 30.1 -546 -546 -561 93 03:24 b1-00335307 FPS-N None fla Out 30.1 -546 -546 -561 94 03:24 r1-00335307 FPS-N None fla Out 30.1 -546 -546 -561 95 03:26 b1-00335308 FPS-N None fla Out 30.1 -546 -546 -561 96 03:26 r1-00335308 FPS-N None fla Out 30.1 -546 -546 -561 97 03:29 b1-00335309 FPS-N None fla Out 30.1 -546 -546 -561 98 03:29 r1-00335309 FPS-N None fla Out 30.1 -546 -546 -561 99 12:14 b1-00335310 FPS-N None bia Clo 0.0 -546 -546 -561 100 12:14 r1-00335310 FPS-N None bia Clo 0.0 -546 -546 -561 ---- Problems in Detail ---- While running Jose's test, we had 8 deadlocked robots that prevented us to completely home all the robots. In two cases, there was a cluster of 3 robots that were collided and unable to return to home position. On collision only had 2 offending robots. Thus 8 deadlocked robots. The standard commands of jaeger explode 5 jaeger unwind gave us errors (see main log for those error messages). We grabbed Conor before he departed, and he helped us on wiggling free three robots so that the deadlock problem disappeared. Here is what we did: 1) Open the latest snapshot PDF file in /data/logs/jaeger/snapshots/ and look for any red robots, which are the collided ones 2) Identify which robot that needs to move away so that the other robots in a collision can safely fold itself back to home position. Try to decide how best to move that robot, in terms of the alpha angle. See Conor's reference paper to get a better understanding of the alpha and beta angles of rotation for the robot. 3) Once you have decided which robot to move, issue the following command. In our incident this evening, we identified robots #452, 979, 136 that needed to move out of the way. jaeger status 452 (find the line in the log window that says "position status - robot ID, alpha, beta, more numbers...." jaeger goto 452 0 180 (0 is the desired alpha angle we wanted to move and 180 is the beta angle - this was determined from our snapshot PDF and for this robot case only) These commands solved the clearance problem of these 2 collided robots. Then we worked on the two sets of 3 collided robots. 4) From the snapshot, we identified robot #979 prevented free movement of two neighbor robots. We decided to do an explode command to ONLY this robot. To do this, type: jaeger explode --one 979 5 (979 is the robot ID and 5 is the degree of explosion) That command didn't move it much out of the way, so we decided to tell it to get out of the way of the other two robots. 5) Issued the command: jaeger goto 979 270 165 (based on the location of the robot in the snapshot, we determined we needed to put the robot at alpha angle=270 but left beta at 165) This cleared robot 979 out of the way of two other collided robots. For the last set of 3 collided robots, we used a different command, using a bigger explosion value 6) In this last collided set, we identified robot #136 needed to get out of the way of two neighbor robots. We issued the command: jaeger explode --one 136 20 This command moved the offending robot out of the way. 7) Now to fold all the collided robots back to home, we issued: jaeger unwind And all the robots returned home safely. To see the subtle moves we did, via trial and error, please see the corresponding snapshot PDFs #114-121. These PDFs are located in the directory: observer@sdss5-hub:/data/logs/jaeger/snapshots/59557