Subject: 2.5m obslog 20220118 (SJD 59598) From: kinemuchi@apo.nmsu.edu Date: Wed, 19 Jan 2022 11:18:30 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday January 18, 2022 (SJD 59598) ---- Observing Team ---- Dan Oravetz Karen Kinemuchi Jose Sanchez-Gallego (online) ---- Observing Plan ---- https://wiki.sdss.org/pages/viewpage.action?spaceKey=COM&title=Commissioning+Nightly+Plan#CommissioningNightlyPlan-TuesdayJanuary18 - FVC loop bug fix test - Trigger and document different 'jaeger configuration execute' failures - Continue dithers if clear ---- Observing Summary ---- Remained closed due to snow, clouds, and high humidity in the second half of the night. Site is in fog at the end of the night. Completed Cloudy night tests #1 and #2 Test #1: Performed FVC loops on Designs: 35919, 35938, 36060, 35988. Error message occurred for the first 3 designs: error="Command 'fvc loop' failed because of an uncaught error 'TypeError: cannot pickle 'Command' object'. See traceback in the log for more information." After restart of jaeger actor, the message disappeared for 35988. Test #2: Completed loading 47 different designs 35 completed successfully with no error messages 12 designs produced errors (see below) Successful field design ids were: 35936,35961,35966,35967,35968,35969,35970, 35971,35972,35974,35975,35976,35977,35978,35979,35980,35981,35982,35984,35985, 35986, 35988, 35989, 35991,35992,35993,35994,35996,35997,36000,36002,36004, 36005,36006,36008 Fields with errors (see log for more details): Design 35964: Same error message for a lot of the robots, e.g.: 02:41:04 .jaeger 0 jaeger w text="Positioner 16 replied to SEND_TRAJECTORY_DATA UID=4300 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 02:39:47 error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" Design 35965: Error: 02:47:48 .jaeger 0 jaeger w text="Positioner 302 replied to SEND_TRAJECTORY_DATA UID=4448 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 02:47:48 APO.doravetz_2 194 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" Design 35983: Error: Positioner 162 replied to SEND_TRAJECTORY_DATA UID=10403 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 03:16:26 APO.doravetz_2 228 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" 03:18:20 .jaeger 0 jaeger w text="Found positioners with different firmware versions. (JaegerUserWarning)" 03:18:21 APO.doravetz_2 231 jaeger f error="Initialisation failed: Some positioners failed to initialise: Positioner 170: failed initialising." Design 35983: Error: Failed generating paths: Attempt 6: 5 deadlocks remain but the number of retries has been exhausted A collision was detected in positioner 565. Sending SEND_TRAJECTORY_ABORT and locking the FPS. "Positioner 1329 replied to SEND_TRAJECTORY_DATA UID=21786 with 'VALUE_OUT_OF_RANGE'. Design 35987: Error: A collision was detected in positioner 608. Sending SEND_TRAJECTORY_ABORT and locking the FPS. error="Failed calculating trajectory: Failed generating a valid unwind trajectory. 2 deadlocks were found" Design 35990: Error: A collision was detected in positioner 608. Sending SEND_TRAJECTORY_ABORT and locking the FPS. Design 35995: Error: error="Failed generating paths: Attempt 6: 2 deadlocks remain but the number of retries has been exhausted" Design 35998: Error: A collision was detected in positioner 608. Sending SEND_TRAJECTORY_ABORT and locking the FPS." error="Command 'unwind' failed because of an uncaught error 'RuntimeError: Grid is Kaiju-collided, cannot generate paths'. See traceback in the log for more information." Positioner 765 replied to SEND_TRAJECTORY_DATA UID=4734 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" --> see also Problems in Details section on our analysis on this design. Design 35999: Error: A collision was detected in positioner 664. Sending SEND_TRAJECTORY_ABORT and locking the FPS." Design 36001: Error: ext="Positioner 762 replied to SEND_TRAJECTORY_DATA UID=3448 with 'VALUE_OUT_OF_RANGE'. --> see log for more details on this one. Design 36003: Error: A collision was detected in positioner 608. Sending SEND_TRAJECTORY_ABORT and locking the FPS." Positioner 1170 replied to SEND_TRAJECTORY_DATA UID=3717 with 'VALUE_OUT_OF_RANGE'. Design 36007: Error: A collision was detected in positioner 608. Sending SEND_TRAJECTORY_ABORT and locking the FPS." NOTES ===== (1) Not sure if we have a formal threshold these days for /data, but as of this afternoon it's currently at 95% with 1.2T remaining. (2) See Problems in Detail for what we discovered behind the error messages we saw during Test #2. (3) If a design fails after 6 attempts and you get X number of deadlocks, run the jaeger configuration load # command again. (4) Boson now has a feature where the snapshots will appear as they are produced. Very helpful in identifying collided robots and determining angles for the jaeger goto commands. (5) Note from Jose to observers: For acceptable criteria on the FVC loop's RMS full fit values --> 06:34:39 APO.Jose: I'd say if it's 10-15 after the first loop just leave it there. If it's >20 do a second loop and after that accept whatever except if it's >30. See nightlog around 6:30Z for this discussion. (5) We did not get an INVALID_TRAJECTORY error this evening. (6) We noticed on grafana that the robot air went up to 25C, but all other temps appeared to be fine. ---- Observing Log ---- ============================================================================ AFTERNOON CHECKOUT ============================================================================ Clouds+precipitation heading our direction from the west. All systems alive and responsive. The FPS was already powered on upon arrival. APOGEE evening cals and BOSS cals taken. Positioners are homed: 23:12:12 CmdStarted: APO.Jose 13 jaeger unwind --status 23:12:12 APO.Jose 13 jaeger > 23:12:14 APO.Jose 13 jaeger i folded=T Ventilation delayed due to approaching weather system. ============================================================================ NIGHT ============================================================================ 01:36Z Jose has handed off the telescope to us, so now we will commence with the evening's cloudy night tasks. Starting with the Test #1 on the commissioning page: 01:41:12 jaeger configuration load 35919 --scale 0.999882 01:42:39 Config Loaded: 35919-2273 01:43:07 jaeger configuration execute 01:44:07 jaeger fvc loop Received an error: 01:46:55 APO.doravetz_2 143 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'TypeError: cannot pickle 'Command' object'. See traceback in the log for more information." 01:46:55 CmdDone: APO.doravetz_2 143 jaeger fvc loop Last RMS value before this error was: 01:46:49 RMS full fit 8.980 um." 01:47:33 jaeger fvc loop 01:48:00 RMS full fit 9.100 um." 01:48:44 RMS full fit 8.920 um." Received the same error: 01:48:50 APO.doravetz_2 144 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'TypeError: cannot pickle 'Command' object'. See traceback in the log for more information." 01:48:50 CmdDone: APO.doravetz_2 144 jaeger fvc loop Try again: 01:49:10 jaeger fvc loop 01:49:38 RMS full fit 8.900 um." 01:50:22 RMS full fit 8.890 um." Same error: 01:50:27 APO.doravetz_2 145 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'TypeError: cannot pickle 'Command' object'. See traceback in the log for more information." 01:50:27 CmdDone: APO.doravetz_2 145 jaeger fvc loop Moving on to another configuration. 01:50:42 jaeger configuration reverse 01:51:58 jaeger configuration load 35938 --scale 0.999882 01:53:21 Config Loaded: 35938-2274 01:53:33 jaeger configuration execute Tried to issue fvc loop and got this error: 01:54:26 APO.doravetz_2 149 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'IndexError: list index out of range'. See traceback in the log for more information." 01:54:26 CmdDone: APO.doravetz_2 149 jaeger fvc loop Tried again, and this time, no errors: 01:55:10 jaeger fvc loop Last RMS value before this command failed and gave us the "pickle" error: 01:58:33 RMS full fit 9.380 um." 01:58:40 APO.doravetz_2 152 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'TypeError: cannot pickle 'Command' object'. See traceback in the log for more information." 01:58:40 CmdDone: APO.doravetz_2 152 jaeger fvc loop Next fvc loop: 01:58:45 jaeger fvc loop RMS values: 01:59:13 RMS full fit 9.550 um." 01:59:56 RMS full fit 18.970 um." 02:00:02 APO.doravetz_2 153 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'TypeError: cannot pickle 'Command' object'. See traceback in the log for more information." 02:00:02 CmdDone: APO.doravetz_2 153 jaeger fvc loop Next fvc loop: 02:00:13 jaeger fvc loop Subsequent RMS values: 02:00:40 RMS full fit 19.480 um." 02:01:24 RMS full fit 14.660 um." 02:02:08 RMS full fit 9.260 um." 02:02:52 RMS full fit 9.430 um." Next fvc loop: 02:03:09 jaeger fvc loop RMS values: 02:03:37 RMS full fit 9.250 um." 02:04:20 RMS full fit 9.310 um." 02:04:26 APO.doravetz_2 155 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'TypeError: cannot pickle 'Command' object'. See traceback in the log for more information." 02:04:26 CmdDone: APO.doravetz_2 155 jaeger fvc loop Next fvc loop: 02:04:38 jaeger fvc loop 02:05:05 RMS full fit 9.350 um." 02:05:49 RMS full fit 9.430 um." 02:05:54 APO.doravetz_2 156 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'TypeError: cannot pickle 'Command' object'. See traceback in the log for more information." 02:05:54 CmdDone: APO.doravetz_2 156 jaeger fvc loop 02:06:12 jaeger configuration reverse Going on to another design id: 02:07:34 jaeger configuration load 36060 --scale 0.999882 02:08:59 Config Loaded: 36060-2275 02:11:09 jaeger configuration execute 02:12:18 jaeger fvc loop 02:15:46 RMS full fit 17.080 um." 02:15:52 APO.doravetz_2 160 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'TypeError: cannot pickle 'Command' object'. See traceback in the log for more information." Next fvc loop: 02:15:56 jaeger fvc loop Subsequent RMS values: 02:16:24 RMS full fit 16.540 um." 02:17:08 RMS full fit 15.970 um." Same pickle error 02:17:14 APO.doravetz_2 161 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'TypeError: cannot pickle 'Command' object'. See traceback in the log for more information." Next fvc loop: 02:17:21 jaeger fvc loop Subsequent RMS values: 02:17:49 RMS full fit 15.580 um." 02:18:33 RMS full fit 15.080 um." 02:18:39 APO.doravetz_2 162 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'TypeError: cannot pickle 'Command' object'. See traceback in the log for more information." Next fvc loop: 02:18:46 jaeger fvc loop 02:20:04 APO.doravetz_2 163 jaeger f error="Command 'fvc loop' failed because of an uncaught error 'TypeError: cannot pickle 'Command' object'. See traceback in the log for more information." Subsequent RMS values: 02:19:14 RMS full fit 14.500 um." 02:19:58 RMS full fit 14.170 um." 02:20:56 jaeger configuration reverse 02:21:10 APO.doravetz_2 164 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 9.91 seconds after start" Finished with Test #1. Moving on to Test #2. 02:21:45 jaeger unlock 02:22:01 jaeger unwind 03:20:19 APO.doravetz_2 237 jaeger : snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0064.pdf Robot 1340 is not folded, try an unwind. 03:21:42 jaeger unwind 20 do a second loop and after that accept whatever except if it's >30. 06:35:37 APO.Jose: I think at this point is not that much the corrections. I think it boils down to the RMS being dominated by one or two positioners that won't converge. I think we're going to use a better statistic than a plain RMS, but not sure which one yet. 06:33:36 RMS full fit 26.070 um." 06:34:19 RMS full fit 25.000 um." 06:35:03 RMS full fit 26.350 um." 06:35:03 FVC Loop - RMS:26.35 Delta RMS: -1.35 Image #: 0046 Jose is satisfied with these results. Resuming Test #2. 06:37:23 jaeger unwind 06:45:33 jaeger configuration load 35988 --scale 0.999882 06:46:17 Config Loaded: 35988-2304 06:46:23 jaeger configuration execute 06:47:12 jaeger configuration reverse 06:48:01 jaeger configuration load 35989 --scale 0.999882 06:49:11 Config Loaded: 35989-2305 06:49:17 jaeger configuration execute 06:50:06 jaeger configuration reverse 06:50:55 jaeger configuration load 35990 --scale 0.999882 06:51:58 Config Loaded: 35990-2306 06:52:04 jaeger configuration execute 06:52:43 .jaeger 0 jaeger f text="A collision was detected in positioner 608. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 06:52:43 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 06:52:45 .jaeger 0 jaeger e locked=T 06:52:45 .jaeger 0 jaeger i fps_status=0x14 06:52:47 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0128.pdf 06:52:47 APO.doravetz_2 412 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59598/trajectory-59598-0117.json 06:52:47 APO.doravetz_2 412 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 38.82 seconds after start" In the snapshot, we don't see the offending robot, but we know where it is from previous collision events. 06:56:45 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0129.pdf Offending robot is highlighted in pink. 06:56:37 jaeger configuration reverse 06:56:48 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0130.pdf Offending robot is unhighlighted. 06:57:19 jaeger unlock 06:57:31 jaeger unwind 06:58:57 jaeger configuration load 35991 --scale 0.999882 07:00:26 Config Loaded: 35991-2307 07:00:31 jaeger configuration execute 07:01:18 jaeger configuration reverse 07:03Z Running Dmitry's quick test to check on FPI by taking a test image from the APOGEE instrument gui window. Manually opened the shutters. 07:03:01 apogee.expose=apRaw-40360010, 5, A, ArcLamp and manually closed everything. We see light in the test image. 07:02:05 jaeger configuration load 35992 --scale 0.999882 07:04:05 Config Loaded: 35992-2308 07:04:10 jaeger configuration execute 07:04:56 jaeger configuration reverse 07:05:41 jaeger configuration load 35993 --scale 0.999882 07:07:31 Config Loaded: 35993-2309 07:07:37 jaeger configuration execute 07:08:24 jaeger configuration reverse 07:09:12 jaeger configuration load 35994 --scale 0.999882 07:11:13 Config Loaded: 35994-2310 07:11:19 jaeger configuration execute 07:12:08 jaeger configuration reverse 07:12:57 jaeger configuration load 35995 --scale 0.999882 Unable to resolve: 07:13:11 APO.doravetz_2 440 jaeger d text="Recomputing 88 coordinates." 07:13:14 APO.doravetz_2 440 jaeger i text="Generating path pair." 07:13:34 APO.doravetz_2 440 jaeger i text="Deadlocks found. Attempting resolution." 07:13:34 APO.doravetz_2 440 jaeger i text="Attempt 1: 7 deadlocks found." 07:13:34 APO.doravetz_2 440 jaeger d text="Trying to unlock positioner 40." 07:13:40 .jaeger 0 jaeger d alive_at=1642576420.4323275 07:13:53 APO.doravetz_2 440 jaeger i text="Attempt 2: 7 deadlocks found." 07:13:53 APO.doravetz_2 440 jaeger d text="Trying to unlock positioner 148." 07:14:12 APO.doravetz_2 440 jaeger i text="Attempt 3: 5 deadlocks found." 07:14:12 APO.doravetz_2 440 jaeger d text="Trying to unlock positioner 819." 07:14:31 APO.doravetz_2 440 jaeger i text="Attempt 4: 2 deadlocks found." 07:14:31 APO.doravetz_2 440 jaeger d text="Trying to unlock positioner 40." 07:14:40 .jaeger 0 jaeger d alive_at=1642576480.4361238 07:14:50 APO.doravetz_2 440 jaeger i text="Attempt 5: 2 deadlocks found." 07:14:50 APO.doravetz_2 440 jaeger d text="Trying to unlock positioner 40." 07:15:09 APO.doravetz_2 440 jaeger f error="Failed generating paths: Attempt 6: 2 deadlocks remain but the number of retries has been exhausted" Jose's comment: 07:16:53 APO.Jose: That's expected although it should be fairly unlikely. It happens when it cannot solve deadlocks after a number of tries. Usually in that case if you load the design again it will probably work. The reasoning: 07:17:21 APO.Jose: It may happen more in APOGEE designs because they have fewer targets assigned and the decollision can create weird effects. So we will try it again: 07:17:17 jaeger configuration load 35995 --scale 0.999882 07:18:05 Config Loaded: 35995-2311 07:18:11 jaeger configuration execute 07:18:59 jaeger configuration reverse 07:19:48 jaeger configuration load 35996 --scale 0.999882 07:20:32 Config Loaded: 35996-2312 07:20:38 jaeger configuration execute 07:21:25 jaeger configuration reverse 07:22:12 jaeger configuration load 35997 --scale 0.999882 07:23:01 Config Loaded: 35997-2313 07:23:07 jaeger configuration execute 07:23:54 jaeger configuration reverse 07:26:27 jaeger configuration load 35998 --scale 0.999882 07:27:11 Config Loaded: 35998-2314 07:27:16 jaeger configuration execute 07:27:47 .jaeger 0 jaeger f text="A collision was detected in positioner 608. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 07:27:47 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 07:27:47 .jaeger 0 jaeger i fps_status=0x14 07:27:48 .jaeger 0 jaeger e locked=T 07:27:52 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0146.pdf 07:27:52 .jaeger 0 jaeger w text="Snapshot for locked FPS: /data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0146.pdf (JaegerUserWarning)" 07:27:52 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0146.pdf 07:27:53 APO.doravetz_2 455 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59598/trajectory-59598-0134.json 07:27:53 APO.doravetz_2 455 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 31.65 seconds after start" 07:27:53 CmdDone: APO.doravetz_2 455 jaeger configuration execute Again, the same offending robot: 07:27:47 Jaeger collision: 608 07:35:22 jaeger unlock 07:35:29 APO.doravetz_2 457 jaeger d text="Calculating unwind trajectory." 07:35:29 APO.doravetz_2 457 jaeger f error="Command 'unwind' failed because of an uncaught error 'RuntimeError: Grid is Kaiju-collided, cannot generate paths'. See traceback in the log for more information." Trying an explode: 07:35:39 jaeger explode 5 7:35:50 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0147.pdf 07:35:50 .jaeger 0 jaeger w text="Snapshot for locked FPS: /data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0147.pdf (JaegerUserWarning)" 07:35:52 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0148.pdf 07:35:53 APO.doravetz_2 458 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59598/trajectory-59598-0135.json 07:35:53 APO.doravetz_2 458 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 8.41 seconds after start" 07:36:01 jaeger unlock 07:36:05 jaeger unwind 07:36:05 APO.doravetz_2 460 jaeger d text="Calculating unwind trajectory." 07:36:22 APO.doravetz_2 460 jaeger i text="Executing unwind trajectory." 07:36:22 APO.doravetz_2 460 jaeger d text="Sending trajectory data." 07:36:25 .jaeger 0 jaeger w text="Positioner 765 replied to SEND_TRAJECTORY_DATA UID=4734 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 07:36:25 APO.doravetz_2 460 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" Checking status 07:36:37 APO.doravetz_2 461 jaeger i positioner_status=765,-0.2128,122.6755,0x8db06791,T,F,F,F,04.01.21,3,2,55 Here is a case of a negative alpha value. 07:38:51 jaeger status 765 07:38:51 .hub 0 cmds d CmdQueued=79985,1642577931.30,"APO.Jose",57,"jaeger",67,"status 765" 07:38:52 InvalidKeywordData=jaeger.positioner_status, ['765', '-0.2128', '122.6755', '0x8db06791', 'T', 'F', 'F', 'F', '04.01.21', '3', '2', '55'] 07:42Z Restarted jaeger actor. 07:41:37 jaeger unwind 07:41:54 APO.doravetz_2 468 jaeger d text="Sending trajectory data." 07:41:57 .jaeger 0 jaeger w text="Positioner 765 replied to SEND_TRAJECTORY_DATA UID=3019 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 07:41:57 APO.doravetz_2 468 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59598/trajectory-59598-0136.json 07:41:57 APO.doravetz_2 468 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" See Problems in Detail for the discussion and json file that shows what is happening. 07:59:45 jaeger explode --one 765 5 08:00:11 jaeger unwind 08:00:31 .jaeger 0 jaeger w text="Positioner 765 replied to SEND_TRAJECTORY_DATA UID=3048 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 08:00:31 APO.doravetz_2 485 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59598/trajectory-59598-0138.json 08:00:31 APO.doravetz_2 485 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" 08:00:43 jaeger status 765 08:00:59 .hub 0 cmds d CmdQueued=80198,1642579259.35,"APO.Jose",78,"jaeger",11,"status 765" 08:01:00 InvalidKeywordData=jaeger.positioner_status, ['765', '-0.2128', '117.6835', '0x8db06791', 'T', 'F', 'F', 'F', '04.01.21', '3', '2', '56'] 08:03:16 APO.Jose: In the previous explode \"765\": { \"alpha\": [ [ -0.21261636167764664, 1.0 ], [ -0.21261636167764664, 4.75 ] ], 08:03:12 CmdStarted: APO.doravetz_2 491 jaeger goto 765 0.0 117.6835 08:03:45 CmdStarted: APO.doravetz_2 492 jaeger status 765 08:03:45 APO.doravetz_2 492 jaeger > 08:03:45 APO.doravetz_2 492 jaeger i positioner_status=765,-0.007,117.6858,0x8db067e1,T,F,F,F,04.01.21,3,2,57 08:03:45 APO.doravetz_2 492 jaeger : 08:03:51 APO.Jose: And with that goto \"alpha\": [ [ -0.21276723593473434, 1.0 ], [ -0.20567432708210415, 4.341666666666667 ], [ 0.0, 4.35 ] ], 08:04:45 APO.doravetz_2: So the trajectory used to move it in the 'goto' was not a weird \"stay in the same place at two times\" thing 08:04:56 APO.Jose: The goto still uses kaijum but it seems the two point trajectory was easy enough for it to not do anything weird. 08:05:37 jaeger unwind 08:32:12 APO.doravetz_2 528 jaeger : snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0157.pdf Alpha number is not negative, but still failed. Investigating. 08:36:14 APO.Jose: Alpha looks fine here but beta looks problematic \"beta\": [ [ 170.0, 1.0 ], [ 169.31031094704065, 1.00833333333334 ], [ 169.2436442803716, 20.708333333333336 ], [ -0.28968905295934705, 34.85000000000001 ], [ -1.651800369158532, 34.89166666666667 ] ] 08:37:00 APO.Jose: It asks to move 170 degrees in 14 seconds. That sounds like a lot, and then an extra 1.5 degrees on 0.04s 08:37:14 APO.Jose: I think it has to be acceleration. 08:39:17 APO.Jose: The 169->-0.2 is probably ok. The code should be asking the motors to move at a max of 2 RPM, so 12 degrees per second. That's about what it does there. 08:39:40 APO.doravetz_2: oh, those are good numbers for reference, thanks 08:40:25 APO.Jose: But the last step is about 32 degrees per second. 08:41:02 APO.Jose: That's may be possible for the robot in terms of speed, but probably not in acceleration if it has to do it starting at 2RPM. 08:41:51 APO.doravetz_2: it's just spinning its proverbial tires at that point 08:42:12 APO.Jose: I think this all points to an issue with kaiju when there are negative numbers involved. It may be an issue with the smoothing algorithm. 09:11:10 jaeger configuration load 36002 --scale 0.999882 09:11:07 Config Loaded: 36002-2318 09:11:13 jaeger configuration execute 09:12:00 jaeger configuration reverse 09:12:47 jaeger configuration load 36003 --scale 0.999882 09:13:38 Config Loaded: 36003-2319 09:13:44 jaeger configuration execute 09:14:23 .jaeger 0 jaeger f text="A collision was detected in positioner 608. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 09:14:23 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 09:14:23 .jaeger 0 jaeger i fps_status=0x14 09:14:25 .jaeger 0 jaeger e locked=T 09:14:29 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0160.pdf 09:14:29 APO.doravetz_2 545 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59598/trajectory-59598-0149.json 09:14:29 APO.doravetz_2 545 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 40.14 seconds after start" 09:14:29 CmdDone: APO.doravetz_2 545 jaeger configuration execute 09:14:29 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0160.pdf 09:14:29 .jaeger 0 jaeger w text="Snapshot for locked FPS: /data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0160.pdf (JaegerUserWarning)" 09:14:23 Jaeger collision: 608 09:14:56 Jaeger collision: 608 09:15:17 Jaeger collision: 1285 09:14:44 CmdStarted: APO.doravetz_2 546 jaeger unlock 09:14:48 CmdStarted: APO.doravetz_2 547 jaeger explode 10 09:15:17 .jaeger 0 jaeger f text="A collision was detected in positioner 1285. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 09:15:17 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 09:15:19 .jaeger 0 jaeger e locked=T 09:15:22 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0163.pdf 09:15:22 .jaeger 0 jaeger w text="Snapshot for locked FPS: /data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0163.pdf (JaegerUserWarning)" 09:15:24 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0164.pdf 09:15:24 APO.doravetz_2 549 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59598/trajectory-59598-0151.json 09:15:24 APO.doravetz_2 549 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 8.27 seconds after start" 09:15:24 CmdDone: APO.doravetz_2 549 jaeger explode 10 09:15:27 CmdStarted: APO.doravetz_2 550 jaeger unlock 09:15:38 CmdStarted: APO.doravetz_2 551 jaeger unwind 09:16:04 APO.doravetz_2 551 jaeger d text="Sending trajectory data." 09:16:06 .jaeger 0 jaeger w text="Positioner 1170 replied to SEND_TRAJECTORY_DATA UID=3717 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 09:16:06 APO.doravetz_2 551 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59598/trajectory-59598-0152.json 09:16:06 APO.doravetz_2 551 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" 09:16:18 CmdStarted: APO.doravetz_2 552 jaeger status 1170 09:16:18 APO.doravetz_2 552 jaeger i positioner_status=1170,-0.2148,59.123,0x8db06791,T,F,F,F,04.01.21,6,1,68 09:16:47 jaeger goto 1170 0.0 59.123 09:17:06 CmdStarted: APO.doravetz_2 554 jaeger unwind 09:17:32 APO.doravetz_2 554 jaeger d text="Sending trajectory data." 09:17:34 .jaeger 0 jaeger w text="Positioner 1376 replied to SEND_TRAJECTORY_DATA UID=3749 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 09:17:34 .jaeger 0 jaeger w text="Positioner 1345 replied to SEND_TRAJECTORY_DATA UID=3749 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 09:17:34 APO.doravetz_2 554 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59598/trajectory-59598-0154.json 09:17:34 APO.doravetz_2 554 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" 09:17:50 APO.doravetz_2 555 jaeger : snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0166.pdf 09:17:59 jaeger unwind --force 09:18:28 .jaeger 0 jaeger w text="Positioner 1376 replied to SEND_TRAJECTORY_DATA UID=3760 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 09:18:28 .jaeger 0 jaeger w text="Positioner 1345 replied to SEND_TRAJECTORY_DATA UID=3760 with 'VALUE_OUT_OF_RANGE'. (JaegerUserWarning)" 09:18:28 APO.doravetz_2 556 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59598/trajectory-59598-0155.json 09:18:28 APO.doravetz_2 556 jaeger f error="Trajectory failed with error: Something went wrong sending the trajectory: At least one SEND_TRAJECTORY_COMMAND failed" 09:18:38 CmdStarted: APO.doravetz_2 557 jaeger status 1376 09:18:38 APO.doravetz_2 557 jaeger > 09:18:39 APO.doravetz_2 557 jaeger i positioner_status=1376,-0.1959,120.9799,0x8db06791,T,F,F,F,04.01.21,2,3,69 09:19:06 jaeger goto 1376 0.0 120.9799 09:19:19 CmdStarted: APO.doravetz_2 559 jaeger status 1345 09:19:19 APO.doravetz_2 559 jaeger > 09:19:20 APO.doravetz_2 559 jaeger i positioner_status=1345,-0.2137,112.638,0x8db067e1,T,F,F,F,04.01.21,4,2,70 09:19:33 jaeger goto 1345 0.0 112.638 09:19:54 jaeger unwind 09:21:28 jaeger configuration load 36004 --scale 0.999882 09:22:51 Config Loaded: 36004-2320 09:22:57 jaeger configuration execute 09:23:44 jaeger configuration reverse 09:24:30 jaeger configuration load 36005 --scale 0.999882 09:25:19 Config Loaded: 36005-2321 09:25:25 jaeger configuration execute 09:26:14 jaeger configuration reverse 09:27:03 jaeger configuration load 36006 --scale 0.999882 09:28:05 Config Loaded: 36006-2322 09:28:11 jaeger configuration execute 09:28:58 jaeger configuration reverse 09:29:44 jaeger configuration load 36007 --scale 0.999882 09:30:35 Config Loaded: 36007-2323 09:30:41 jaeger configuration execute 09:31:18 Jaeger collision: 608 09:32:37 Jaeger collision: 608 09:31:18 .jaeger 0 jaeger f text="A collision was detected in positioner 608. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 09:31:18 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 09:31:19 .jaeger 0 jaeger d alive_at=1642584678.8384535 09:31:20 .jaeger 0 jaeger e locked=T 09:31:20 .jaeger 0 jaeger i fps_status=0x14 09:31:24 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0176.pdf 09:31:24 APO.doravetz_2 572 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59598/trajectory-59598-0165.json 09:31:24 APO.doravetz_2 572 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 38.51 seconds after start" 09:31:24 CmdDone: APO.doravetz_2 572 jaeger configuration execute 09:31:24 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0176.pdf 09:31:24 .jaeger 0 jaeger w text="Snapshot for locked FPS: /data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0176.pdf (JaegerUserWarning)" 09:32:18 jaeger unlock 09:32:30 jaeger explode 10 09:32:37 .jaeger 0 jaeger f text="A collision was detected in positioner 608. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 09:32:37 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 09:32:39 .jaeger 0 jaeger e locked=T 09:32:42 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0177.pdf 09:32:42 .jaeger 0 jaeger w text="Snapshot for locked FPS: /data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0177.pdf (JaegerUserWarning)" 09:32:45 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/59598/fps_snapshot_59598_0178.pdf 09:32:45 OBSERVER.karen 15 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/59598/trajectory-59598-0166.json 09:32:45 OBSERVER.karen 15 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 8.47 seconds after start" 09:34:09 jaeger unlock 09:34:30 jaeger unwind 09:37:10 jaeger configuration load 36008 --scale 0.999882 09:38:12 Config Loaded: 36008-2324 09:38:18 jaeger configuration execute 09:39:03 jaeger configuration reverse 09:44:35 jaeger unwind 09:44:41 jaeger power off 11:02Z Giving up. 11:16Z Started apogee morning cals ---- BOSS Spectrographs Collimation Summary ---- Afternoon focus check: 23:17:22 boss exposure 4.0, file=336380 23:18:03 boss exposure 4.0, file=336381 23:18:45 sp1.motor.move= -462, -461, -462 23:18:45 Hartmann collimate output sp1: offset: r= -989 (In Focus); b= -0.2 (In Focus) sp1: pred. move: spAverageMove= -462 sp1: pred. resid: r= -526, b= -1.7, txt= OK, spTemp = 4.1 ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0040 0.0040 Axial B: 0.0110 0.0030 Axial C: -0.1150 -0.1150 Trans D: -0.0342 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 1.89 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6560 -9.6570 Axial B: -9.6680 -9.6710 Axial C: -9.6950 -9.6970 Trans D: 0.1040 0.1140 GALILS Commanded: 4891850 4890350 4910500 -50 -3400 Actual: 4859192 4859199 4859136 -50 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.96 0.02 -0.01 -0.62 42.24 ---- Telescope Status ---- Status at: 10:49Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 685 687 705 704 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 193 lbs, 22.7 psi ---- Software Used ---- --- Afternoon disk space --- (observers39) [observer@sdss5-hub: ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sdb1 2.0T 156G 1.9T 8% /home 10.25.1.230:/data 21T 19T 1.2T 95% /data Get Versions, 01/18/22, 23:23:53 --- STUI software --- STUI : 1.7.16 alerts : 2.2.0 apo : FAILED boss : 6.0.1 -boss.specMec : 3.0.4 guider : FAILED ecamera : FAILED fliswarm : 0.2.1-alpha.0 jaeger : 0.13.2-alpha.0 kaiju : 1.2.2-alpha.0 coordio : 1.2.1-alpha.2 fps_calibrati : APO.2021.12.25 cherno : 0.1.1-alpha.0 hal : 0.1.1-alpha.0 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not available apogeecal : not available apogeeql : v1_7_0dev STUIAdditions : 3.1.11 --- Other Versions --- sdss-obstools : 3.7.7b0 TPM : 1.2-31-gd7eb MCP : 2.1 ---- APOGEE Data Summary ---- 59597 morning cals: APOGEE data list, mjd=59597, all raw_data: /data/apogee/utr_cdr/59597/apRaw-*.fits archive: /data/apogee/archive/59597/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59597/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset %Flux Comm ------------------------------------------------------------------------------ 59597 37 13:18 40350037 Dark 60 A 13.0 FPS- 0 a-b-c - - 59597 38 13:28 40350038 Dark 60 A 13.0 FPS- 0 a-b-c - - 59597 39 13:39 40350039 Dark 60 A 13.0 FPS- 0 a-b-c - - 59597 40 13:41 40350040 QuarFlat 10 A 13.0 FPS- 0 a-b-c - 99 59597 41 13:43 40350041 QuarFlat 10 A 13.0 FPS- 0 a-b-c - 99 59597 42 13:44 40350042 QuarFlat 10 A 13.0 FPS- 0 a-b-c - 99 59597 43 13:47 40350043 Arc-Thar 12 B 13.5 FPS- 0 a-b-c 0.51 - 59597 44 13:54 40350044 Arc-Une 40 B 13.5 FPS- 0 a-b-c - - 59597 45 13:56 40350045 Arc-Thar 12 A 13.0 FPS- 0 a-b-c -0.07 - 59597 46 14:03 40350046 Arc-Une 40 A 13.0 FPS- 0 a-b-c - - 59597 47 14:09 40350047 Dark 30 A 13.0 FPS- 0 a-b-c - - 59597 48 14:14 40350048 InteFlat 30 A 13.0 FPS- 0 a-b-c - 279 59597 49 14:20 40350049 InteFlat 30 A 13.0 FPS- 0 a-b-c - 279 59597 50 14:25 40350050 InteFlat 30 A 13.0 FPS- 0 a-b-c - 279 59597 51 14:30 40350051 Dark 30 A 13.0 FPS- 0 a-b-c - - 59597 52 14:36 40350052 Arc---- 30 A 13.0 FPS- 0 a-b-c - - FPI ------------------------------------------------------------------------------ We noticed 40350052 (FPI) didn't have light/lines. We'll double-check the morning cals script. ---- Problems in Detail ---- From Jose, in an email to us: This is the trajectory that's trying to send to 765. Each point in the alpha and beta lists is [angle, time]. The first few points in alpha are oscillating around zero and then there are two points with the exact same angle and different time. I can see how that could create a divide by zero somewhere in the firmware code. "765": { "alpha": [ [ -0.21261636167764664, 1.0 ], [ 1.9832816170321579, 1.2166666666666668 ], [ 0.03883717258771223, 1.4083333333333334 ], [ -0.27227393852339865, 1.475 ], [ -0.27227393852339865, 2.658333333333333 ], [ 0.18328161703215684, 3.016666666666667 ], [ 0.8277260614766012, 3.1 ], [ 9.327726061476584, 3.8083333333333336 ], [ 9.727726061476597, 3.875 ], [ 9.727726061476599, 25.875 ], [ 10.0, 25.883333333333336 ] ], "beta": [ [ 122.67559289932251, 1.0 ], [ 127.59781512154447, 1.425 ], [ 128.70892623265541, 1.8916666666666666 ], [ 131.62003734376637, 2.2333333333333334 ], [ 134.286704010433, 2.716666666666667 ], [ 169.57559289931987, 5.658333333333334 ], [ 169.98346508873774, 5.716666666666667 ], [ 170.0, 25.883333333333336 ] ] In the chat, the following is our discussion of this: 07:48:08 APO.Jose: Yes, I can see the issue quite clearly. I sent you an email with the trajectory that is being sent to 765. It's not a normal trajectory; it's kind of oscillating and then there is a point in which the positioner does not move but time passes. I think that could create a problem in the firmware. 07:49:09 APO.doravetz_2: ah, yeah, we see what you mean 07:49:55 APO.Jose: Another interesting thing is that it seems that none of the values is really \"out of range\". All the values are valid I think, but it's probably the interpolation that it fails. From what I understood of the firmware I would have expected that to be an INVALID_TRAJECTORY error. But I guess no. 07:51:33 APO.Jose: This may have happened because at some point we allowed negative values in the axes. I'll have to check with Conor; maybe in the past we had effectively, if not on purpose, avoided that. 07:51:35 APO.doravetz_2: so they have to have smooth continuous motion to work? 07:51:52 APO.doravetz_2: i.e., being told to \"stay put\" for some delta_t isn't allowed? 07:52:33 APO.Jose: Not really smooth. They can be jagged as long as the acceleration to go from one point to another is reasonable. If the problem is really that two points have the same value for different times that's something the firmware should allow, so it sould like a bug. 07:52:51 APO.Jose: *sounds 07:52:52 APO.doravetz_2: ok 07:53:37 APO.doravetz_2: it's interesting at the end of the alpha values, the 9.7277... 07:53:59 APO.doravetz_2: that one is that same to many decimal places spaced 22 sec apart then it moves to 10 at the very end 07:55:12 APO.Jose: Right ... We could figure out a few tests to send trajectories with identical values at different times and see if that fails consistently. 07:55:34 APO.Jose: We had a version of something similar in the past with the first point being kind of unbound, but that was fixed. 07:55:39 APO.doravetz_2: if positioners need to wait and let others pass at times to avoid collisions, the jaggedness would seem to be a necessary thing 07:55:42 APO.doravetz_2: ok 07:56:51 APO.Jose: Yes, the positioners staying essentially stopped for a while is quite normal in kaiju's trajectories. But maybe not with exactly the same value. Or maybe it's the fact that's the same value and negative. 07:57:08 APO.doravetz_2: ok 07:57:31 APO.doravetz_2: the small oscillatory behavior doesn't seem useful though 07:57:33 APO.Jose: In this case I think the only option is either an explode or a cowboy move. Basically something that will generate a different kind of trajectory. 07:58:01 APO.doravetz_2: how does the explode affect the alpha part if we only saw beta changes with it? 07:58:12 APO.Jose: Yeah. Conor has some code that after a trajectory has been generated smoothes it, but it can sometimes generate some weird points. 07:58:47 APO.Jose: The explode should move both axes. Although probably has a preference for beta because it's easier to separate the positioners with beta. 08:06:57 APO.doravetz_2: so my next question would be what if a 'goto' doesn't always work? 08:07:39 APO.doravetz_2: how robust is the trajectory calculated for a small 'goto' move against this kind of behavior? 08:07:59 APO.Jose: Then I would do a goto --go-cowboy. That overrides kaiju completely and just sends a two-point trajectory. That should always work but it's not safe so you want to check carefully. But if you're only moving 0.2 degrees that's usually fine. 08:09:14 APO.Jose: Actually I made a mistake and the trajectory I sent before with the two identical values corresponds to the explode that suceeded. So maybe identical values is not the issue. It may have something to do with the oscillation around 0. That's what I'm seeing in the second unwind that also failed. 08:10:02 APO.doravetz_2: do you recall what the max allowable acceleration threshold is? 08:10:54 APO.Jose: No, we never got a clear value for it. But if you look at the trajectory I sent over email it asks to do 2 degrees in 0.2 seconds. That may be too much. 08:11:08 APO.Jose: Something similar happens in the second unwind that failed. 08:11:10 APO.doravetz_2: exactly what we were looking at 08:12:20 APO.doravetz_2: And after the 2 deg in 0.2 it wants to do -2 deg in 0.2 08:12:39 APO.Jose: With this data we should be able to figure out what happened, but it may require talking with Ricardo from EPFL. But I it seems there are a few things to try. 08:12:51 APO.Jose: Right, that kind of super fast zigzag may be too much for it. 08:13:58 APO.doravetz_2: So somewhere in the firmware it's aware that it can't do what it's being asked, but the value for, e.g., acceleration isn't actually defined in there? 08:14:11 APO.doravetz_2: or it is explicitly in there and you need to talk to ricardo to find out what those limits are? 08:14:52 APO.doravetz_2: i'm thinking about the over-current type of events - the fact that the firmware knows it can't do what's being asked indicates it knows about the limits 08:15:18 APO.Jose: It's in the firmware but not in the form of degrees/s. It some more convoluted form of steps per time unit and some other additional parameters so we've never known the exact velocity/acceleration value in units we understand. 08:15:19 APO.doravetz_2: but in an over-current event, for example, it SHOULD be able to do what's asked, but it actually physically goes over for some reason 08:15:38 APO.doravetz_2: oh ok 08:16:07 APO.Jose: Yes, it knows about its limits, it's just hard to parse :)