From: Karen Kinemuchi Subject: Re: [sloan-25mlog 5850] Re: 2.5m obslog 20220119 (SJD 59599) Date: Thu, 20 Jan 2022 14:16:18 -0700 --Apple-Mail=_0A00AC30-6231-4044-9D1B-3137D47B1C35 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Jose, For your points, see my comments below: > On Jan 20, 2022, at 11:55 AM, Jos=C3=A9 S=C3=A1nchez-Gallego = wrote: >=20 > Hi Kaike, Karen, >=20 > Thanks for the details statistics, this is quite clarifying about what = issues you're normally finding. A few quick notes: >=20 > For the "jaeger unwind" failed with X deadlock found, I think = exploding should handle most of those cases, but maybe it does not. An = alternative thing you can try is to always first try jaeger = configuration reverse. If that fails with a kaiju collided error, = explode a small amount (say jaeger explode 2) then try jaeger = configuration reverse. If that still fails then try the unwind. We did try a combination of explode/unwind and that seemed to work = consistently well. We did try jaeger configuration reverse a few times = at the beginning of the night, but we would get the error message that = the previous position didn=E2=80=99t match to what was expected and the = command would fail. I think it worked once, and it was after a = =E2=80=9Cjaeger configuration execute=E2=80=9D. It seemed after an FVC = loop and explode 5, this command was more likely to fail (probably = because the robots were too far from the initial configuration load?). We were exploding with a bigger value because sometimes we=E2=80=99d see = a lot of red collided robots, but exploding by a small amount and = issuing the reverse command makes sense. Something to definitely try to = help cut down the time we=E2=80=99ve spent on fixing deadlocked robots! > I think adding what robots are deadlocked to the error message is = possible. > Boson should never use 100% of CPU, especially if you're not guiding. = I released one or two versions that had an issue, so if you happened to = update to one of them maybe that explains it. Please try updating to the = latest release (right now 0.0.4-alpha.16 = ) by = downloading the DMG or checking the version in Boson. If you see more = 100% CPU usage or it running obviously slow, please let me know = indicating what windows were open at the time and what you were doing = (just reconfiguring, guiding, etc). I=E2=80=99m hoping whoever comes on after me will be able to install the = latest version of Boson. I kept getting a pop up window throughout the = night asking for updating and installing the =E2=80=9CHelper=E2=80=9D, = which all required admin privilege. The obsadmin credentials would get = accepted, but then nothing happened. In the end, Kaike advised to just = cancel whenever it popped up. I had a lot of trouble with this update, = so it is still not complete. We used the activity monitor to see what was causing the slow down on = Boson, and that=E2=80=99s where we saw the 100% CPU. I had running only = the snapshot viewer (I had closed down the gfa viewer), and I did notice = that the snapshots upload would slow down. It got so bad that I could = not activate the Boson menus or windows, and the Mac reported that Boson = was =E2=80=9Cnon-responsive=E2=80=9D. I had to force quit out of it 2-3 = times. Other times, I was able to cmd-R and get things going again, and = at least once was able to cleanly close Boson down and reopen without = any issue during one of the slow downs. I think I was running = 0.0.4-alpha.13? I can=E2=80=99t remember, but we did look at the = version number at some point. I was zooming in and out of the snapshot image displayed to figure out = which robot id was in trouble, but I saw Dan=E2=80=99s message of the = new feature in Boson to help identify robots easily in the latest = version. That will help immensely! So in the end, the Boson problems came from me not successfully being = able to update it on sdss5-obs1. I don=E2=80=99t know if we need = Shane/iTracey=E2=80=99s help on this, unless someone else can get the = machine to accept the update? Hope this explanation helps! Karen= --Apple-Mail=_0A00AC30-6231-4044-9D1B-3137D47B1C35 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi = Jose, For your points, see my comments = below:On Jan 20, 2022, at 11:55 AM, Jos=C3=A9 = S=C3=A1nchez-Gallego wrote:Hi Kaike, Karen,Thanks for the details = statistics, this is quite clarifying about what issues you're normally = finding. A few quick notes:For the "jaeger unwind" failed with X deadlock found, I = think exploding should handle most of those cases, but maybe it does = not. An alternative thing you can try is to always first try jaeger = configuration reverse. If that fails with a kaiju collided error, = explode a small amount (say jaeger explode 2) then try jaeger = configuration reverse. If that still fails then try the = unwind.We did try a combination of explode/unwind and = that seemed to work consistently well. We did try jaeger = configuration reverse a few times at the beginning of the night, but we = would get the error message that the previous position didn=E2=80=99t = match to what was expected and the command would fail. I think it = worked once, and it was after a =E2=80=9Cjaeger configuration = execute=E2=80=9D. It seemed after an FVC loop and explode 5, this = command was more likely to fail (probably because the robots were too = far from the initial configuration load?).We were exploding with a bigger value because = sometimes we=E2=80=99d see a lot of red collided robots, but exploding = by a small amount and issuing the reverse command makes sense. = Something to definitely try to help cut down the time we=E2=80=99ve = spent on fixing deadlocked robots!I think adding what robots are deadlocked to the error = message is possible.Boson should never use 100% of = CPU, especially if you're not guiding. I released one or two versions = that had an issue, so if you happened to update to one of them maybe = that explains it. Please try updating to the latest release (right now = 0.0.4-alpha.16) by downloading the DMG or checking the = version in Boson. If you see more 100% CPU usage or it running obviously = slow, please let me know indicating what windows were open at the time = and what you were doing (just reconfiguring, guiding, = etc).I=E2=80=99m hoping whoever comes on after me will = be able to install the latest version of Boson. I kept getting a = pop up window throughout the night asking for updating and installing = the =E2=80=9CHelper=E2=80=9D, which all required admin privilege. = The obsadmin credentials would get accepted, but then nothing = happened. In the end, Kaike advised to just cancel whenever it = popped up. I had a lot of trouble with this update, so it is still = not complete.We used the activity = monitor to see what was causing the slow down on Boson, and that=E2=80=99s= where we saw the 100% CPU. I had running only the snapshot viewer = (I had closed down the gfa viewer), and I did notice that the snapshots = upload would slow down. It got so bad that I could not activate = the Boson menus or windows, and the Mac reported that Boson was = =E2=80=9Cnon-responsive=E2=80=9D. I had to force quit out of it 2-3 = times. Other times, I was able to cmd-R and get things going = again, and at least once was able to cleanly close Boson down and reopen = without any issue during one of the slow downs. I think I was = running 0.0.4-alpha.13? I can=E2=80=99t remember, but we did look = at the version number at some point.I was zooming in and out of the = snapshot image displayed to figure out which robot id was in trouble, = but I saw Dan=E2=80=99s message of the new feature in Boson to help = identify robots easily in the latest version. That will help = immensely!So = in the end, the Boson problems came from me not successfully being able = to update it on sdss5-obs1. I don=E2=80=99t know if we need = Shane/iTracey=E2=80=99s help on this, unless someone else can get the = machine to accept the update? Hope this explanation = helps! Karen= --Apple-Mail=_0A00AC30-6231-4044-9D1B-3137D47B1C35--