From: Kaike Subject: Re: [sloan-25mlog 5690] 2.5m obslog 20211015 (SJD 59503) Date: Sat, 16 Oct 2021 16:44:25 -0600 --Apple-Mail=_33FF1334-6471-46E1-9DD8-C230F3B7613B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hello Jose, Dan, Dylan, and Audrey, Thanks for the tests.=20 =20 Tracy and Ryan: A couple of nights ago, we have issue with = connections of M1 and M2 controllers, which was tracked down to a UPS = unit. Does the Xyplex terminal server also use that UPS unit?=20 Thanks, -Kaike > On Oct 16, 2021, at 10:05 AM, Jos=C3=A9 S=C3=A1nchez-Gallego = wrote: >=20 > Hi Dylan, Audrey, Dan, >=20 > Thanks for this detailed testing and for Dan's detailed description of = the problem yesterday. >=20 > It seems the problem is mostly concentrated on the camera and mech = controllers, while the DAQ has only happened once. If we consider the = failure of the DAQ a random netburner problem (and it may not be) that = would make some sense since the camera and mech are on the same network = (the usual 10.25) and connected to the Xyplex terminal server, while the = DAQ has its own dedicated fibre link and a dedicated physical network = interface. >=20 > I've been looking at the logs and while I see the moveColl commands = and the disconnects that happen after it, there doesn't appear to be = anything that looks a software error. I do see some tracebacks that are = expected when one of the controllers disconnects, but nothing that would = indicate a cascade somewhere else in the code that causes the = disconnection. >=20 > After the trouble with the galils the other night I wonder if the = problem could be with the Xyplex terminal server or thereabouts, since = all the devices we have had problems with have been connected to it. If = the connection is somehow unstable I could see how multiple consecutive = moveColl could be more likely to catch one of those micro-disconnections = and cascade that into the software needing to reconnect. That could = explain why the 4x15 min exposures didn't fail, since during those the = communication with the devices is more sporadic (only periodic status = requests) and those may not coincide with a lost connection. >=20 > This could also be a network problem in the 10.25 network, but I'm not = sure what's the best way of diagnosing that. Maybe Shane and Tracey can = look into adding or querying some diagnostic tools on the network. >=20 > A third possibility, just to make sure it's not a software issue, = would be to run the Python 2 version of bossICC. I would expect if this = is a software issue related to the migration to Python 3 that the issue = would be much more obvious, or at least consistent, but it may be worth = a try. >=20 > Thanks, >=20 > Jos=C3=A9 >=20 > On Oct 16, 2021 at 03:35:13, dgatlin@apo.nmsu.edu = wrote: >=20 > APACHE POINT OBSERVATORY SDSS 2.5M >=20 > OBSERVING LOG > Friday October 15, 2021 >=20 > (SJD 59503) >=20 > ---- Observing Team ---- >=20 > Dylan Gatlin > Audrey Oravetz (R) >=20 >=20 > ---- Observing Plan ---- >=20 > Test BOSS exposures, improve alt balance >=20 >=20 > ---- Observing Summary ---- >=20 > We didn't open tonight, and spent some time testing the telescope = balance and > the boss disconnect issues. >=20 > Notes: > - The counterweights for cart 12 are now at 500, which is a much = better balance > than before. It is a little top heavy at low altitudes and bottom = heavy at high > altitudes. >=20 > - We took a series of test exposures trying to identify the source of = the boss > issues. We're not certain if it's a software or connection issue, but = we > identified some problematic commands and you should be able to = reproduce the > issue very quickly by looping a `boss moveColl` command. See the main = log for a > complete description. >=20 >=20 > ---- Observing Log ---- >=20 > Mirror numbers in spec > APOGEE cals started, including 30 read FPI > BOSS bias and dark taken, okay, cannot take test flat/arc with current = cart >=20 > The weather is nice outside, but the observing plan doesn't call for = opening, so > we will not ventilate. >=20 > The day staff noted particularly bad telescope balance, so we took = some time > to test it and find a better balance. It was set when we loaded cart = 12 to 380 > and has stayed there since. At all altitudes, we have noticed that the = telescope > tends to fall up. >=20 > In increments of 20, we increased the counterweights, then recorded = the motor > current and voltage at various altitudes. The measured current and = voltage > depends on the direction of the previous move, so a set going down and = a set > going up were collected. This procedure is quite cumbersome and took = about 2 > hours. We stopped with the counterweights at 500, which is quite good. = The > telescope will still tend to fall up at above 45 degrees according to = our tests. >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > After stowing the telescope after the counterweight tests, we tried to = diagnose > the boss sp1mech issues. We took a series of 60 biases by simply = looping the > command `boss exposure bias` with nothing else. This caused sp1mech to = die once > at 05:17Z. It appears that the sp1mech died as soon as prereading = began. >=20 > After we restarted the sp1mech using `boss reloadControllers = controllers=3Dsp1mech` > we got stuck reading the previous exposure. We kept getting the = following > message every time we ran a `boss status` > 05:37:24 OBSERVER.dgatlin 126 boss w text=3D"exposureState timer does = not add up (remain=3D-1099.43 total=3D69.70)" >=20 > The remain timer kept increasing, until we ran a `boss exposure = recover`, which > read out the image, but killed the loop command. >=20 > We then took a set of 4 15 minute exposures. If we get an error per = hour from > sp1mech, then we should see one of these exposures fail. Our last test = showed > that you don't need a `boss moveColl` to cause a failure, this test is = a way of > seeing if the constant commands are an issue. >=20 > No issues on those 4 exposures, trying again just to be sure. This ran = without > error >=20 > We tried to run the moveColl command in isolation to see if that was = the main > source of the sp1mech failures. We looped the following command >=20 > boss moveColl spec=3Dsp1 a=3D650 b=3D-650 c=3D0 > boss moveColl spec=3Dsp1 a=3D-315 b=3D315 c=3D315 > boss moveColl spec=3Dsp1 a=3D-335 b=3D335 c=3D-315 >=20 > This was a very quick test that could be repeated a lot very easily. = It Working > our way up in loops gradually. This sequence would frequently kill the = sp1mech. > It wasn't consistent and it could happen with any one of the three = commands. > Once we identified this command as the main source of disconnects, we = stopped > running it. >=20 > 10:27Z started apogee morning cals. Didn't want to start BOSS cals = because we're > not sure we can trust it unsupervised, but we do have lots of darks = and biases. >=20 >=20 > ---- Mirror Numbers ---- >=20 >=20 > PRIMARY: > -------- > Scale: 1.000000 >=20 > MIGS TONIGHT NOMINAL > Axial A: 0.0100 0.0080 > Axial B: 0.0360 0.0330 > Axial C: -0.1130 -0.1140 > Trans D: -0.0369 -0.0395 > Lateral E: 0.3315 0.3310 > Lateral F: 0.5345 0.5335 >=20 > GALILS > Commanded: 12500 -1600 5350 47050 -8800 -8400 > Actual: 12500 -1600 5350 47050 -8800 -8400 >=20 > SETMIR VALUES > PrimDesOrient: 0.0 -59.3 -8.75 -512.73 1906.51 > PrimOrient: 0.01 -59.26 -8.77 -512.68 1906.6 >=20 > SECONDARY: > ---------- >=20 > Focus: 0.0000 > Air temp: 9.61 > Altitude: 30.000077845272088 >=20 > MIGS TONIGHT NOMINAL > Axial A: -9.4550 -9.4610 > Axial B: -9.4670 -9.4760 > Axial C: -9.4980 -9.5050 > Trans D: 0.0930 0.0880 >=20 > GALILS > Commanded: 4635900 4631300 4654100 -50 -3450 > Actual: 4607300 4607143 4607243 -50 -3450 >=20 > SETMIR VALUES > SecDesOrient: 3657.0 0.0 0.0 0.0 42.22 > SecOrient: 3656.99 0.01 -0.04 -0.02 42.68 >=20 >=20 >=20 > ---- Telescope Status ---- >=20 >=20 > Started ventilation > encl25m : open > encl25m : closed >=20 > No interlocks bypassed >=20 > Status at: 0:29:02Z > Telescope stowed at: 121.0, 30.0, 0.0 mount > Instrument mounted: 19 > Counterweights at: 509.4, 512.8, 509.5, 508.7 > LN2 autofill systems: Connected and turned on > 180L LN2 dewar scale: SP1 191.0 lbs, 21.7 psi >=20 >=20 >=20 >=20 > ---- Software Used ---- >=20 > Get Versions, 10/15/21, 23:24:59 > -- STUI software:=20 > STUI : 1.7.13 > alerts : 2.1.6 > apo : 3.0.0 > boss : 6.0.0 > -boss.specMec : 3.0.4 > guider : 4.0.0 > ecamera : FAILED > fliswarm : 0.1.1-alpha.0 > hartmann : 2.0.0 > hub : 5.0.0a0 > mcp : 2.1 > -mcp.plc : 34 > -mcp.azFiduci : v1_146 > -mcp.altFiduc : v1_146 > -mcp.rotFiduc : v1_146 > sop : 0.1.0-alpha.0 > tcc : 1.4.0 > apogee : not availble > apogeecal : not availble > apogeeql : v1_7_0dev > STUIAdditions : origin/sdss5 > -- done -- >=20 >=20 > ____________________________________________________________ > Logs of Nightly APO Activities > > = https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html = > ____________________________________________________________ > Logs of Nightly APO Activities > = https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html --Apple-Mail=_33FF1334-6471-46E1-9DD8-C230F3B7613B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hello= Jose, Dan, Dylan, and Audrey, Thanks for the tests. Tracy and = Ryan: A couple of nights ago, we have issue with connections of M1 and = M2 controllers, which was tracked down to a UPS unit. Does the Xyplex = terminal server also use that UPS unit? Thanks,-KaikeOn Oct 16, 2021, at 10:05 AM, Jos=C3=A9 S=C3=A1nchez-Gallego = = wrote:Hi Dylan, Audrey, Dan,Thanks for = this detailed testing and for Dan's detailed description of the problem = yesterday.It seems the problem is mostly concentrated on = the camera and mech controllers, while the DAQ has only happened once. = If we consider the failure of the DAQ a random netburner problem (and it = may not be) that would make some sense since the camera and mech are on = the same network (the usual 10.25) and connected to the Xyplex terminal = server, while the DAQ has its own dedicated fibre link and a dedicated = physical network interface.I've been looking at the = logs and while I see the moveColl commands and the disconnects that = happen after it, there doesn't appear to be anything that looks a = software error. I do see some tracebacks that are expected when one of = the controllers disconnects, but nothing that would indicate a cascade = somewhere else in the code that causes the disconnection.After the trouble with the galils the other night I wonder if = the problem could be with the Xyplex terminal server or thereabouts, = since all the devices we have had problems with have been connected to = it. If the connection is somehow unstable I could see how multiple = consecutive moveColl could be more likely to catch one of those = micro-disconnections and cascade that into the software needing to = reconnect. That could explain why the 4x15 min exposures didn't fail, = since during those the communication with the devices is more sporadic = (only periodic status requests) and those may not coincide with a lost = connection.This could also be a network problem in the 10.25 = network, but I'm not sure what's the best way of diagnosing that. Maybe = Shane and Tracey can look into adding or querying some diagnostic tools = on the network.A third possibility, just to make sure it's not = a software issue, would be to run the Python 2 version of bossICC. I = would expect if this is a software issue related to the migration to = Python 3 that the issue would be much more obvious, or at least = consistent, but it may be worth a try.Thanks,Jos=C3=A9 On Oct 16, 2021 = at 03:35:13, dgatlin@apo.nmsu.edu wrote: =20 = &n= bsp; APACHE POINT = OBSERVATORY SDSS 2.5M = &n= bsp; &nbs= p; OBSERVING LOG = &n= bsp; &nbs= p;Friday October 15, 2021 = &n= bsp; &nbs= p; (SJD 59503)---- Observing Team ----Dylan GatlinAudrey Oravetz (R)---- Observing Plan ----Test BOSS exposures, improve alt balance---- Observing Summary ----We didn't open tonight, and spent some time = testing the telescope balance and the boss disconnect = issues.Notes:- The = counterweights for cart 12 are now at 500, which is a much better = balance than before. It is a little top heavy at low = altitudes and bottom heavy at high altitudes.- We took a series of test exposures trying to = identify the source of the boss issues. We're not certain = if it's a software or connection issue, but we identified = some problematic commands and you should be able to reproduce the issue very quickly by looping a `boss moveColl` command. See = the main log for a complete description.---- Observing Log ----Mirror numbers in specAPOGEE cals started, = including 30 read FPIBOSS bias and dark taken, okay, = cannot take test flat/arc with current cartThe weather is nice outside, but the observing plan doesn't = call for opening, so we will not ventilate.The day staff noted particularly bad telescope = balance, so we took some time to test it and find a = better balance. It was set when we loaded cart 12 to 380 = and has stayed there since. At all altitudes, we have noticed that the = telescope tends to fall up.In= increments of 20, we increased the counterweights, then recorded the = motor current and voltage at various altitudes. The = measured current and voltage depends on the direction of = the previous move, so a set going down and a set going up = were collected. This procedure is quite cumbersome and took about 2 hours. We stopped with the counterweights at 500, which is = quite good. The telescope will still tend to fall up at = above 45 degrees according to our tests.=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3DAfter = stowing the telescope after the counterweight tests, we tried to = diagnose the boss sp1mech issues. We took a series of 60 = biases by simply looping the command `boss exposure bias` = with nothing else. This caused sp1mech to die once at = 05:17Z. It appears that the sp1mech died as soon as prereading began.After we restarted the sp1mech using `boss = reloadControllers controllers=3Dsp1mech` we got stuck = reading the previous exposure. We kept getting the following message every time we ran a `boss status`05:37:24 OBSERVER.dgatlin 126 boss w text=3D"exposureState = timer does not add up (remain=3D-1099.43 total=3D69.70)"The remain timer kept increasing, until we ran a `boss = exposure recover`, which read out the image, but killed = the loop command.We then took a set of 4 = 15 minute exposures. If we get an error per hour from = sp1mech, then we should see one of these exposures fail. Our last test = showed that you don't need a `boss moveColl` to cause a = failure, this test is a way of seeing if the constant = commands are an issue.No issues on those 4 = exposures, trying again just to be sure. This ran without = errorWe tried to run the moveColl command = in isolation to see if that was the main source of the = sp1mech failures. We looped the following commandboss moveColl spec=3Dsp1 a=3D650 b=3D-650 c=3D0boss moveColl spec=3Dsp1 a=3D-315 b=3D315 c=3D315boss moveColl spec=3Dsp1 a=3D-335 b=3D335 c=3D-315This was a very quick test that could be = repeated a lot very easily. It Working our way up in = loops gradually. This sequence would frequently kill the sp1mech. It wasn't consistent and it could happen with any one of the = three commands. Once we identified this command as the = main source of disconnects, we stopped running it.10:27Z started apogee morning cals. Didn't = want to start BOSS cals because we're not sure we can = trust it unsupervised, but we do have lots of darks and biases.---- Mirror Numbers ----PRIMARY:--------Scale: 1.000000MIGS TONIGHT NOMINALAxial A: 0.0100 = 0.0080Axial B: 0.0360 0.0330Axial C: = -0.1130 -0.1140Trans D: -0.0369 -0.0395Lateral E: 0.3315 0.3310Lateral F: = 0.5345 0.5335GALILSCommanded: 12500 -1600 5350 47050 -8800 -8400Actual: 12500 -1600 5350 47050 -8800 = -8400SETMIR VALUESPrimDesOrient: 0.0 -59.3 -8.75 -512.73 1906.51PrimOrient: 0.01 -59.26 -8.77 -512.68 = 1906.6SECONDARY:----------Focus: = 0.0000Air temp: = 9.61Altitude: = 30.000077845272088MIGS = TONIGHT NOMINALAxial A: -9.4550 -9.4610Axial B: -9.4670 -9.4760Axial C: -9.4980 = -9.5050Trans D: 0.0930 0.0880GALILSCommanded: 4635900 4631300 4654100 -50 = -3450Actual: 4607300 4607143 4607243 = -50 -3450SETMIR VALUESSecDesOrient: 3657.0 0.0 0.0 0.0 42.22SecOrient:= 3656.99 0.01 -0.04 -0.02 42.68---- Telescope Status ---- = Started ventilation = encl25m : open = encl25m : closedNo = interlocks bypassedStatus at: = 0:29:02ZTelescope stowed at: 121.0, = 30.0, 0.0 mountInstrument mounted: = 19Counterweights at: 509.4, 512.8, 509.5, = 508.7LN2 autofill systems: Connected and turned = on180L LN2 dewar scale: SP1 191.0 lbs, = 21.7 psi---- Software Used ----Get Versions, 10/15/21, 23:24:59 -- STUI = software: STUI = : 1.7.13alerts : 2.1.6apo = : 3.0.0boss : = 6.0.0-boss.specMec : 3.0.4guider = : 4.0.0ecamera = : FAILEDfliswarm = : 0.1.1-alpha.0hartmann = : 2.0.0hub = : 5.0.0a0mcp = : 2.1-mcp.plc : 34-mcp.azFiduci : v1_146-mcp.altFiduc : = v1_146-mcp.rotFiduc : v1_146sop = : = 0.1.0-alpha.0tcc = : 1.4.0apogee : not = availbleapogeecal : not = availbleapogeeql : = v1_7_0devSTUIAdditions : origin/sdss5 = -- done --____________________________________________________________Logs of Nightly APO Activities https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/= index.html ____________________________________________________________Logs of Nightly APO Activities https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/= index.html= --Apple-Mail=_33FF1334-6471-46E1-9DD8-C230F3B7613B--