Subject: Re: [sloan-25mlog 3295] 2.5m obslog 20161224 (SJD 57747) From: John Downey Date: Sun, 25 Dec 2016 10:20:49 -0700 --Apple-Mail=_7DC7A758-D8E0-43E3-995A-E0C05E487B8C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Dec 25, 2016, at 9:49 AM, Kaike Pan wrote: >=20 >=20 >> On Dec 25, 2016, at 7:49 AM, John Downey > wrote: >>=20 >>>=20 >>> 2) During afternoon checkout, We ran couple of tests on rotator = position when >>> cartridges are swapped. We dismounted/mounted cartridge twice in = auto-mode, and >>> noticed that rotator moved from 0.0 to +3' after a cartridge is = removed from the >>> telescope. Then I moved to manual mode, dismounted/mounted a = cartridge, and >>> found that rotator didn't moved when a cartridge is dismounted from = the >>> telescope. Viktor said that he noticed the same behavior last night: = rotator >>> seems more stable in manual-mode (not sure what causes this if it is = true), so >>> we leave it in manual mode. We request the cold observer to check = rotator >>> position when a cartridge is dismounted or mounted EVERY TIME.=20 >>=20 >> Ok so this is a new behavior - in the past I've not seen the rotator = move on dismount, but instead on axis init of one of the other axes if I = remember correctly. >=20 > The rotator wandered to +2.6=E2=80=99 after dismounted a cartridge, = and went further to 5.0=E2=80=99 after an =E2=80=9Caxis init=E2=80=9D. >=20 The first is something I've not seen before, the second is something = we've seen often and can be considered normal behavior. There is some = advantage to using the MCP for correction since you can initialize just = the ROT with the ALT pin still in (as we discussed last night). Between = the pin, ALT brakes, and windbaffle brake, there is little risk to using = the MCP to move the rotator even with no cart mounted. >>=20 >> I can't imagine what could be different about manual mode with = respect to allowing the rotator to wander. I do find it very curious = that we've seen various instances of the rotator preferring one position = slightly off 0:00:00, when the causes have basically been unrelated. = That is likely something physical, but I would then expect to see the = same when the amplifier was not holding the motor in position, such as = when a stop button is in. >=20 > That is what I feel. But viktor said that he didn=E2=80=99t see = rotator wandering in manual mode, and we tested dismounting/mounting one > cartridge and didn=E2=80=99t rotator wondering (actually it went to = +0.5=E2=80=99 from 0.0 after dismounting and mounting).=20 Until we see this demonstrated, I will still have to assume it is a = false correlation. It may be real, but it may also be the kind of = assumption that makes these hugely complex safety interlock systems very = difficult to troubleshoot. =20 The standard for it not wandering during dismount has to be a check of = the MCP position before and after - and I can't see the cold observer = stopping a cart change to go check the position on the MCP three times = during the process for each cart change in a night. This is why its = imperative to use the interlocked version until we understand the fault = - the interlock is there to protect the hardware, not to slow down cart = changes. The interlock is most likely doing exactly what it is supposed = to be doing, so bypassing it is playing with fire. >>=20 >> Did the experiment include putting the rotator back to 0:00:00 with = the MCP once it had wandered? >=20 > No problem to slew it back to 0.0. When viktor was issuing the = command, I watched the motion and conformed the rotator > moving.=20 Then what is the problem? Surely the time necessary to do this is = trivial compared to the risk to equipment through operating with the = non-interlocked pendant? I don't understand why the cart change = procedure cannot be modified to work around the problem rather than = switching the pendant to the dangerous position to get around it. The = interlock is there for a reason, and should only be bypassed if there is = some reason it is not possible to correct the condition causing the = interlock to prevent lift motion. >>=20 >> For the record, I am not happy with the habit of simply moving to = manual when there is a problem. If everyone is very careful it works = fine, but we do risk damaging slitheads if anyone forgets to be very = careful, and the longer we substitute manual mode for auto mode, the = more likely it is that someone will forget once or twice and have an = accident. >=20 > I used manual-mode much more in SDSS II and SDSS III, but barely used = it in SDSS IV. As long as observers being careful and checked the = rotator=E2=80=99s mark EVERY TIME when removing/mounting a cartridge, we = should be fine. Yes, I prefer to use the auto mode. I actually wrote a = the procedure for this in trouble shooting document for shared system: >=20 This is ok, so long as the responsibility for the safety of the hardware = and personnel is clearly and solely that of the person who moves the = pendant to manual. This is why you will find it back on the auto plug = every day - the person who moves it should always do so with the full = understanding of the risks they take, and should be the only one = operating the equipment in that mode. That same person should put the = pendant back on the auto plug at closing time - so that they do not = leave a hazardous condition for the next observer who may simply assume = that all is fine because that is what they found. This is the kind of = potential miscommunication that causes accidents. I would prefer to come down this week and look at what the problem = actually is than to run all week in manual mode. If the problem = persists I may have to come down Monday or Tuesday - and for me to know = that, we will have to work in auto until there is a problem and then = switch to manual. > Rotator Wondering off from 0.0 Position During Cartridge Change = > Background: When the lift is unresponsive or cartridge kinematic can't = be latched, it is very possible that the rotation is off from 0.0. Check = MCP to find rotator position. If the rotator is more than +/- 1.5 = arcminutes off from zero, the slit heads can catch on the spectrograph = opening, depending on how the cartridge is set on the alignment pins on = the lift (they allow a small amount of motion). If it is not due to = rotator being off, check the interlocks display to find the cause. For = your information, If being out of position, you will see one of the = switches rot_inst_chg_a, or rot_inst_chg_b shown as red in the interlock = logic tree. Those switches are set to allow a rotator position of = approximately +/-1.5 arcminutes around zero. >=20 > Procedure: A) For the case when the rotator is out of its zero = position before the old cartridge is dismounted or after the old = cartridge is removed from the telescope but before the new cartridge is = mounted: >=20 > Lower the lift (when rotator is out of position, the lift can't go up, = but it can always go down).=20 > Move the Lindy cart from underneath the telescope, > Pull the stop buttons out (BUT KEEP the alt pin in), > Initialize the axis and move the rotator to zero (this can be done via = STUI telescope control window, MCP, or tcc) > Engage a stop button. > Then proceed with the normal cartridge change. > B) For the case when the new cartridge is in the position but cannot = be latched: >=20 > turn the latch control key to the manual. > latch the new cartridge. > latch the lindy cart. (If the lift would not do down, you may need to = switch the pendant to the manual side). > turn the control key to the auto. > Note: looks like that there is alignment issue, when the new cart is = reaching the mounting position, rotator could go out of position. If the = cartridge is at position to be mounted, lindy cart has a force of ~1100 = lb, at least ~1050 lb, you can just switch the key to the manual mode = and latch the cartridge, you don't need to actually switch the the = pendant to the manual side. (and latch control) back to the auto side. >=20 >=20 >=20 >=20 >=20 >=20 >>=20 >>=20 >> ____________________________________________________________ >> Logs of Nightly APO Activities > >> = https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html = >=20 --Apple-Mail=_7DC7A758-D8E0-43E3-995A-E0C05E487B8C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 On Dec 25, 2016, at 9:49 AM, Kaike Pan = wrote:On Dec 25, 2016, at 7:49 AM, John Downey wrote:2) During afternoon checkout, We = ran couple of tests on rotator position whencartridges = are swapped. We dismounted/mounted cartridge twice in auto-mode, andnoticed that rotator moved from 0.0 to +3' after a cartridge = is removed from thetelescope. Then I moved to manual = mode, dismounted/mounted a cartridge, andfound that = rotator didn't moved when a cartridge is dismounted from thetelescope. Viktor said that he noticed the same behavior last = night: rotatorseems more stable in manual-mode (not sure = what causes this if it is true), sowe leave it in manual = mode. We request the cold observer to check rotatorposition= when a cartridge is dismounted or mounted EVERY TIME. Ok so this is a new behavior - in = the past I've not seen the rotator move on dismount, but instead on axis = init of one of the other axes if I remember correctly.The rotator = wandered to +2.6=E2=80=99 after dismounted a cartridge, and went further = to 5.0=E2=80=99 after an =E2=80=9Caxis init=E2=80=9D.The first is something I've not seen before, the = second is something we've seen often and can be considered normal = behavior. There is some advantage to using the MCP for correction = since you can initialize just the ROT with the ALT pin still in (as we = discussed last night). Between the pin, ALT brakes, and windbaffle = brake, there is little risk to using the MCP to move the rotator even = with no cart mounted.I can't imagine what could be different about = manual mode with respect to allowing the rotator to wander. I do = find it very curious that we've seen various instances of the rotator = preferring one position slightly off 0:00:00, when the causes have = basically been unrelated. That is likely something physical, but I = would then expect to see the same when the amplifier was not holding the = motor in position, such as when a stop button is in.That = is what I feel. But viktor said that he didn=E2=80=99t see rotator = wandering in manual mode, and we tested dismounting/mounting = onecartridge and didn=E2=80=99t rotator wondering = (actually it went to +0.5=E2=80=99 from 0.0 after dismounting and = mounting). Until we see this demonstrated, I will still have = to assume it is a false correlation. It may be real, but it may = also be the kind of assumption that makes these hugely complex safety = interlock systems very difficult to troubleshoot. The standard for it not wandering during dismount = has to be a check of the MCP position before and after - and I can't see = the cold observer stopping a cart change to go check the position on the = MCP three times during the process for each cart change in a night. = This is why its imperative to use the interlocked version until we = understand the fault - the interlock is there to protect the hardware, = not to slow down cart changes. The interlock is most likely doing = exactly what it is supposed to be doing, so bypassing it is playing with = fire.Did the experiment include putting the rotator back to = 0:00:00 with the MCP once it had wandered? No problem to slew it back to 0.0. When viktor = was issuing the command, I watched the motion and conformed the = rotatormoving. Then what is the problem? Surely the time = necessary to do this is trivial compared to the risk to equipment = through operating with the non-interlocked pendant? I don't = understand why the cart change procedure cannot be modified to work = around the problem rather than switching the pendant to the dangerous = position to get around it. The interlock is there for a reason, = and should only be bypassed if there is some reason it is not possible = to correct the condition causing the interlock to prevent lift = motion.For the record, I am not happy with the habit of simply = moving to manual when there is a problem. If everyone is very = careful it works fine, but we do risk damaging slitheads if anyone = forgets to be very careful, and the longer we substitute manual mode for = auto mode, the more likely it is that someone will forget once or twice = and have an accident.I used manual-mode much more in SDSS = II and SDSS III, but barely used it in SDSS IV. As long as = observers being careful and checked the rotator=E2=80=99s mark = EVERY TIME when removing/mounting a cartridge, we should be fine. Yes, I = prefer to use the auto mode. I actually wrote a the procedure for this = in trouble shooting document for shared system:This is ok, so long as the responsibility for the = safety of the hardware and personnel is clearly and solely that of the = person who moves the pendant to manual. This is why you will find = it back on the auto plug every day - the person who moves it should = always do so with the full understanding of the risks they take, and = should be the only one operating the equipment in that mode. That = same person should put the pendant back on the auto plug at closing time = - so that they do not leave a hazardous condition for the next observer = who may simply assume that all is fine because that is what they found. = This is the kind of potential miscommunication that causes = accidents.I would prefer to come = down this week and look at what the problem actually is than to run all = week in manual mode. If the problem persists I may have to come = down Monday or Tuesday - and for me to know that, we will have to work = in auto until there is a problem and then switch to manual.Rotator Wondering = off from 0.0 Position During Cartridge ChangeBackground: When the lift is unresponsive or cartridge = kinematic can't be latched, it is very possible that the rotation is off = from 0.0. Check MCP to find rotator position. If the rotator is more = than +/- 1.5 arcminutes off from zero, the slit heads can catch on the = spectrograph opening, depending on how the cartridge is set on the = alignment pins on the lift (they allow a small amount of motion). If it = is not due to rotator being off, check the interlocks display to find = the cause. For your information, If being out of position, you will see = one of the switches rot_inst_chg_a, or rot_inst_chg_b shown as red in = the interlock logic tree. Those switches are set to allow a rotator = position of approximately +/-1.5 arcminutes around zero.Procedure: A) For the case when the = rotator is out of its zero position before the old cartridge is = dismounted or after the old cartridge is removed from the telescope but = before the new cartridge is mounted:Lower the lift (when rotator is out of = position, the lift can't go up, but it can always go down). Move the Lindy cart from underneath the = telescope,Pull the stop buttons out (BUT KEEP the = alt pin in),Initialize the axis and move the rotator = to zero (this can be done via STUI telescope control window, MCP, or = tcc)Engage a stop button.Then = proceed with the normal cartridge change.B) For the case when the new cartridge = is in the position but cannot be latched:turn the latch control key to the = manual.latch the new cartridge.latch the lindy cart. (If the lift would not do down, you may = need to switch the pendant to the manual side).turn = the control key to the auto.Note: looks like that there is alignment issue, when the new = cart is reaching the mounting position, rotator could go out of = position. If the cartridge is at position to be mounted, lindy cart has = a force of ~1100 lb, at least ~1050 lb, you can just switch the key to = the manual mode and latch the cartridge, you don't need to actually = switch the the pendant to the manual side. (and latch control) back to = the auto side.____________________________________________________________Logs of Nightly APO Activities https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/= index.html= --Apple-Mail=_7DC7A758-D8E0-43E3-995A-E0C05E487B8C--