Subject: Re: [sloan-25mlog 2098] 2.5m obslog 20150205 (SJD 57059) From: Conor Sayres Date: Fri, 6 Feb 2015 12:13:30 -0800 Ok, for now lets keep an eye on it, and please note whenever it happens = (the STUI log pastes are very helpful). If you happen to have made a note of when this happened over the past = two days, I=92d like to investigate. I had thought that TCC bugs (fixed = now) were responsible for the axis errors we saw the past couple of = nights. During the event last night, the TCC was happy. =20 It could be that the TCC is getting too close to the MCPs motion limits, = but we adopted the same limits that were present in the old TCC... On Feb 6, 2015, at 11:52 AM, Kaike Pan wrote: > Hi Concor, >=20 > Thanks for looking into this. See commends below.=20 >=20 >> On Feb 6, 2015, at 12:39 PM, Conor Sayres wrote: >>=20 >> Hi Kaike, >>=20 >> I looked into the TCC logs, and discussed this with Russell. It = doesn=92t look like a TCC issue. It appears the the MCP simply gave up = in the middle of a slew. Here=92s what the TCC logs indicate: >>=20 >> 2:23:41 Slew Issued >> 2:23:46 Slewing, expected duration 76.11 >> 2:23:59 Out of closed loop > 2:25:02 Slew Ends >>=20 >> If this becomes a regular occurrence (or even reproducible) we could = think about troubleshooting, perhaps lowering the velocity limits for = computed slews? But right now it just looks like an MCP burp to us. >=20 > I don=92t know what trigged the bake engaging during slews. However, = this rarely happens with the old tcc, maybe couple of times per year. > Since we switch to the new tcc, the frequency is much higher, we saw = it almost every day.=20 >=20 >>=20 >> As for the stop-button in errors: >>=20 >> Axis errors/warnings are defined in the axelim block (BadStatusMask = and WarnStatusMask). Perhaps Dan or Kaike could edit this file to = arrange things the way you want to see em? Right now the stop button in = bit indicates an error (which is how we thought the old tcc operated). = You can move that to indicate a warning instead. =20 >=20 > Is there a way to know a stop button is engaged (so the telescope is = not ready to move). If yes, it=92s better not to outputs badaxisstat = errors at all (neither warming nor error) when a stop button is engaged. >=20 >>=20 >> This is a good chance to practice with the git and the tccdata = directory.=20 >>=20 >> http://www.apo.nmsu.edu/Telescopes/TCC/OperatorsManual.html#DataFiles >>=20 >> Note, when you make a change to any block, you will need to add and = commit your changes using git. This allows us to retain a complete = history of each file change in the directory.=20 >>=20 >> You will need to "load block AxeLim" and potentially "set inst" for = the new axis warning settings to take effect. >>=20 >> Conor >>=20 >> On Feb 6, 2015, at 9:24 AM, Kaike Pan wrote: >>=20 >>> Hi Conor and Russell, >>>=20 >>> Looks like that the axes brake engaging happens more often with the = new tcc than with the old >>> one. Can you please investigate what triggers this? =46rom time = stamps of dome flats, I have rough >>> time stamps for the two occurrences. Thanks, >>>=20 >>> -kaike >>>=20 >>>=20 >>>> On Feb 6, 2015, at 6:24 AM, asimmons@apo.nmsu.edu wrote: >>>>=20 >>>>=20 >>>> APACHE POINT OBSERVATORY SDSS 2.5M >>>>=20 >>>> OBSERVING LOG >>>> Thursday February 05, 2015 >>>>=20 >>>> (SJD 57059) >>>>=20 >>>> ---- Observing Team ---- >>>>=20 >>>> Dmitry Bizyaev >>>> Audrey Oravetz >>>>=20 >>>> Kaike Pan (afternoon) >>>>=20 >>>> ---- Observing Summary ---- >>>>=20 >>>> Clear, productive night. >>>>=20 >>>> Cart 9, APOGEE2 plate 7773, 200-45; 2XABBA; m-36,195; f-284 >>>> Cart 4, APOGEE2&MANGA plate 8118, SQRT-1; 2XABBA; = m-20,39,195,278,288; f-284 >>>> 4 X 15 min exposures, C >>>> Cart 8, APOGEE2 plate 7942, 191+00; 2XABBA; m-35,184,195; = f-88,158,217,227,290 >>>> Cart 3, APOGEE2&MAN plate 8044, 176+00; 2XABBA; = m-195;f-133,158,247,248,250,258 >>>> 4 X 15 min exposures, C >>>> Cart 7, APOGEE2 plate 8347, 210+60; 2XABBA; m-81,195; f-197,247,268 >>>> Cart 2, APOGEE2&MANGA plate 8098, 120+75; 5XAB; m-157,163,195,227 >>>> 5 X 15 min exposures, C >>>> Sky Flat >>>> Cart 1, APOGEE2&MANGA plate 8268, 300+75; 2XABBA; m-195,258,259; = f-217,227,248 >>>> 4 X 15 min exposures, C >>>> Cart 6, APOGEE2 plate 7953, M5PAL5; 5XAB; m-181,195; f-15,34,297 >>>>=20 >>>> Notes to developers and engineers: >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>>=20 >>>> After the dome flats (with the gotoGangChange command) with the = first two >>>> cartridges of the night, the altitude axis brake engaged when = slewing >>>> to the gangChange position. A clip of some STUI messages were = copied to the >>>> problem section. There were no obvious indications on our part as = to why the >>>> brake came on. >>>>=20 >>>> We are running tagged version of the new tcc: 1.0.5. It outputs = error floods on >>>> axis status when the telescope is parked with stop buttons (see = afternoon >>>> section for error example), please clean these errors.=20 >>>>=20 >>>>=20 >>>> ---- Observing Plan ---- >>>>=20 >>>> Thu 5 FEB 2015 57059.0 0.97 01:17 12:59 03:17 15:02 = Day =20 >>>> APOGEE-2 (11.7 h)=20 >>>>=20 >>>>=20 >>>> ---- Observing Log ---- >>>>=20 >>>>=20 >>>> When the telescope is parked at stow or other position with stop = buttons, the >>>> new tcc outputed a error flood: >>>>=20 >>>> Here is one example of the errors: >>>> 22:49:48 .tcc 0 tcc e BadAzStatus; >>>> AzStat=3D121.000213,0.000000,4929893388.51704,0x00007800; = AzDTime=3D0.14 >>>> 22:49:48 .tcc 0 tcc e BadAltStatus; >>>> AltStat=3D30.000001,0.000000,4929893388.67665,0x00007800; = AltDTime=3D0.14 >>>> 22:49:49 .tcc 0 tcc e BadRotStatus; >>>> RotStat=3D0.000924,0.000000,4929893388.83901,0x00007800; = RotDTime=3D0.13 >>>>=20 >>>> 23:45Z mounted cart 9, started ventilation. >>>>=20 >>>> Night: >>>> =3D=3D=3D=3D=3D=3D >>>>=20 >>>> 00:40Z - Opened enclosure. Started filling the spectrographs. >>>>=20 >>>> 00:40Z - gotoField cart 9, APOGEE2 plate 7773, 200-45. GSFOGTF. >>>> 01:16Z - 2 X ABBA exposures. Seeing 1.9". Clear. >>>>=20 >>>> 57059 12 01:00 14970012 Dark 10 A 13.0 9-7773 a-b-c = - - =20 >>>> 57059 13 01:02 14970013 Dark 10 A 13.0 9-7773 a-b-c = - - =20 >>>> 57059 14 01:15 14970014 Object 47 A 13.0 9-7773 a-b-c = - - =20 >>>> 57059 15 01:23 14970015 Object 47 B 13.5 9-7773 a-b-c = - - =20 >>>> 57059 16 01:32 14970016 Object 47 B 13.5 9-7773 a-b-c = - - =20 >>>> 57059 17 01:40 14970017 Object 47 A 13.0 9-7773 a-b-c = - - =20 >>>> 57059 18 01:48 14970018 Object 47 A 13.0 9-7773 a-b-c = - - =20 >>>> 57059 19 01:57 14970019 Object 47 B 13.5 9-7773 a-b-c = - - =20 >>>> 57059 20 02:05 14970020 Object 47 B 13.5 9-7773 a-b-c = - - =20 >>>> 57059 21 02:13 14970021 Object 47 A 13.0 9-7773 a-b-c = - - =20 >>>> 57059 22 02:22 14970022 DomeFlat 5 A 13.0 9-7773 a-b-c = - 107=20 >>>> 57059 23 02:25 14970023 Dark 10 A 13.0 9-7773 a-b-c = - - =20 >>>> 57059 24 02:27 14970024 Dark 10 A 13.0 9-7773 a-b-c = - - >>>>=20 >>>> [observer@sdss4-hub ~]$ aptest 57059 14970022 >>>> flat=3D /data/apogee/utr_cdr/57059/apRaw-14970022.fits >>>> master=3D /home/observer/bin/apRaw-03720068.fits >>>> nfibers=3D 300 >>>> average ratio=3D 0.819481677184 >>>> missing fibers (flat/master >>> missing: fiber=3D 36, intFlat=3D 72, intMaster=3D 5225, = ratio=3D0.02 >>>> missing: fiber=3D 195, intFlat=3D 100, intMaster=3D 6056, = ratio=3D0.02 >>>> faint fibers (0.2 >>> faint: fiber=3D 284, intFlat=3D 2243, intMaster=3D 5716, = ratio=3D0.479 >>>>=20 >>>> The altitude brake came on during the slew to gangChange position = after the dome >>>> flat was taken. There was no indication as to why the brake = engaged. The brakes >>>> were cleared with axis stop/axis init. See problem section for a = small clip of >>>> the STUI message window during this time. >>>=20 >>> Looks like that it happened between: 02:22 to 02:25 >>>=20 >>>>=20 >>>> ----------------- >>>>=20 >>>> 02:30Z - gotoField cart 4, APOGEE2&MANGA plate 8118, SQRT-1. = GSFOGTF. >>>> 02:43Z - 2 X ABBA exposures. Seeing 2.0". Clear. >>>> 02:35:34 sp1.motor.move=3D 2504, 2501, 2502 >>>> 02:35:39 sp2.motor.move=3D 1165, 1165, 1165 >>>> 02:35:40 Hartmann collimate output on cart #4 >>>> sp1: offset: r=3D2308 (Out of focus); b=3D-8.5 (Out of focus)=20 >>>> sp2: offset: r=3D1262 (Out of focus); b=3D-3.4 (In Focus)=20 >>>> sp1: pred. move: spAverageMove=3D 2503 >>>> sp2: pred. move: spAverageMove=3D 1166 >>>> sp1: pred. spResiduals: r=3D-195, b=3D-0.6, txt=3DOK, spTemp =3D = 6.4 >>>> sp2: pred. spResiduals: r=3D95, b=3D0.3, txt=3DOK, spTemp =3D 5.9 >>>> 02:43Z - 4 X 15 min exposures, C. Seeing 2.0". >>>> 02:41:30 boss exposure 900.0, file=3D194096, C >>>> 02:58:13 boss exposure 900.0, file=3D194097, C >>>> 03:15:01 boss exposure 900.0, file=3D194098, C >>>> 03:31:44 boss exposure 900.0, file=3D194099, C >>>>=20 >>>> 57059 25 02:41 14970025 Object 47 A 13.0 4-8118 a-b-c = - - =20 >>>> 57059 26 02:49 14970026 Object 47 B 13.5 4-8118 a-b-c = - - =20 >>>> 57059 27 02:57 14970027 Object 47 B 13.5 4-8118 a-b-c = - - =20 >>>> 57059 28 03:06 14970028 Object 47 A 13.0 4-8118 a-b-c = - - =20 >>>> 57059 29 03:14 14970029 Object 47 A 13.0 4-8118 a-b-c = - - =20 >>>> 57059 30 03:23 14970030 Object 47 B 13.5 4-8118 a-b-c = - - =20 >>>> 57059 31 03:31 14970031 Object 47 B 13.5 4-8118 a-b-c = - - =20 >>>> 57059 32 03:39 14970032 Object 47 A 13.0 4-8118 a-b-c = - - =20 >>>> 57059 33 03:48 14970033 DomeFlat 5 A 13.0 4-8118 a-b-c = - 103=20 >>>> 57059 34 03:54 14970034 Dark 10 A 13.0 4-8118 a-b-c = - - =20 >>>> 57059 35 03:55 14970035 Dark 10 A 13.0 4-8118 a-b-c = - -=20 >>>>=20 >>>> [observer@sdss4-hub ~]$ aptest 57059 14970033 >>>> flat=3D /data/apogee/utr_cdr/57059/apRaw-14970033.fits >>>> master=3D /home/observer/bin/apRaw-03720068.fits >>>> nfibers=3D 300 >>>> average ratio=3D 0.789920125862 >>>> missing fibers (flat/master >>> missing: fiber=3D 20, intFlat=3D 43, intMaster=3D 5370, = ratio=3D0.01 >>>> missing: fiber=3D 39, intFlat=3D 60, intMaster=3D 5630, = ratio=3D0.01 >>>> missing: fiber=3D 195, intFlat=3D 23, intMaster=3D 6056, = ratio=3D0.00 >>>> missing: fiber=3D 278, intFlat=3D 42, intMaster=3D 5319, = ratio=3D0.01 >>>> missing: fiber=3D 288, intFlat=3D 195, intMaster=3D 5518, = ratio=3D0.04 >>>> faint fibers (0.2 >>> faint: fiber=3D 284, intFlat=3D 2459, intMaster=3D 5716, = ratio=3D0.545 >>>>=20 >>>> Another instance of the altitude brake engaging after the dome flat = at the start >>>> of the slew to gangChange position. >>>> =97=97=97=97=97=97=97=97 >>> It happened after/around 03:48. >>=20 >=20