Subject: Re: [sdss3-25mlog 1001] Re: 2.5m obslog 20121225 (MJD 56287) From: Wendell & Cathy Jordan Date: Wed, 26 Dec 2012 13:59:24 -0700 I'm not sure if the warranty just ran out, but this UPS was just = installed in the comm rack during summer shutdown 2011. As I can best = recall, it's a top of the line 2500VA double conversion unit. The front = panel bar graph display showed that it was typically loaded from 10% to = 20% in normal operation over the past 18 months. A front panel hot swap = battery pack (comprised of three batteries) is already in the works. = Warranty issues can be examined when we get back. Wendell =20 On Dec 26, 2012, at 12:43 PM, Mark Klaene wrote: > let me guess the warranty just ran out ? >=20 >=20 > On Dec 26, 2012, at 12:39 PM, Dan Long wrote: >=20 >> Hi Mark, >>=20 >> Just so you know, there is a problem with the ~year old APC UPS = that controls the communications rack in the lower enclosure. A few = times in recent weeks we have found it in line only mode with a battery = fault indication. The first time we saw this, we ran a self test and it = came back normal. It happened a second time just before the break, the = self test again returned it to UPS mode, but caused us to look for new = batteries. Yesterday seemed to be yet another episode, which with the = repeated outages, caught the UPS in line only mode when the power = dropped. >>=20 >> I suppose it is possible that yesterday's problem was different, = that it really did run out of battery backup due to the constant = switching, but given the other events, I am suspicious of that UPS. = Therefore the switch to a different one today, in hopes of avoiding = several hours of recovery effort for a brief power problem. >>=20 >> Cheers, >>=20 >> Dan >>=20 >>=20 >>=20 >> On Dec 26, 2012, at 11:53 AM, Mark Klaene wrote: >>=20 >>>=20 >>> On Dec 26, 2012, at 10:11 AM, Kaike Pan wrote: >>>=20 >>>>=20 >>>> On Dec 26, 2012, at 4:30 AM, asimmons@apo.nmsu.edu wrote: >>>>=20 >>>>>=20 >>>>> APACHE POINT OBSERVATORY SDSS 2.5M >>>>>=20 >>>>> OBSERVING LOG >>>>> Tuesday December 25, 2012 >>>>>=20 >>>>> (MJD 56287) >>>>>=20 >>>>> ---- Observing Team ---- >>>>>=20 >>>>> Dan Oravetz >>>>> Audrey Oravetz >>>>>=20 >>>>> Kaike Pan (afternoon) >>>>> Joe Hueherhoff (help) >>>>> Matt Nelson (help) >>>>> Fritz Stauffer (help) >>>>> John Parejko (help) >>>>>=20 >>>>>=20 >>>>> ---- Observing Plan ---- >>>>>=20 >>>>> 25 Dec 2012 56287.0 0.87 01:17 12:51 00:34 12:09 APOGEE 12.2 = hours >>>>>=20 >>>>> ---- Observing Summary ---- >>>>>=20 >>>>> We witnessed several power bumps during tonight's afternoon = checkout. The power >>>>> bumps caused a lot of problems with the software. We were forced = to contact many >>>>> people for help. Eventually all of the systems were restored (see = afternoon >>>>> checkout and problem sections for details).=20 >>>>>=20 >>>>> The generator stayed on from ~22:00Z until 4-5:00Z.=20 >>>>>=20 >>>>> Clouds kept us closed all night. >>>>>=20 >>>>=20 >>>> A few issued for engineers to investigate (and for next = opt-phonecon?) >>>>=20 >>>> 1. Why a power glitch left the communication rack in the status = no-response. I though that a UPS will=20 >>>> provide power till the generator takes over. It should not lose = power.=20 >>>=20 >>> it was not a power glitch that caused the problems , we had over 100 = power glitches over an hour time. In some cases 10-20 in a minute. The = most excessive power problem I have ever seen. In the past when we see = anything like this starting to happen we manually place us on generator = . That was not an option this time given the holiday. The root of the = problems (besides what caused the power drops to begin with - likely = ice and trees--) is that incoming transmission line power is delta = wired and the line protection in cloudcroft can't sense some of these = issues that show up as big problems for our WYE system. I have tried to = get the co-op to fix this and they have "patched" it but the correct fix = is too expensive for them to do. The transfer switch on the generator = is relatively dumb and can't react to these sudden drops/corrections. = We have tweaked it as far as I dare to get it to switch quicker. = Without running on generator the UPS's constantly switches on-load / = off-load without having time to recharge. The FERRUPS UPSs are more = susceptible than others to this problem, but they also have their = pluses. It is possible this particular unit was less than 100% perfect = but works fine the other 99.99% of the time. It is also likely that the = load for this UPS is all non-linear and thus more of a problem in this = failure mode. >>>=20 >>> This is not a new problem. It is just one that shows up very = infrequently. We have fought this same issue for years and one reason we = have started a move to APC UPS units. >>>=20 >>> mk >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>>=20 >>>> 2. What caused the issue with rebooting MCP (the whole crate had to = be reset)? More information on >>>> what are consequences of reset the crate, systems on the same = crate:TPM, LN2, Memory, Sys IO, MEI >>>> will be rebooted too. Any side effects on rebooting them? >>>>=20 >>>> 3. apogeecal actor was at a hung status. Observers do not have = instructions for restart apogee and apogeecal. What >>>> are side effects for restarting these actors. Any concerns for = giving the instructions to observers?=20 >>>>=20 >>>> _____________________________________________________ >>>> Message 994 of sdss3-25mlog - Logs of nightly APO activities. >>>> = https://trac.sdss3.org/tracmailman/browser/private/sdss3-25mlog/all/994.ht= ml >>>> sdss3-25mlog@sdss3.org >>>> https://sdss3.org/mailman/listinfo/sdss3-25mlog >>>=20 >>> Mark Klaene >>> Site Operations Manager=20 >>> Apache Point Observatory voice: 575 437 6822 >>> PO Box 59 fax: 575 434 = 5555 >>> 2001 Apache Point Rd cell: 575 430 0210 >>> Sunspot NM 88349-0059 web: www.apo.nmsu.edu=20 >>>=20 >>>=20 >>>=20 >>>=20 >>> _____________________________________________________ >>> Message 997 of sdss3-25mlog - Logs of nightly APO activities. >>> = https://trac.sdss3.org/tracmailman/browser/private/sdss3-25mlog/all/997.ht= ml >>> sdss3-25mlog@sdss3.org >>> https://sdss3.org/mailman/listinfo/sdss3-25mlog >>=20 >> _____________________________________________________ >> Message 1000 of sdss3-25mlog - Logs of nightly APO activities. >> = https://trac.sdss3.org/tracmailman/browser/private/sdss3-25mlog/all/1000.h= tml >> sdss3-25mlog@sdss3.org >> https://sdss3.org/mailman/listinfo/sdss3-25mlog >=20 > Mark Klaene > Site Operations Manager=20 > Apache Point Observatory voice: 575 437 6822 > PO Box 59 fax: 575 434 5555 > 2001 Apache Point Rd cell: 575 430 0210 > Sunspot NM 88349-0059 web: www.apo.nmsu.edu=20 >=20 >=20 >=20 >=20 > _____________________________________________________ > Message 1001 of sdss3-25mlog - Logs of nightly APO activities. > = https://trac.sdss3.org/tracmailman/browser/private/sdss3-25mlog/all/1001.h= tml > sdss3-25mlog@sdss3.org > https://sdss3.org/mailman/listinfo/sdss3-25mlog