From: Bill Ketzeback Subject: Re: [sloan-25mlog 5829] Re: 2.5m obslog 20220101 (SJD 59581) Date: Sun, 2 Jan 2022 12:30:37 -0700 --Apple-Mail-45006431-61AF-4F82-AFB8-8AACEFAB716E Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable No, when I left at noon we were still on generator power.=20 Sent from my iPhone > On Jan 2, 2022, at 12:17 PM, Jos=C3=A9 S=C3=A1nchez-Gallego wrote: >=20 > =EF=BB=BF > Hi Dan, >=20 > That sounds more concerning then. If the temperature was only a few degree= s below zero and the chiller should work up to -29C, then something is happe= ning beyond the software issue of jaeger not reseting the set point fast eno= ugh. >=20 > Is the observatory back on normal power? I wonder if the generator is not p= roviding enough power to the chiller and it cannot cool the fluid enough whi= le on generator power. If the power came back today at around 9:27 am, that'= s a good indication that the problem may be with the generator. >=20 > Thanks, >=20 > Jos=C3=A9 >=20 >=20 >> On Jan 2, 2022 at 10:47:57, Daniel Oravetz wrote:= >> Thanks, Jos=C3=A9. It looks like it only took all of 5-10 minutes for the= >> chiller temperature to go from 13C to 1C. That's great to see. >>=20 >> After the power bump at ~19:00 UTC yesterday, the ambient air temperature= >> was only slightly > operating limit of (-10 to -12C) -- so when the chiller temperature >> stopped dropping it wasn't as if we were experiencing unusually cold >> weather. When Dylan was flipping through the manual last night, he also >> noted in the manual that "The chiller can operate with full cooling >> capacity of ambient air temperatures from -29C and up to 35C." >>=20 >>> Hi all, >>>=20 >>> It seems this morning something changed and the compressor alarm went of= f. >>> The temperature outside is slightly warmer, but still below zero, not su= re >>> if that's what did it. I restarted jaeger which reset the set point to o= ne >>> below ambient and the chiller temperature is now happily diving from 13 >>> degC to 1 degC. I haven't seen any alarms indicating that the set point i= s >>> too far from the current temperature. >>>=20 >>> I've added some extra logic to jaeger to check the set point every minut= e. >>> If the current set point for the chiller does not match the last one tha= t >>> jaeger set (which should happen if the chiller reboots and resets to 20 >>> degC), it will reset the set point to one below ambient. Hopefully that'= s >>> fast enough to prevent a significant warm up. I haven't tested this code= >>> and since the logic is becoming increasingly complicated it may be good t= o >>> schedule a test to confirm it's actually working. >>>=20 >>> About the FVC issues last night, I think those points to some issue with= >>> the generator power and the cameras. I noticed that the FVC stopped >>> reporting yesterday around 2pm and then it restarted a couple times but >>> only for a few moments. I did a fliswarm reconnect -c fvc now and that >>> seems to have restored it, let's see if it stays like that. >>>=20 >>> Thanks, >>>=20 >>> Jos=C3=83=C2=A9 >>>=20 >>>=20 >>> On Jan 2, 2022 at 03:30:04, dgatlin@apo.nmsu.edu wrote: >>>=20 >>> > >>> > APACHE POINT OBSERVATORY SDSS 2.5M >>> > >>> > OBSERVING LOG >>> > Saturday January 01, 2022 >>> > >>> > (SJD 59581) >>> > >>> > ---- Observing Team ---- >>> > >>> > Dylan Gatlin >>> > Dan Oravetz (R) >>> > >>> > >>> > ---- Observing Plan ---- >>> > >>> > More dither loop tests >>> > >>> > >>> > ---- Observing Summary ---- >>> > >>> > We were on generator power, and tried to see how the fps would perform= >>> > like that. >>> > We ran into some issues with the fvc and some false collisions that ma= y >>> > be >>> > related, but we stopped once we noticed that the chiller was running a= t >>> > really high pressure and at a high temperature. It was mostly caused b= y >>> > the >>> > loss of power, but the extreme cold may be contributing to it. See the= >>> > main log >>> > for the whole story. >>> > >>> > There is ~2" of new snow since the plows came through yesterday. >>> > >>> > Problem collisions and snapshots: snapshot 2 with robot 507, snapshot 5= >>> > with >>> > fiber 184, snapshot 7 with fiber 184. >>> > >>> > ---- Observing Log ---- >>> > >>> > Afternoon: >>> > There are about 2 inches of fresh snow, but most things were plowed >>> > today. >>> > Snow >>> > is everywhere and the wind is blowing. As of 22:48Z we were on generat= or >>> > power, >>> > so all telescope systems will stay off for now. >>> > APOGEE cals taken, including 30 read FPI >>> > BOSS cals taken, blue ring not adjusted >>> > >>> > 22:42:47 boss exposure 4.0, file=3D335866 >>> > 22:43:27 boss exposure 4.0, file=3D335867 >>> > 22:44:13 sp1.motor.move=3D -1274, -1273, -1273 >>> > 22:44:13 Hartmann collimate output on cart #19 >>> > sp1: offset: r=3D -1649 (Out of focus); b=3D 2.8 (In Focus) >>> > sp1: pred. move: spAverageMove=3D -1274 >>> > sp1: pred. resid: r=3D -375, b=3D -1.2, txt=3D OK, spTemp =3D 2.6 >>> > 22:44:36 boss exposure 4.0, file=3D335868 >>> > 22:46:35 boss exposure 25.0, file=3D335869 >>> > >>> > Mirror numbers in spec >>> > >>> > After plowing my way to the enclosure, I found about a quarter inch of= >>> > snow along >>> > the inside of the west wall. Using a shovel, I carefully scooped snow >>> > out >>> > of the >>> > enclosure. I laid down a few absorbant socks along the az seam but >>> > couldn't find >>> > more pads. If the current snow melts, there should be enough pads to >>> > absorb it. >>> > The pads currently there are frozen, so they can't be removed. >>> > >>> > Night: >>> > >>> > 00:57Z We turned on the fps to see how it will behave on a generator >>> > 00:58:19 CmdStarted: APO.dgatlin_2 96 jaeger goto 483 180 0 >>> > First I tried to move a single fiber at the edge, but this failed with= a >>> > collision detected at robot 507, which is no where near 483 and wasn't= >>> > commanded >>> > to move. The second time I tried this command, it went fine. >>> > >>> > 01:02:21 CmdStarted: APO.dgatlin_2 102 jaeger configuration load 36165= >>> > This failed when I first moved to it due to fiber 184, the same thing >>> > happened >>> > after a second try >>> > >>> > I tried running an fvc loop after the second time since it looked like= >>> > the >>> > fibers >>> > were at their destinations. I got a strange error >>> > >>> > 01:08:40 CmdStarted: APO.dgatlin_2 112 jaeger fvc loop >>> > 01:08:40 CmdDone: jaeger.fliswarm 9 fliswarm talk -c fvc expose 5.0 >>> > 01:08:40 APO.dgatlin_2 112 jaeger f error=3D"Failed processing image: T= he >>> > exposure >>> > succeeded but did not output the filename." >>> > 01:08:40 CmdDone: APO.dgatlin_2 112 jaeger fvc loop >>> > >>> > Note that the exposure didn't actually succeed, it failed as soon as I= >>> > tried to >>> > start an fvc loop. This failure resulted in the fbi lamps staying on, s= o >>> > I >>> > turned them off manually. >>> > >>> > 01:12:31 CmdStarted: APO.dgatlin_2 117 fliswarm talk -c fvc expose 5 >>> > 01:12:31 CmdDone: APO.dgatlin_2 117 fliswarm talk -c fvc expose 5 >>> > >>> > Note that this command ended as soon as it started, which shouldn't be= >>> > possible >>> > After some review of the startup procedures, it looks like we missed t= he >>> > `fliswarm reconnect`, which was causing the errors. >>> > >>> > We ran an fvc loop. The fibers looked like they were at their >>> > destination, >>> > and >>> > and fvc loop confirmed it with an rms of 7.05. This happened a couple >>> > times >>> > last night where fibers had false collisions when they arrived at thei= r >>> > config destinations. >>> > >>> > We tried starting a dither loop, but the fvc camera disconnected again= . >>> > We had the same issues as above while taking exposures >>> > We ran `fliwarm reconnect` again which worked >>> > >>> > `fliswarm status` returned the following error >>> > >>> > 01:39:37 cerebro.tron 3018 fliswarm w error=3Dfvc,"Command 'status' fa= iled >>> > because >>> > of an uncaught error 'TypeError: 'NoneType' object is not a mapping'. >>> > See >>> > traceback in the log for more information." >>> > Design 36165, config 1008, snapshot 13, fvc 6 >>> > >>> > 1 (10) --- 01:42:26 >>> > jaeger configuration dither 0.1 >>> > jaeger fvc loop --no-apply >>> > jaeger configuration reverse >>> > tui wait 5 >>> > jaeger fvc loop --no-apply >>> > 2 (10) --- 01:44:32 >>> > jaeger configuration dither 0.1 >>> > jaeger fvc loop --no-apply >>> > jaeger configuration reverse >>> > tui wait 5 >>> > jaeger fvc loop --no-apply >>> > 3 (10) --- 01:46:38 >>> > jaeger configuration dither 0.1 >>> > jaeger fvc loop --no-apply >>> > jaeger configuration reverse >>> > tui wait 5 >>> > jaeger fvc loop --no-apply >>> > 4 (10) --- 01:48:43 >>> > jaeger configuration dither 0.1 >>> > jaeger fvc loop --no-apply >>> > jaeger configuration reverse >>> > tui wait 5 >>> > jaeger fvc loop --no-apply >>> > >>> > This failed because of the fvc loop like before. >>> > >>> > After this failed, we noticed that the chiller flow rate was 48 (too >>> > high), and >>> > after some inspection, the chiller temp was 17ish (too high). It looks= >>> > like >>> > we switched to generator power around 1800Z, and Jaeger set the chille= r >>> > temp >>> > to 20 briefly. The chiller went up to 20, but eventually jaeger lowere= d >>> > the >>> > temperature back to 1, and it never went back down. Once we realized >>> > this, >>> > we >>> > turned off the fps and called Ryan. Ryan suggested incrimentally >>> > lowering >>> > the >>> > set point so that it's only adjusting a few degrees at a time. >>> > >>> > The chiller has alarms which can now be seen in Grafana, two of them a= re >>> > triggered, ALARM_COMP_FAULT_SV and ALARM_SYS_FAULT_SV. If the chiller >>> > set >>> > point is more than 2.8deg C below the current temp, it will not be abl= e >>> > to >>> > cool >>> > to that level, you'll need to work your way down. On the chiller, ther= e >>> > is >>> > an >>> > led to indicate this, but it's not in Grafana anywhere. When we have a= >>> > set >>> > point within the 2.8deg limit, the chiller compressor still isn't >>> > turning >>> > on, >>> > which also has an LED indicator. When we raise the set point to above >>> > the >>> > current temperature, the heater turns on and that is indicated with an= >>> > LED. >>> > We talked to Ryan about this and it seems that the chiller is too cold= >>> > to >>> > operate. Unlike the rest of the chiller, the compressor is outside, an= d >>> > it >>> > is currently around -12C. >>> > >>> > It seems like the issue evolved as follows. >>> > >>> > 1. utility power failed, within a minute or so the generator kicked in= . >>> > 2. The chiller turned back on, and in the restart set the chiller temp= >>> > to >>> > 20C >>> > 3. A significant amount of time after the chiller raised its temp to >>> > 20C, >>> > jaeger >>> > kicked in and lowered the setpoint back to 1C. The chiller never tried= >>> > to >>> > cool >>> > back to 1C because the temperature difference between the current and >>> > setpoint >>> > were too large, so it did nothing and the temperature settled at some >>> > idle >>> > value of 17ish C. >>> > 4. The temperature dropped to -12C, raising the chiller fault alarm. >>> > 5. We cannot leave this state since the chiller is too cold to activat= e, >>> > even >>> > if we raise the setpoint to within 2.8C. >>> > >>> > It's possible that if we kept the setpoint within the 2.8C limit, the >>> > chiller >>> > compressor wouldn't be stuck, but this is mostly a guess and it needs t= o >>> > be >>> > tested. >>> > After adjusting the setpoint to 15.6, the chiller dropped VERY slowly t= o >>> > get >>> > close. We gradually lowered the setpoint to stay between 1 and 2 below= >>> > the >>> > actual value. At 11:20Z, it was at 15.2C >>> > Started morning cals at 10:50Z >>> > >>> > ---- BOSS Spectrographs Collimation Summary ---- >>> > >>> > Time TSP1 R off B off Move Resid >>> > >>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D >>> > 22:44:07 2.7 -1649 2.8 -1274 -1.2 >>> > 08:03:16 1.1 -1319 0.8 -788 -1.7 >>> > >>> > >>> > ---- Mirror Numbers ---- >>> > >>> > >>> > PRIMARY: >>> > -------- >>> > Scale: 1.000000 >>> > >>> > MIGS TONIGHT NOMINAL >>> > Axial A: 0.0040 0.0040 >>> > Axial B: 0.0030 0.0030 >>> > Axial C: -0.1140 -0.1150 >>> > Trans D: -0.0340 -0.0338 >>> > Lateral E: 0.3315 0.3315 >>> > Lateral F: 0.5350 0.5350 >>> > >>> > GALILS >>> > Commanded: 12500 -1600 5300 47050 -8800 -8400 >>> > Actual: 12500 -1600 5300 47050 -8800 -8400 >>> > >>> > SETMIR VALUES >>> > PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 >>> > PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 >>> > >>> > SECONDARY: >>> > ---------- >>> > >>> > Focus: 0.0000 >>> > Air temp: -5.89 >>> > Altitude: 30.0 >>> > >>> > MIGS TONIGHT NOMINAL >>> > Axial A: -9.6580 -9.6570 >>> > Axial B: -9.6720 -9.6710 >>> > Axial C: -9.6960 -9.6970 >>> > Trans D: 0.1060 0.1140 >>> > >>> > GALILS >>> > Commanded: 4896750 4898350 4914600 0 -3400 >>> > Actual: 4859249 4859324 4859161 0 -3400 >>> > >>> > SETMIR VALUES >>> > SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 >>> > SecOrient: 3857.03 0.05 0.01 0.01 41.61 >>> > >>> > >>> > >>> > ---- Telescope Status ---- >>> > >>> > >>> > No interlocks bypassed >>> > >>> > Status at: 0:19:56Z >>> > Telescope stowed at: 121.0, 30.0, -0.0 mount >>> > Instrument mounted: ECam >>> > Counterweights at: 685.3, 687.0, 705.0, 704.3 >>> > LN2 autofill systems: Connected and turned on >>> > 180L LN2 dewar scale: SP1 183.0 lbs, 21.1 psi >>> > >>> > >>> > >>> > >>> > ---- Software Used ---- >>> > >>> > /dev/sdb1 2.0T 154G 1.9T 8% /home >>> > 10.25.1.230:/data 21T 18T 3.0T 86% /data >>> > >>> > Get Versions, 01/01/22, 22:31:29 >>> > -- STUI software: >>> > STUI : 1.7.14 >>> > alerts : 2.2.0 >>> > apo : 3.0.1 >>> > boss : 6.0.1-alpha.0 >>> > -boss.specMec : FAILED >>> > guider : 4.0.0 >>> > ecamera : FAILED >>> > fliswarm : 0.1.1-alpha.0 >>> > jaeger : 0.13.1-alpha.0 >>> > kaiju : 1.2.0-alpha.3 >>> > coordio : 1.1.4-alpha.2 >>> > fps_calibrati : APO.2021.12.25 >>> > cherno : 0.1.0-alpha.0 >>> > hal : 0.1.0-alpha.0 >>> > hartmann : 2.0.1-alpha.0 >>> > hub : 5.0.0a0 >>> > mcp : 2.1 >>> > -mcp.plc : 34 >>> > -mcp.azFiduci : v1_148 >>> > -mcp.altFiduc : v1_148 >>> > -mcp.rotFiduc : v1_148 >>> > tcc : 1.4.0 >>> > apogee : not availble >>> > apogeecal : not availble >>> > apogeeql : v1_7_0dev >>> > STUIAdditions : nosop >>> > -- done -- >>> > >>> > --------------Other Versions-------------- >>> > sdss-obstools : 3.7.6 >>> > TPM : 1.2-31-gd7eb >>> > MCP : 2.1 >>> > idlspec2d : v6_0_4 >>> > Kronos : FAILED >>> > roboscheduler : FAILED >>> > >>> > >>> > >>> > ____________________________________________________________ >>> > Logs of Nightly APO Activities >>> > https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.h= tml >>> > >>> ____________________________________________________________ >>> Logs of Nightly APO Activities >>> https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.htm= l >>>=20 >>=20 >>=20 > ____________________________________________________________ > Logs of Nightly APO Activities > https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html --Apple-Mail-45006431-61AF-4F82-AFB8-8AACEFAB716E Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable No, when I left at noon we were still on ge= nerator power. Sent from my iPhoneOn Jan 2, 2022, at 12:17 PM, Jos=C3= =A9 S=C3=A1nchez-Gallego wrote:= =EF=BB=BFH= i Dan,That sounds more concerning then. If t= he temperature was only a few degrees below zero and the chiller should work= up to -29C, then something is happening beyond the software issue of jaeger= not reseting the set point fast enough.Is the observatory back on normal power? I wonder if the gener= ator is not providing enough power to the chiller and it cannot cool the flu= id enough while on generator power. If the power came back today at around 9= :27 am, that's a good indication that the problem may be with the generator.= Thanks,Jos=C3=A9 On Jan 2, 2022 at 1= 0:47:57, Daniel Oravetz wrote: =20 Thanks, Jos=C3=A9. It looks like it only took all of 5-10 minutes for th= echiller temperature to go from 13C to 1C. That's great to see.A= fter the power bump at ~19:00 UTC yesterday, the ambient air temperature= was only slightly < 0C -- certainly well above our low-temperatureope= rating limit of (-10 to -12C) -- so when the chiller temperaturestopped d= ropping it wasn't as if we were experiencing unusually coldweather. When= Dylan was flipping through the manual last night, he alsonoted in the m= anual that "The chiller can operate with full coolingcapacity of ambient= air temperatures from -29C and up to 35C." Hi all, It seems this morning s= omething changed and the compressor alarm went off. The temperature outside is slightly warmer, bu= t still below zero, not sure if that's what did it. I restarted jaeger which reset the set point t= o one below ambient an= d the chiller temperature is now happily diving from 13 degC to 1 degC. I haven't seen any alarms i= ndicating that the set point is too far from the current temperature. I've added some extra logic to jaeger to check the set point every minute= . If the current set p= oint for the chiller does not match the last one that jaeger set (which should happen if the chill= er reboots and resets to 20 degC), it will reset the set point to one below ambient. Hopefully that'= s fast enough to preve= nt a significant warm up. I haven't tested this code and since the logic is becoming increasingly c= omplicated it may be good to schedule a test to confirm it's actually working. About the FVC issues last night, I think those points to some i= ssue with the generato= r power and the cameras. I noticed that the FVC stopped reporting yesterday around 2pm and then it= restarted a couple times but only for a few moments. I did a fliswarm reconnect -c fvc now and th= at seems to have resto= red it, let's see if it stays like that. Tha= nks, Jos=C3=83=C2=A9 On Jan 2, 202= 2 at 03:30:04, dgatlin@apo.nmsu.edu wrote:>>  = ; &nb= sp; APACHE POINT OBSERVATORY SDSS 2.5M>> &n= bsp; = OBSERVING LOG> &n= bsp; = Saturday January 01, 2022>> &n= bsp; = &nbs= p;(SJD 59581)>> ---- Observing T= eam ---->> Dylan Gatlin> Dan Oravetz (R)= >>= > ---- Observing Plan ---->> More d= ither loop tests>>> ---- Observing Summary ---->> We were on generator power, and tried to see h= ow the fps would perform> like that.> W= e ran into some issues with the fvc and some false collisions that may> be> related, but we stopped once we notice= d that the chiller was running at> really high pressure and at a high temperature. It was mostly= caused by> the= > loss of power, but th= e extreme cold may be contributing to it. See the> main log> for the whole story.>> There is ~2" of new snow since the plows came through yesterday.= >> Problem collisions and snapshots: snaps= hot 2 with robot 507, snapshot 5> with>= ; fiber 184, snapshot 7 with fiber 184.>&= gt; ---- Observing Log ---->> Afternoon:= > There are about 2= inches of fresh snow, but most things were plowed> today.> Snow> is everywhere and the wind is blowing. As of 22:48Z we were on gener= ator> power,> so all telescope systems= will stay off for now.> APOGEE cals taken, including 30 read FPI> BOSS cals taken, blue ring not adjusted>> 22:42:47 boss exposure &n= bsp;4.0, file=3D335866= > 22:43:27 boss exposure 4.0, file=3D335867> 22:44:13 sp1.moto= r.move=3D -1274, -1273, -1273> 22:44:13 Hartmann collimate output on cart #19> sp1: offset: r=3D -1649= (Out of focus); b=3D 2.8 (In Focus)> sp1: pred. move: spAverageMove=3D -1274> sp1: pr= ed. resid: r=3D -375, b=3D -1.2, txt=3D OK, spTemp =3D 2.6> 22:44:36 boss exposure = 4.0, file=3D335868> 22:46:35 boss exposure 25.0, file=3D335869>> Mirror numbers in spec>> After plowing my way to the enclosure, I found about a= quarter inch of> s= now along> the insi= de of the west wall. Using a shovel, I carefully scooped snow> out> of the> enclosure. I laid down a few absorbant socks along the az s= eam but> couldn't f= ind> more pads. If t= he current snow melts, there should be enough pads to> absorb it.> The pads currently there are frozen, so they ca= n't be removed.>> Night:>> 00:57Z We turned on the fps to see how it will be= have on a generator>= ; 00:58:19 CmdStarted: APO.dgatlin_2 96 jaeger goto 483 180 0> First I tried to move a single f= iber at the edge, but this failed with a> collision detected at robot 507, which is no where ne= ar 483 and wasn't> c= ommanded> to move. T= he second time I tried this command, it went fine.>> 01:02:21 CmdStarted: APO.dgatlin_2 102 jaeger configuration l= oad 36165> This fai= led when I first moved to it due to fiber 184, the same thing> happened> after a second try>> I tried running an fvc loop after the second time since it= looked like> the> fibers> were at their destinations. I g= ot a strange error>= > 01:08:40 CmdStart= ed: APO.dgatlin_2 112 jaeger fvc loop> 01:08:40 CmdDone: jaeger.fliswarm 9 fliswarm talk -c fvc= expose 5.0> 01:08:= 40 APO.dgatlin_2 112 jaeger f error=3D"Failed processing image: The> exposure> succeeded but did not output the fi= lename."> 01:08:40 C= mdDone: APO.dgatlin_2 112 jaeger fvc loop>= > Note that the exposure didn't actually succeed, it failed as soon as I> tried to> start an fvc loop. This fai= lure resulted in the fbi lamps staying on, so> I> turned them off manually.>>= ; 01:12:31 CmdStarted: APO.dgatlin_2 117 fliswarm talk -c fvc expose 5> 01:12:31 CmdDone: APO.d= gatlin_2 117 fliswarm talk -c fvc expose 5>= > Note that this command ended as soon as it started, which shouldn't be> possible> After some review of the st= artup procedures, it looks like we missed the> `fliswarm reconnect`, which was causing the erro= rs.>> We ran an fvc loop. The fibers l= ooked like they were at their> destination,> and> and fvc= loop confirmed it with an rms of 7.05. This happened a couple> times> last night where fibers had false collision= s when they arrived at their> config destinations.>> We t= ried starting a dither loop, but the fvc camera disconnected again.> We had the same issues as a= bove while taking exposures> We ran `fliwarm reconnect` again which worked>> `fliswarm status` returned the following error>> 01:39:37 cerebro.tron 3018 fliswarm w error=3Df= vc,"Command 'status' failed> because> of= an uncaught error 'TypeError: 'NoneType' object is not a mapping'.> See> traceback in the log for more informatio= n."> Design 36165, c= onfig 1008, snapshot 13, fvc 6>>  = ; 1 (10) --- 01:42:26> jaeger configuration dither 0.1> jaeger fvc loop --no-apply> jaeger configuration= reverse> tui wait 5= > jaeger fvc loop -= -no-apply> &n= bsp;2 (10) --- 01:44:32> jaeger configuration dither 0.1> jaeger fvc loop --no-apply> jaeger configuration rev= erse> tui wait 5> jaeger fvc loop --no= -apply>  = ;3 (10) --- 01:46:38> jaeger configuration dither 0.1> jaeger fvc loop --no-apply> jaeger configuration rever= se> tui wait 5> jaeger fvc loop --no-a= pply> 4= (10) --- 01:48:43> jaeger configuration dither 0.1> jaeger fvc loop --no-apply> jaeger configuration reverse> tui wait 5> jaeger fvc loop --no-apply= >> This failed because of the fvc loop= like before.>> After this failed, we n= oticed that the chiller flow rate was 48 (too> high), and> after some inspection, the chiller temp was 17ish (too h= igh). It looks> lik= e> we switched to g= enerator power around 1800Z, and Jaeger set the chiller> temp> to 20 briefly. The chiller went up to 20, but eventua= lly jaeger lowered>= the> temperature b= ack to 1, and it never went back down. Once we realized> this,> we> turned off the fps and called Ryan. Ryan suggested incrimentally> lowering> the> set point so that it's only adjusting a f= ew degrees at a time.&= gt;> The chiller ha= s alarms which can now be seen in Grafana, two of them are> triggered, ALARM_COMP_FAULT_SV and A= LARM_SYS_FAULT_SV. If the chiller> set>= point is more than 2.8deg C below the current temp, it will not be able= > to> cool> to that level, you'll need to work your way down.= On the chiller, there= > is> an> led to indicate this, but= it's not in Grafana anywhere. When we have a> set> point within the 2.8deg limit, the chiller compressor still is= n't> turning> on,> which also has an LED indicator. When w= e raise the set point to above> the> c= urrent temperature, the heater turns on and that is indicated with an> LED.> We talked to Ryan about this and it s= eems that the chiller is too cold> to> o= perate. Unlike the rest of the chiller, the compressor is outside, and> it> is currently around -12C.>> It seems like the issue evolved as follows.>> 1. utility power failed, within a minute= or so the generator kicked in.> 2. The chiller turned back on, and in the restart set the chil= ler temp> to> 20C> 3. A significant amount of time after t= he chiller raised its temp to> 20C,> j= aeger> kicked in an= d lowered the setpoint back to 1C. The chiller never tried> to> cool> back to 1C because the temperature difference between the curre= nt and> setpoint> were too large, so i= t did nothing and the temperature settled at some> idle> value of 17ish C.> 4. The temperature dropped to -12C, raising the chiller f= ault alarm.> 5. We c= annot leave this state since the chiller is too cold to activate,> even> if we raise the setpoint to within 2.8C.>> It's possible that if we kept the se= tpoint within the 2.8C limit, the> chiller= > compressor wouldn't be stuck, but this is mostly a guess and it needs t= o> be> tested.> After adjusting the setpoint to 15.6, the= chiller dropped VERY slowly to> get> c= lose. We gradually lowered the setpoint to stay between 1 and 2 below> the> actual value. At 11:20Z, it was at 15.= 2C> Started morning= cals at 10:50Z>> ---- BOSS Spectrogra= phs Collimation Summary ---->> Time &n= bsp; TSP1 R off B off Move &n= bsp;Resid>> =3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D> 22:44:07 2.7  = ;-1649 2.8 -1274 -1.2> 08:03:16 1.1  = ;-1319 0.8 -788 -1.7>>> ---- Mirror Numbers ---->>= > PRIMARY:> --------> Scale: 1.000000>> MIGS TONIGHT NOMINAL> Axial A: 0.0040 0.0040> Axial B: 0.0030 0.0030> Axial C: -0.1140 -0.1150> Trans D: -0.0340 -0.0338= > Lateral E: 0.3315 0.3315> Lateral F: 0.5350 0.5350>> GALILS> Commanded: 12500 -1600 5300 47050 -8800 -84= 00> Actual: &= nbsp; 12500 -1600 5300 47050 -8800 -8400>> SETMIR VALUES> PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3> PrimOrient: 1.07 -59= .51 -9.21 -512.13 1906.31>> SECONDARY:= > ---------->> Focus: 0.000= 0> Air temp: = -5.89> A= ltitude: 30.0>> MIGS TONIG= HT NOMINAL> Axial A= : -9.6580 -9.6570> A= xial B: -9.6720 -9.6710> Axial C: -9.6960 -9.6970> Trans D: 0.1060 0.1140>> G= ALILS> Commanded: 4= 896750 4898350 4914600 0 -3400> Actual: 4859249 4859324 4859161 0 -3400>> SETMIR VALUES> SecDesOrient: 3857.0 0.0 0.0 0.0 42.22> SecOrient: &= nbsp;3857.03 0.05 0.01 0.01 41.61>>= >> ---- Telescope Status ---->>> No interlocks bypassed>> Statu= s at: 0:19:56Z&g= t; Telescope stowed at: 121.0, 30.0, -0.0 mount> Instrument mounted: EC= am> Counterweights a= t: 685.3, 687.0, 705.0, 704.3> LN2 autofill systems: Connected and turned on> 180L LN2 dewar scale: &n= bsp;SP1 183.0 lbs, 21.1 psi>>>>> ---- Software Used ---->> /dev/sdb1 &= nbsp; 2.0T 154G 1.9T &n= bsp;8% /home> 10.25= .1.230:/data 21T  = ;18T 3.0T 86% /data>> Get V= ersions, 01/01/22, 22:31:29> -- STUI software:> STUI : 1= .7.14> alerts  = ; : 2.2.0> apo &nb= sp; : 3.0.1= > boss : 6.0.1-alph= a.0> -boss.specMec := FAILED>= ; guider : 4.0.0> ecamera &nb= sp; : FAILED> f= liswarm : 0.1.1-alpha.0> jaeger &nb= sp; : 0.13.1-alpha.0> kaiju : 1.2.0-alpha.= 3> coordio &n= bsp; : 1.1.4-alpha.2> fps_calibrati : APO.2021.12.25> cherno = : 0.1.0-alpha.0> hal : 0.= 1.0-alpha.0> hartma= nn : 2.0.1-alpha.0> hub  = ; : 5.0.0a0> mcp := 2.1> -mcp.plc &nbs= p; : 34> -mcp.azFiduci : v1_148> -mcp.altFiduc : v1_148> -mcp.rotFiduc : v1_148> tcc &n= bsp; : 1.4.0> apogee : not availble> apogeecal &n= bsp; : not availble> apogeeql : v1_7_0dev> STUIAdditions : nosop> -- done -->> --------------Other Versions--------------> sdss-obstools &n= bsp; : 3.7.6> TPM = : 1.2-31-gd7eb> MCP &nb= sp; : 2.1> idlspec2d &nb= sp; : v6_0_4> Kronos &nbs= p; : FAILED> roboscheduler &nb= sp;: FAILED>>>> ____________________________________________________________> Logs of Nightly AP= O Activities > https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.htm= l> __________________________________= __________________________ Logs of Nightly APO Activities https://trac.sdss.org/tracmailman/browser/private/sloan= -25mlog/index.html= =20 ____________________________________________________________Logs of Nightly APO Activities https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index= .html= --Apple-Mail-45006431-61AF-4F82-AFB8-8AACEFAB716E--