Date: Sat, 6 Sep 2025 17:24:05 -0000 Subject: Re: [infrastructure 22577] Re: [sloan-25mlog 7873] 2.5m obslog From: kpan@apo.nmsu.edu I will check the data and try to generate a rotator fiducial table after I get back home tonight. Will send out a update > Regarding the rot fiducials, as far as I can say this is working but > we'll > need to confirm by creating the fiducial table. Currently the old fiducial > table is being loaded so the reported errors are huge. As we discussed, I > think the best approach would be to start by generating a fiducial table > by > just accepting the reported pos1 for each fiducial crossing (minus the > ones > that do not read, if any). With that fiducial table loaded you can do a > second pass using the procedure that you have followed so far and > plotFiducials. Not sure if that is necessary. > > One thing to note is that while the reported fiducial IDs are now > sequential, they have changed w.r.t. the IDs that were reported before for > a given physical position. The fiducial IDs now increase as one moves in > the positive direction and 0 degrees is between fiducials 41 and 42. This > is the new mapping, which we confirmed yesterday (ignore the different > colours and symbols) > > > > I'm also attaching it at a CSV table. > > Thanks, > > José > > > On Sep 6, 2025 at 03:11:13, Daniel Oravetz via sloan-25mlog sloan-25mlog@sdss.org> wrote: > >> >> APACHE POINT OBSERVATORY SDSS 2.5M >> >> OBSERVING LOG >> Friday September 05, 2025 >> >> (SJD 60924) >> >> ---- Observing Team ---- >> >> Dan Oravetz >> José Sánchez-Gallego >> >> ---- Observing Plan ---- >> >> Setup engCam and take test images. >> Work on ROT fiducials. >> >> ---- Observing Summary ---- >> >> Good progress made tonight (within the enclosure and under a rainy/foggy >> sky). >> >> José addressed a couple of bugs in the MCP: one related to ID mapping >> math >> and another related to an initialized, but unassigned, variable, which >> was >> ultimately causing the tlatch process to die. It's not clear how things >> ever worked before given the latter bug, but after he pushed-through the >> fixes, we were successfully able to move ROT through the range -180 to >> 360 >> without any oscillations and with the fiducials reading sequentially in >> both directions. Next step is to get a good set of fiducial-crossing >> data >> and build new tables to install and test. >> >> José also renamed all the SP2 keywords to 'SP1', so there aren't any >> SP2 >> keywords circulating anymore while running with the SP2 controller on >> SP1. >> >> I was eventually able to get engCam communicating earlier in the night >> and >> took a couple of test images. (The flat seems to indicate that the >> engCam >> is rather dusty.) The main problem was a faulty connection between the >> yellow ethernet cable from the camera itself to the female-to-female >> ethernet extender that is installed in the cartridge body of the engCam. >> That connection is very flaky and sensitive to any "wiggles" in the >> ethernet cable, so I bypassed it altogether and just connected the red >> ethernet cable directly to the camera. No other engCam communciation >> problems after that, besides the Galil. >> >> No luck talking to the engCam Galil. While it's doable to establish a >> telnet connection and ping the necessary address, there seems to be no >> device there. JohnD has plans to check the wiring/connector. >> >> Both Grafana and the MCPTPM GUI are displaying values of > the >> 180L dewar, despite it being connected and the liquid valve open. I >> verified two manual secondary fills tonight (and some primary fills >> while >> we were moving ROT), so not sure why the pressure isn't being reported >> properly (but the weight is displayed correctly). The most recent manual >> secondary fill completed a little after 09:00Z. >> >> Absorbent pads have been laid out along the west door. >> >> ---- Observing Log ---- >> >> >> 01:15Z - Checked engCam hardware connections, cleaned the glycol >> connectors, and then connected the glycol lines. >> >> Started ecameraICC software and verified a 'pong' reply to a STUI >> 'ecamera >> ping'. >> >> Tried a 'setTemp' command: >> >> 01:38:15 CmdStarted: OBSERVER.telescope 58 ecamera setTemp temp=-35 >> 01:38:15 OBSERVER.telescope 58 ecamera i text="setting camera cooler >> setpoint to -35.0 degC" >> 01:38:15 OBSERVER.telescope 58 ecamera f text=Camera; UNPARSEDTEXT="not >> connected: cannot set temperature." >> 01:38:15 CmdDone: OBSERVER.telescope 58 ecamera setTemp temp=-35 >> 01:38:15 .ecamera 7 ecamera w text="this command has already been >> finished!!!! (OBSERVER.telescope 7): setTemp temp=-35" >> >> 01:41:53 CmdStarted: OBSERVER.telescope 64 ecamera reconnect >> 01:41:54 .tcc 0 tcc i TCCPos=nan,nan,nan; >> AxePos=120.999935,29.999693,-0.005844; >> ObjInstAng=nan,nan,5263824436.8112555; >> SpiderInstAng=nan,nan,5263824436.8112555 >> 01:41:54 .tcc 0 tcc i AxisCmdState=Halted,Halted,Halted; >> AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT >> 01:41:58 .alerts 0 alerts i activeAlerts >> 01:41:58 .alerts 0 alerts i disabledAlertRules="(mcp.heartbeat, ok, -1)" >> 01:41:59 .ecamera 0 ecamera i text="trying to connect to camera at >> 10.25.1.71...." >> >> 01:42:24 .ecamera 0 ecamera w text="BAD THING: could not connect to >> camera: Alta camera connection is down" >> >> 01:46:24 CmdStarted: OBSERVER.telescope 65 ecamera reconnect >> 01:46:25 .boss 676 boss : >> 01:46:26 .tcc 0 tcc i TCCPos=nan,nan,nan; >> AxePos=120.999939,29.999685,-0.005844; >> ObjInstAng=nan,nan,5263824436.8112555; >> SpiderInstAng=nan,nan,5263824436.8112555 >> 01:46:26 .tcc 0 tcc i AxisCmdState=Halted,Halted,Halted; >> AxisErrCode=HaltRequested,HaltRequested,HaltRequested; AxisNoTrack=TTT >> 01:46:27 .apogeebpr 0 apogeebpr i BPRMode=bpr; ManualFill=off; >> AMIFill=off >> 01:46:27 .apogeebpr 0 apogeebpr i BPREncTemp=20.1; BPRInletTemp=19.8; >> BPROutletTemp=17.9 >> 01:46:27 .apogeebpr 0 apogeebpr i MKSSetPtPress=580.0; MKSPress=580.0; >> BPRManPress=37.57632799999999 >> 01:46:27 .apogeebpr 0 apogeebpr : BPRFillValve=closed; >> BPRVentValve=open; >> BPRBypassValve=closed; BPRDumpValve=open >> 01:46:27 .alerts 0 alerts i activeAlerts >> 01:46:27 .alerts 0 alerts i disabledAlertRules="(mcp.heartbeat, ok, -1)" >> 01:46:27 .alerts 0 alerts i activeAlerts >> 01:46:27 .alerts 0 alerts i disabledAlertRules="(mcp.heartbeat, ok, -1)" >> 01:46:27 .alerts 0 alerts i activeAlerts >> 01:46:27 .alerts 0 alerts i disabledAlertRules="(mcp.heartbeat, ok, -1)" >> 01:46:28 .ecamera 0 ecamera i text="trying to connect to camera at >> 10.25.1.71...." >> 01:46:28 OBSERVER.telescope 65 ecamera i version="2.3.0" >> 01:46:28 OBSERVER.telescope 65 ecamera i stack=1 >> 01:46:28 OBSERVER.telescope 65 ecamera i cameraConnected=True >> 01:46:28 OBSERVER.telescope 65 ecamera i binning=1,1 >> 01:46:28 OBSERVER.telescope 65 ecamera i dataDir=/data/ecam/60924; >> nextSeqno=1 >> 01:46:28 OBSERVER.telescope 65 ecamera i flatCartridge=-1; >> darkFile=None; >> flatFile=None >> 01:46:28 OBSERVER.telescope 65 ecamera i >> cooler=-60.0,19.0,17.8,0.0,0,Off >> 01:46:28 OBSERVER.telescope 65 ecamera i simulating=Off,None,1 >> 01:46:29 OBSERVER.telescope 65 ecamera : >> 01:46:29 CmdDone: OBSERVER.telescope 65 ecamera reconnect >> >> 01:46:48 CmdStarted: OBSERVER.telescope 66 ecamera status >> 01:46:48 OBSERVER.telescope 66 ecamera i version="2.3.0" >> 01:46:49 OBSERVER.telescope 66 ecamera i stack=1 >> 01:46:49 OBSERVER.telescope 66 ecamera i cameraConnected=True >> 01:46:49 OBSERVER.telescope 66 ecamera i binning=1,1 >> 01:46:49 OBSERVER.telescope 66 ecamera i dataDir=/data/ecam/60924; >> nextSeqno=1 >> 01:46:49 OBSERVER.telescope 66 ecamera i flatCartridge=-1; >> darkFile=None; >> flatFile=None >> 01:46:49 OBSERVER.telescope 66 ecamera i >> cooler=-35.0,16.9,17.6,20.9,0,RampingToSetPoint >> 01:46:49 OBSERVER.telescope 66 ecamera i simulating=Off,None,1 >> 01:46:49 OBSERVER.telescope 66 ecamera : >> 01:46:49 CmdDone: OBSERVER.telescope 66 ecamera status >> >> EngCam is working now, setTemp works, test images successfully obtained. >> Dusty looking flat exposure. >> ROT move 0 to 100: >> >> 05:51:32 APO.jose: Good. The IDs went from 42 to 62 which is what I >> would >> expect with the new mapping of fiducials for 0 to 100. >> >> ROT move 100 to 360: >> >> Fiducial 117 crossed last (~356 deg). >> >> During the moves we did not experience any ROT oscillations. >> >> Move from 360 to -180. >> >> Didn't notice any fiducials being read in the negative direction. The >> tlatch process died. >> >> Restarted the process. >> >> Move from 202 to 300: success. 105 last fiducial read. >> >> Move from 300 to 200: tlatch died immediately. >> >> Reverted to the tagged code to see if it happens there as well. >> >> Move from 236.5 to 275: successful move. Fiducials read. tlatch alive. >> >> Move from 275 to 245: successful move. Fiducials read in descending ID >> fashion. >> >> Jose: One clue is that the segfault happens before the log message about >> the rotator fiducial gets send. That constrains it a bit. >> >> Recompiled code. >> >> Move from 245 to 275: successful. >> >> Move from 275 to 245: tlatch died. >> >> I think the problem happens when we cross the same fiducial twice. Jose >> added a debug message. >> >> Move from 265 to 275: success. >> >> Move from 275 to 265: Jose: OK, it died and I see what the problem may >> be. >> It returned a huge fiducial ID. >> >> Repeated small moves after Jose added additional debug messages. >> >> Jose: So, the problem is indeed when it detects a crossing of the same >> fiducial. In that case the code should return ID=0 and that is a special >> case that basically skips most of the code. >> 07:10:44 OBSERVER.telescope: ok >> 07:10:49 APO.jose: But somehow it's returning a huge ID, in the 32000 >> range, and that crashes things below. I can see how it crashes things >> but >> not why it returns that number, >> >> Move 265 to 165: tlatch died immediately. No fiducial crossing >> registered. >> Not even Jose's debug messages. But there was a bug in a debug message. >> Jose fixed that and we tried again. >> >> Move from 165 to 65: success. Fiducials registered. >> >> Move from 65 to 80: tlatch died. No fiducials registered, but Jose is >> zeroing-in on where the change in value happens. >> >> Move from 80 to 95: successful. >> >> Move from 95 to 80: successful. Jose seems to have located the problem, >> but doesn't understand how things were able to work previously. >> >> Really cannot see how it was working before. Basically I was returning >> an >> initialised but not assigned variable. The 32000-ish value is probably a >> compiler value when you initialise an int. >> >> 07:35:17 APO.jose: I think we should go to 0 deg and call it good. >> >> 07:35:30 APO.jose: I think the next step is to build the new fiducial >> table and make sure it makes sense. >> >> _______________________________________________ >> Logs of Nightly APO Activities >> >> https://urldefense.com/v3/__https://mailman.sdss.org/archives/list/sloan-25mlog@sdss.org/message/GCE5G5KL3W7OTR55L3FUC4WTGYRBFZJQ/__;!!K-Hz7m0Vt54!kZRanwVrM8RgtVj-djBLi1yMaSIHt-g1H0A4uVscoBbzt09Wrs2cS7mxp3vaA-SQM-MDTASX8BtwIoq9aPC-7g$ >> >> > _______________________________________________ > Discussions of common APO site hardware and software work needed for SDSS > projects. For e.g. > - cartridges and cartridge handling > - fibers and fiber mapping > > https://mailman.sdss.org/archives/list/infrastructure@sdss.org/message/A6VB4SWSAZV453IMAMO4BNR6MEP3W7OH/ > >