From: "Wilson, John C (jcw6z)" Subject: Re: [sloan-25mlog 4724] Re: 2.5m obslog 20191023 (SJD 58780) Date: Thu, 24 Oct 2019 18:43:10 +0000 --_000_BY5PR13MB3190E5DC28CAC7E86F39C0C9976A0BY5PR13MB3190namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi John, Thanks for this very informative email! It occurred to me a few days ago that surely there is a tremendous amount o= f useful procedural, background and troubleshooting information contained i= n the 20+ years of infrastructure and 2.5m obslog, etc., mailing lists. Th= ere must be an existing software solution for mining those emails for some = subset that could form the core of searchable documentation for the greater= 2.5m system. Thanks, John. ________________________________ From: sloan-25mlog on behalf of John Downey= Sent: Thursday, October 24, 2019 11:48 AM To: dmbiz@apo.nmsu.edu Cc: obs-report-25m@apo.nmsu.edu Subject: [sloan-25mlog 4724] Re: 2.5m obslog 20191023 (SJD 58780) Fiducial read problem is fixed. We will do test moves later this morning a= s part of mcp testing that was already planned. For the initial troublesho= oting we were using a test bed with a spare read head that allows us to con= firm that the problem is the head or markers vs the rest of the system. The cause was that the wiring at the MCP had been disturbed Wednesday morni= ng for photos needed to build a less temporary way of getting these signals= into the MCP. That was probably the first thing I would have checked if I= 'd have had to come in. Lucky it was ALT - we can usually run all night wi= thout accumulating big ALT errors, not so with AZ. 2. ALT fiduials did not read in the evening and start of the night. The last ALT crossing SET_FIDUCIAL_ERROR 1571833952 ALTITUDE 1 25 was reported at 12:32:32 UT. Some day time tests were done around afternoon at ALT=3D60, which means sometimes might happened with the read head between the morning and afternoon time. Interesting that the logs are updated with the UPDATE_ENCODER 1571897401 ALTITUDE 1 23142280 0 0 UPDATE_ENCODER 1571897401 ALTITUDE 2 23129476 0 0 records about the altitude. I know some or most of the night staff may know some of the following alrea= dy, but I'm going to spell it all out in the interest of plugging any holes= in people's knowledge. I certainly have those so I generally assume every= one does. Fiducial crossings from the mcpFiducial-mjd.dat log look like this: ALT_FIDUCIAL 1571833952 2 7531843 7529606 7531818 7529299 2= 8.918 2997 85900 -282 -624 There are several useful parts of this line - the time stamp is the second = column in this case is what I was looking for. Unix time is kept in second= s since Jan 1 1970, which is why that time stamp is an annoyingly huge numb= er. It's easy enough to find a unix time converter online which will provi= de you with the corresponding UTC time and date. I like this one: https:/= /www.epochconverter.com/ because it provides both the UT and your time zone= . So the last good read of an altitude fiducial was at 6:32:32 am on Wednesda= y - closing after a night's observing. The fact that there were no other l= ogged crossings for a single but not all axes throughout the sdss5 flexure = tests, afternoon checks, or initial observations last night indicates that = this is likely a hardware problem rather than a crashed program. This was = confirmed by logging in to the mcp and using the ./proc feature to look at = the running processes - tlatch is the one that handles fiducial crossings, = and once I was home around 9 pm last night I was able to confirm it was run= ning. The easy way to find the last crossing for an axis one is interested in is = to go to /data/logs/mcp on hub and type something like "cat mcpFiducials-[m= jd].dat | grep [axis abbreviation in caps]" In this case the first mjd of = 58780 turned up some lines with ALT but nothing with the "ALT_FIDUCIAL" tha= t I was looking for. Going back to 58779 found the last one listed above. While I don't mind doing the above steps myself when there is a problem, it= s useful to the nightstaff to be able to use these particular log files as = well. Just looking at the log for 58780 in this case tells you to suspect = that the software side is running fine. If no crossings for any axis were = reported, tlatch might have crashed. If all axes were reported throughout = the period where you know STUI is not reporting fiducials, you might suspec= t a software problem between two programs (though that too is unlikely if i= ts just one axis that is not reporting). Another case where this file is very useful is if you find you have an inex= plicably large error. The fourth and fifth columns are the expected encode= r values at the crossed fiducial (ID number is third column). The sixth an= d seventh columns are the actual encoder reads for the first and second enc= oders. Subtracting the sixth column from the fourth and the seventh from t= he fifth will give you the fiducial error reported in STUI. If the large e= rrors are about the same size for both encoders the problem could be real -= a slip, a misread, etc. If the large error is only on one axis, it's debr= is or running over a moth and safe to accept a fiducial and confirm the err= or is gone (and a really good idea to clean that drive surface so the probl= em doesn't repeat the next time you run over what's left of the moth. Moth= s are around 20,000 to 30,000 encoder counts thick on the first pass btw :D= ) --_000_BY5PR13MB3190E5DC28CAC7E86F39C0C9976A0BY5PR13MB3190namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable P {margin-top:0;margin-bo= ttom:0;} Hi John, Thanks for this very informative email! It occurred to me a few days ago that surely there is a tremendous amount o= f useful procedural, background and troubleshooting information contained i= n the 20+ years of infrastructure and 2.5m obslog, etc., mailing lists.= There must be an existing software solution for mining those emails for some subset that could form the core = of searchable documentation for the greater 2.5m system. Thanks, John. From: sloan-25mlog on behalf of John Downey Sent: Thursday, October 24, 2019 11:48 AM To: dmbiz@apo.nmsu.edu Cc: obs-report-25m@apo.nmsu.edu Subject: [sloan-25mlog 4724] Re: 2.5m obslog 20191023 (SJD 58780) Fiducial read problem is fixed. We will do test moves later this= morning as part of mcp testing that was already planned. For the ini= tial troubleshooting we were using a test bed with a spare read head that a= llows us to confirm that the problem is the head or markers vs the rest of the system. The cause was that the wiring at the MCP had been disturbed Wednesday = morning for photos needed to build a less temporary way of getting these si= gnals into the MCP. That was probably the first thing I would have ch= ecked if I'd have had to come in. Lucky it was ALT - we can usually run all night without accumulating big ALT err= ors, not so with AZ. 2. ALT fiduials did not read in the evening and start of the night. The last ALT crossing SET_FIDUCIAL_ERROR 1571833952 ALTITUDE 1 25 was reported at 12:32:32 UT. Some day time tests were done around afternoon at ALT=3D60, which means sometimes might happened with the read head between the morning and afternoon time. Interesting that the logs are updated with the UPDATE_ENCODER 1571897401 ALTITUDE 1 23142280 0 &nb= sp; 0 UPDATE_ENCODER 1571897401 ALTITUDE 2 23129476 0 &nb= sp; 0 records about the altitude. I know some or most of the night staff may know some of the following alrea= dy, but I'm going to spell it all out in the interest of plugging any holes= in people's knowledge. I certainly have those so I generally assume = everyone does. Fiducial crossings from the mcpFiducial-mjd.dat log look like this: ALT= _FIDUCIAL 1571833952 2 7531843 7529606&nbs= p; 7531818 7529299 28.918 2= 997 85900 -282 -624 There are several useful parts of this= line - the time stamp is the second column in this case is what I was look= ing for. Unix time is kept in seconds since Jan 1 1970, which is why = that time stamp is an annoyingly huge number. It's easy enough to find a unix time converter online which will pro= vide you with the corresponding UTC time and date. I like this one: &= nbsp;https://www.epo= chconverter.com/ because it provides both the UT and your time zone. So the last good read of an altitude f= iducial was at 6:32:32 am on Wednesday - closing after a night's observing.= The fact that there were no other logged crossings for a single but = not all axes throughout the sdss5 flexure tests, afternoon checks, or initial observations last night indicates that= this is likely a hardware problem rather than a crashed program. Thi= s was confirmed by logging in to the mcp and using the ./proc feature to lo= ok at the running processes - tlatch is the one that handles fiducial crossings, and once I was home around 9 p= m last night I was able to confirm it was running. The easy way to find the last crossing= for an axis one is interested in is to go to /data/logs/mcp on hub and typ= e something like "cat mcpFiducials-[mjd].dat | grep [axis abbreviation= in caps]" In this case the first mjd of 58780 turned up some lines with ALT but nothing with the "ALT_FIDUCIA= L" that I was looking for. Going back to 58779 found the last on= e listed above. While I don't mind doing the above ste= ps myself when there is a problem, its useful to the nightstaff to be able = to use these particular log files as well. Just looking at the log fo= r 58780 in this case tells you to suspect that the software side is running fine. If no crossings for any axis= were reported, tlatch might have crashed. If all axes were reported = throughout the period where you know STUI is not reporting fiducials, you m= ight suspect a software problem between two programs (though that too is unlikely if its just one axis that is not rep= orting). Another case where this file is very u= seful is if you find you have an inexplicably large error. The fourth= and fifth columns are the expected encoder values at the crossed fiducial = (ID number is third column). The sixth and seventh columns are the actual encoder reads for the first and second enco= ders. Subtracting the sixth column from the fourth and the seventh fr= om the fifth will give you the fiducial error reported in STUI. If th= e large errors are about the same size for both encoders the problem could be real - a slip, a misread, etc. If= the large error is only on one axis, it's debris or running over a moth an= d safe to accept a fiducial and confirm the error is gone (and a really goo= d idea to clean that drive surface so the problem doesn't repeat the next time you run over what's left of the m= oth. Moths are around 20,000 to 30,000 encoder counts thick on the fi= rst pass btw :D ) --_000_BY5PR13MB3190E5DC28CAC7E86F39C0C9976A0BY5PR13MB3190namp_--