From: "Wilson, John C (jcw6z)" Subject: Re: [sloan-25mlog 5333] Re: 2.5m obslog 20210116 (SJD 59231) Date: Tue, 19 Jan 2021 20:52:56 +0000 --_000_BY5PR13MB3425815F6B50844B54159B6E97A39BY5PR13MB3425namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi Jose, Matt looked but didn't see anything odd in the ICS logs regarding recent sl= owdowns at the start of apogee exposures. This jibes with more recent info= rmation that suggests the slowdown is isolated to only some of the computer= s running stui. Thanks, John. ---------------------------- John Wilson (he/him) University of Virginia Astronomy Department jcw6z@virginia.edu Office (434) 924-4907 Mobile (434) 270-5847 ________________________________ From: sloan-25mlog on behalf of Wilson, Joh= n C (jcw6z) Sent: Sunday, January 17, 2021 4:11 PM To: Jos=E9 S=E1nchez-Gallego ; Kaike Pan Cc: obs-report-25m@apo.nmsu.edu ; shane thomas= Subject: [sloan-25mlog 5333] Re: 2.5m obslog 20210116 (SJD 59231) Hi Jose, I'll ask Matt to take a look at the ICS logs. Thanks, John. ---------------------------- John Wilson (he/him) University of Virginia Astronomy Department jcw6z@virginia.edu Office (434) 924-4907 Mobile (434) 270-5847 ________________________________ From: sloan-25mlog on behalf of Jos=E9 S=E1= nchez-Gallego Sent: Sunday, January 17, 2021 4:01 PM To: Kaike Pan Cc: obs-report-25m@apo.nmsu.edu ; shane thomas= Subject: [sloan-25mlog 5332] Re: 2.5m obslog 20210116 (SJD 59231) Hi Kaike, This still look to me more likely to be a communication or resource issue, = not necessarily with the computer running STUI, but with sdss-hub or other = VM that's running an actor. The fact that it happens at the beginning of an= APOGEE or BOSS exposure could be expected, since when you start a new sequ= ence in STUI many things happen at once. One easy way to confirm this is not a problem with the removal of the dithe= ring would be, if the problem returns, to take a sequence of exposures with= out using SOP, just using the APOGEE and BOSS widgets. If the delay still h= appens then, it cannot be an issue with the change to dithering. I remember there has been some issues with the APOGEE detector. Could these= be related, in the sense that the detector or the actor are not behaving a= s usual at the beginning of the exposures? It may be useful to look at the = ICS logs and see if there's something wrong there. Thanks, Jos=E9 On Jan 17, 2021 at 12:46:46, Kaike Pan > wrote: Hi Jose, The STUI freezes at all computers when it happens, and mostly at the beginn= ing of new APOGEE exposures. I did check disk space and memory usage on ob1= during STUI freezing, they are fine. CPU load is very low, STUI used less = than 2 G memory resource, thel memory used is less than 7G (the computer ha= s 16 G memory) on ob1 at that time. I also tested ob1 and sdss-hub with oth= er commands, ds9 and list_ap, these commands return normal at that time. Th= is indicates 1) it is not a computer resource issue, 2) communication betwe= en ob1 and sdss-hub is fine (since I tested sdss-hub from obs1). Shane, do you have a way to monitor connection speeds among sdss related co= mputers? Thanks, -Kaike On Jan 17, 2021, at 1:30 PM, Jos=E9 S=E1nchez-Gallego > wrote: Hi Kaike, I cannot see how that would be the case. We didn't add anything, just remov= ed a command, and I cannot see how that would affect STUI. It would be usef= ul to check, when this is happening, if the load of any of the VMs (hub, ap= ogee-ics, boss-ics) are higher than usual or if the communication with thos= e machines seems slower than usual. Thanks, Jos=E9 On Jan 17, 2021 at 05:55:43, Kaike Pan > wrote: Hi Jose, I also noticed that STUI froze at the beginning of Apogee exposures two ni= ghts ago. We restarted tron yesterday, it didn=92t fix the issue. Is it possible that= the issue somehow relates to the change removing apogee dither reposition? The issue appear a= fter the change in SOP. Thanks, -Kaike - Late in the night, STUI started to freeze, usually in the first 10 reads = of a new APOGEE exposure - We had some difficulty attaching cart 3 to the telescope because the latc= hes wouldn't engage. It happened once during the night as well. --_000_BY5PR13MB3425815F6B50844B54159B6E97A39BY5PR13MB3425namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable P {margin-top:0;margin-bo= ttom:0;} Hi Jose, Matt looked but didn't see anything odd in the ICS logs regarding recent sl= owdowns at the start of apogee exposures. This jibes with more recent= information that suggests the slowdown is isolated to only some of the com= puters running stui. Thanks, John. ---------------------------- John Wilson (he/him) University of Virginia Astronomy Department jcw6z@virginia.edu Office (434) 924-4907 Mobile (434) 270-5847 From: sloan-25mlog on behalf of Wilson, John C (jcw6z) Sent: Sunday, January 17, 2021 4:11 PM To: Jos=E9 S=E1nchez-Gallego ; Kaike Pan <= kpan@apo.nmsu.edu> Cc: obs-report-25m@apo.nmsu.edu ;= shane thomas Subject: [sloan-25mlog 5333] Re: 2.5m obslog 20210116 (SJD 59231) Hi Jose, I'll ask Matt to take a look at the ICS logs. Thanks, John. ---------------------------- John Wilson (he/him) University of Virginia Astronomy Department jcw6z@virginia.edu Office (434) 924-4907 Mobile (434) 270-5847 From: sloan-25mlog on behalf of Jos=E9 S=E1nchez-Gallego Sent: Sunday, January 17, 2021 4:01 PM To: Kaike Pan Cc: obs-report-25m@apo.nmsu.edu ;= shane thomas Subject: [sloan-25mlog 5332] Re: 2.5m obslog 20210116 (SJD 59231) Hi Kaike, This still look to me more likely to be a communication or resource is= sue, not necessarily with the computer running STUI, but with sdss-hub or o= ther VM that's running an actor. The fact that it happens at the beginning = of an APOGEE or BOSS exposure could be expected, since when you start a new sequence in STUI many things happe= n at once. One easy way to confirm this is not a problem with the removal of the = dithering would be, if the problem returns, to take a sequence of exposures= without using SOP, just using the APOGEE and BOSS widgets. If the delay st= ill happens then, it cannot be an issue with the change to dithering. I remember there has been some issues with the APOGEE detector. Could = these be related, in the sense that the detector or the actor are not behav= ing as usual at the beginning of the exposures? It may be useful to look at= the ICS logs and see if there's something wrong there. Thanks, Jos=E9 On Jan 17, 2021 at 12:= 46:46, Kaike Pan wrote: Hi Jose, The STUI freezes at all computers when it happens, and most= ly at the beginning of new APOGEE exposures. I did check disk space and mem= ory usage on ob1 during STUI freezing, they are fine. CPU load is very low,= STUI used less than 2 G memory resource, thel memory used is less than 7G (the computer has 16 G memory) on ob1 at = that time. I also tested ob1 and sdss-hub with other commands, ds9 and list= _ap, these commands return normal at that time. This indicates 1) it is not= a computer resource issue, 2) communication between ob1 and sdss-hub is fine (since I tested sdss-hub from obs1). Shane, do you have a way to monitor connection speeds among= sdss related computers? Thanks, -Kaike On Jan 17, 2021, at 1:30 PM, Jos=E9 S=E1nchez-Gallego wrote: Hi Kaike, I cannot see how that would be the case. We didn't add anyt= hing, just removed a command, and I cannot see how that would affect STUI. = It would be useful to check, when this is happening, if the load of any of = the VMs (hub, apogee-ics, boss-ics) are higher than usual or if the communication with those machines seems sl= ower than usual. Thanks, Jos=E9 On Jan 17, 2021 at 05:= 55:43, Kaike Pan wrote: Hi Jose, I also noticed that STUI froze at the beginning of Apogee exposures t= wo nights ago. We restarted tron yesterday, it didn=92t fix the issue. Is it possible that= the issue somehow relates to the change removing apogee dither reposition? The issue appear a= fter the change in SOP. Thanks, -Kaike - Late in the night, STUI s= tarted to freeze, usually in the first 10 reads of a new APOGEE exposure - We had some difficulty at= taching cart 3 to the telescope because the latches wouldn't engage. It happene= d once during the night as well. --_000_BY5PR13MB3425815F6B50844B54159B6E97A39BY5PR13MB3425namp_--