Date: Mon, 9 Jan 2023 15:30:18 -0700
Subject: Re: [sloan-25mlog 6476] 2.5m obslog 20230108 (SJD 59953)
From: "Daniel Oravetz"
Hi José,
I have replied inline below.
> Hi Dan,
>
> (1) Since the recent version changes, the goto-field reconfiguration stage
>> yields a bunch of error messages of the form:
>>
>> 10:12:19 .jaeger 0 jaeger f text="[GET_ACTUAL_POSITION, [and then a
>> bunch
>> of positioners IDs follows]
>>
>> It doesn't actually affect the goto-field or abort any of the stages,
>> but
>> it feels as though with the asynchronous changes some tasks are perhaps
>> expecting some kind of reply and not getting it when it used to expect
>> it?
>>
>
> Those errors are not exactly new, but it's interesting that they're
> happening more now? How often do they happen? I did a fair amount of
> goto-fields with the new code and didn't see one. When do they happen? Is
> it while the FPS is moving or at the end? During the move the code itself
> has not changed, so it should not be related to the asynchronous steps,
> and
> once the move has finished, which is when the asynchronous steps are
> launched, jaeger should not be asking for updated positions anymore ...
There were five instances last night (and as you said there are five
hundred each time, so it feels worse than it probably is):
2023-01-09 10:12:19,722Z  My guess is that those errors happen when a command times out, which
> should
> not happen often, but it's not impossible, especially if there's some
> network issue that makes the network unavailable for a short period of
> time. In the case of GET_ACTUAL_POSITION they are probably innocuous,
> since
> the next status attempt will update them. I'll see if I can catch those
> and
> avoid outputting 500 messages.
>
> (2) Had a couple of these errors tonight:
>>
>> jaeger f error="Trajectory failed with error: Something went wrong
>> during
>> the trajectory: Some positioners did not reach their destinations.
>> Trajectory failed 47.82 seconds after start"
>>
>> Each time, the next goto-field output during the reconfiguration
>> re-attempt would indicate the positioners actually did make it home
>> during
>> the 'reverse'. These just seemed to be timeouts and didn't cause any
>> real
>> problems.
>>
>
> That could be a race condition or something related to the previous
> problem, basically jaeger not getting updated positions and not knowing
> that the positions have reached their destinations (although they have). I
> think I removed a couple seconds at the end of a trajectory that I used to
> leave for these cases. I'll see if we should add them back, or do
> something
> about it.
It happened twice last night:
2023-01-08T07:12:00.log:2023-01-09 06:53:07,503Z < 1 609 f
error="Trajectory failed with error: Something went wrong during the
trajectory: Some positioners did not reach their destinations. Trajectory
failed 43.66 seconds after start"
2023-01-09T07:12:00.log:2023-01-09 10:12:21,723Z < 1 682 f
error="Trajectory failed with error: Something went wrong during the
trajectory: Some positioners did not reach their destinations. Trajectory
failed 47.82 seconds after start"