Expected response code to be 200, but was 503:

[njalan]

Trino version is 397.
Sone node face below error message:
ERROR SplitRunner-22-76 io.trino.execution.executor.TaskExecutor Error processing Split 20240228_221856_02171_wf5x6.9.2.0-0 (start = 3.3397277544145065E10, wall = 62562 ms, cpu = 0 ms, wait = 9 ms, calls = 3): PAGE_TRANSPORT_ERROR: Error fetching http://xxxxx:8084/v1/task/20240228_221856_02171_wf5x6.10.0.0/results/2/0: Expected response code to be 200, but was 503:

<title>Error 503 Service Unavailable</title>

HTTP ERROR 503 Service Unavailable

URI:	/v1/task/20240228_221856_02171_wf5x6.10.0.0/results/2/0
STATUS:	503
MESSAGE:	Service Unavailable
SERVLET:	-

[hashhar]

this usually suggests one of the workers did not respond to coordinator requests in timely manner. Check the worker logs and metrics for the IP address for which the error was logged.

It's usually long GC pauses or worker being restarted due to OOM or CPU starvation.