FIX: log and retry supervisor connection during worker startup - #5278
Open
OliverBryant wants to merge 1 commit into
Open
FIX: log and retry supervisor connection during worker startup#5278OliverBryant wants to merge 1 commit into
OliverBryant wants to merge 1 commit into
Conversation
A starting worker's first network operation is a REST call to the supervisor made before any log line is emitted, with no timeout and no retry. When the supervisor endpoint is unreachable the worker dies (or hangs) showing nothing but third-party import noise, and with Docker's restart policy this becomes a silent crash loop that is very hard to diagnose. - Log the supervisor endpoint right after logging is configured so a worker always produces a first-party startup line. - Retry the supervisor address resolution with explicit warnings (attempt count, endpoint, error) for up to 5 minutes, so workers survive supervisor-not-yet-up races in docker compose deployments and join automatically once the supervisor is reachable. - Bound the /v1/cluster/auth and /v1/address metadata requests with a (10, 30) timeout so an unresponsive endpoint cannot hang the client construction indefinitely.
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
qinxuye
reviewed
Aug 5, 2026
| max_attempts = _SUPERVISOR_CONNECT_MAX_ATTEMPTS | ||
| if retry_interval is None: | ||
| retry_interval = _SUPERVISOR_CONNECT_RETRY_INTERVAL | ||
| for attempt in range(1, max_attempts + 1): |
Contributor
There was a problem hiding this comment.
[P1] Enforce the retry budget by elapsed time
The advertised five-minute limit currently counts only the retry sleeps. Each attempt can additionally spend up to 30 seconds in the metadata request, so an endpoint that accepts TCP but never responds can keep the worker here for roughly 35 minutes instead of five. This is one of the failure modes this change is intended to bound. Please use a monotonic wall-clock deadline (or otherwise include request duration in the total budget) and add a regression test with slow failed attempts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #5268.
A starting worker's first network operation is a REST call to the supervisor (
RESTfulClient.__init__->/v1/cluster/auth, then/v1/address), performed before any first-party log line is emitted, with no timeout and no retry. When the supervisor endpoint is unreachable, the worker process dies (or hangs) showing nothing but third-party import noise, e.g.:With
restart: alwaysin docker compose this becomes a silent crash loop, which is exactly the log signature reported in #5268 (repeated torchao import noise, no worker logs). I reproduced both directions on thev3.1.0-cpuimage:Changes
xinference/deploy/cmdline.py: log the supervisor endpoint immediately after logging is configured, and resolve the supervisor internal address through a retry loop (5s interval, up to 5 minutes) with explicit warnings carrying the endpoint, the error, and the attempt count. Workers now survive the common "supervisor not up yet" race and join automatically once it becomes reachable; after the retry budget is exhausted the original exception is re-raised with an actionable error message.xinference/client/restful/restful_client.py: bound the/v1/cluster/authand/v1/addressmetadata GETs with a(10, 30)connect/read timeout so an endpoint that accepts TCP but never responds cannot hang client construction indefinitely. Inference requests are deliberately left untouched.xinference/deploy/test/test_cmdline.py: tests covering retry-then-success and retry-exhaustion.Verification
pytest xinference/deploy/test/test_cmdline.py -k worker_command(3 passed).pre-commit runon the modified files (all hooks passed).v3.1.0-cpuDocker image with the patched files mounted in:Starting Xinference worker. Supervisor endpoint: ...followed bySupervisor at ... is not ready (ConnectionError: ...), retrying in 5 seconds (attempt 1/60)instead of dying silently;Xinference worker ... startedwithout a container restart.Not run: full CI-style test suite (model tests require GPU/network); left to CI.