fix(firehose): add configurable idle timeout to the block stream receive loop - #6710

Problem

A subgraph fed by a firehose block stream can stop indexing indefinitely and silently if the firehose upstream keeps the HTTP/2 stream open but stops sending frames (no message, no error, no end-of-stream). The subgraph stays health: healthy, is not paused, sets no fatalError, emits zero logs, and its latestBlock is frozen until a manual restart/reassignment opens a fresh stream.

Root cause

graph/src/blockchain/firehose_block_stream.rs wraps only the initial stream_blocks establishment in tokio::time::timeout(120s, ...), but the receive loop (for await response in stream) has no per-message/idle timeout. last_response_time is only fed to a metric and never used to trip a timeout. If the upstream holds the stream open and sends nothing, for await never yields, and the hang propagates up the whole consumer chain (BufferedBlockStream, the runner's block_stream.next().await) with no error and no log.

HTTP/2 keepalive pings are intentionally disabled in graph/src/firehose/endpoints.rs (the code comment explains cloud load balancers drop connections that frequently send pings), and tcp_keepalive(15s) only detects a fully dead peer, not a half-open / app-hung upstream — the common case when the upstream is mid-restart behind a proxy/LB.

Fix

Wrap each wait for the next stream message in a configurable idle timeout. When the timeout elapses, the stream is dropped and re-established with backoff (the reconnect path already exists), instead of hanging forever.

• New next_with_idle_timeout(stream, idle) helper: bounds each stream.next() with tokio::time::timeout; returns Err(()) when the idle deadline elapses before the next message.
• New env var GRAPH_FIREHOSE_STREAM_IDLE_TIMEOUT_SECS: when set to a positive number of seconds, the receive loop drops the stream and reconnects if no message arrives within that window. Disabled by default (missing / 0 / unparseable) to preserve current behavior.
• On idle timeout the loop logs an explicit error and takes the existing reconnect path with backoff.

Test

Adds graph/tests/firehose_idle_timeout_tests.rs:

• idle_timeout_from_env_parsing — missing / 0 / unparseable env value disables the timeout; a positive value enables it.
• stream_idle_timeout_returns_item_within_deadline — an item that arrives within the deadline is delivered.
• stream_idle_timeout_breaks_stalled_stream — a stream that sends nothing within the idle timeout returns Err(()) instead of hanging (the issue's exact failure mode).
• stream_idle_timeout_returns_none_when_stream_ends — natural end-of-stream is still observed.
• stream_idle_timeout_disabled_waits_for_item — with no idle timeout the wait is unconditional (backward compatible).

The stalled-stream test does not build against the pre-fix code (the helper functions do not exist) and passes on the fixed code.

Notes

This is the topology-independent primary fix recommended in the issue (option 1). Re-enabling HTTP/2 keepalive (option 2) is left as an operator opt-in and out of scope here.

Fixes #6689