Buffer table silently discards an acknowledged INSERT when the destination schema drifts (DROP / re-CREATE) before the deferred flush #114330

Company or project name

ClickHouse QA (durability testing)

Describe what's wrong

An INSERT into a Buffer table is acknowledged as soon as the rows land in the in-memory buffer; the write to the destination table happens later (background threshold flush, or OPTIMIZE). If the destination's schema diverges in that window, the flush silently discards the acknowledged rows and reports success — the client that received the INSERT acknowledgement never sees any error.

Two variants lose whole acked rows during normal, server-healthy operation:

• the destination is dropped and re-created with a column set that has no name in common with the buffer → the whole buffered block is discarded;
• the destination is dropped (a migration step) before the flush fires → the whole buffered block is discarded.

StorageBuffer::writeBlockToDestination logs these two cases at <Error> level — i.e. the code treats them as error conditions — but then returns normally, so flushBuffer counts the flush as successful and clears the buffer (src/Storages/StorageBuffer.cpp:1085-1089 for the missing-destination case, :1120-1124 for the no-common-columns case). Only a thrown exception restores the block to the buffer, so the rows are gone from everywhere with only a server-log line.

A third, milder variant is documented (a dropped destination column silently drops that column's acknowledged values while the rows survive — StorageBuffer.cpp:1103-1127; documented at docs/en/engines/table-engines/special/buffer.md), so this report is about the two whole-block losses, which the docs do not cover.

This is the same durability class as #112394 (Buffer losing buffered rows on graceful shutdown), but a distinct mechanism: here the server stays up and healthy, and the trigger is an ordinary destination schema change (DROP/re-CREATE/ALTER) between the acknowledgement and the deferred flush.

Does it reproduce on the most recent release?

Reproduces on 26.8.1.1. StorageBuffer::writeBlockToDestination carries the same discard-with-log logic on current master.

How to reproduce

clickhouse-server 26.8.1.1, single node, default settings.

CREATE TABLE dst (k UInt64, v String) ENGINE = MergeTree ORDER BY k;

-- Buffer thresholds set high so no auto-flush fires before the drift; a
-- production Buffer with ordinary thresholds hits the same discard on its
-- background flush.
CREATE TABLE buf AS dst ENGINE = Buffer(currentDatabase(), 'dst', 1,
    3600, 7200, 1000000, 2000000, 100000000, 200000000);

INSERT INTO buf SELECT number, concat('payload_', toString(number)) FROM numbers(50);
-- The INSERT is acknowledged. The rows are visible through the buffer:
SELECT count() FROM buf;   -- 50
SELECT count() FROM dst;   -- 0 (not flushed yet)

-- An ordinary destination migration: drop and re-create with a different schema.
DROP TABLE dst;
CREATE TABLE dst (a UInt64, b String) ENGINE = MergeTree ORDER BY a;

OPTIMIZE TABLE buf;   -- flush; returns success
SELECT count() FROM dst;   -- 0  -> the 50 acknowledged rows are gone

The whole-block loss also reproduces when the destination is simply dropped and not re-created before the flush.

Expected behavior

An acknowledged INSERT should not be silently lost while the server is healthy. At minimum the flush should not report success when it discards an acknowledged block: the discard is already recognized as an error internally (it is logged at <Error>), so it should surface to a place the client or an operator can observe (an exception that keeps the block buffered and retries, or a failure counter / system.errors entry), rather than clearing the buffer and returning success.

Error message and/or stacktrace

Server log only; nothing is returned to the client. From a run of the reproduction:

<Error> StorageBuffer (default.buf): Destination table default.dst have no common columns with block in buffer. Block of data is discarded.
<Error> StorageBuffer (default.buf): Destination table default.dst doesn't exist. Block of data is discarded.

Additional context

Found by the ClickFawkes durability-testing framework (round 90, ack_context_drift lens), rig --mode buffer-schema-drift, 5/5 reproduction with a non-vacuity control (no drift → all 50 rows flush) and the acknowledged rows proven present through the buffer before the drift. Code references are against master at aa3ccbf5e7df (behavior identical in clickhouse-private at 7d45de34d903).