Skip to content

test(mcp): add concurrent write_note MCP integration tests - #1183

Open
FBISiri wants to merge 4 commits into
basicmachines-co:mainfrom
FBISiri:siri/concurrent-write-note-integration-tests
Open

test(mcp): add concurrent write_note MCP integration tests#1183
FBISiri wants to merge 4 commits into
basicmachines-co:mainfrom
FBISiri:siri/concurrent-write-note-integration-tests

Conversation

@FBISiri

@FBISiri FBISiri commented Aug 3, 2026

Copy link
Copy Markdown

Summary

Adds integration tests for concurrent write_note MCP tool calls. The project has concurrency controls (FileService semaphore, entity_service race condition handling) but no integration tests exercising them at the MCP tool layer.

Tests Added

Test Description Concurrency
test_concurrent_write_different_notes 10 notes across different directories 10 concurrent
test_concurrent_write_same_directory 12 notes in same dir, verify unique permalinks 12 concurrent
test_concurrent_write_then_search Write 8 notes + verify FTS index consistency 8 concurrent
test_concurrent_write_and_read Concurrent writes + reads for consistency Mixed
test_concurrent_write_high_volume 25-note stress test (@pytest.mark.slow) 25 concurrent

Details

  • Uses asyncio.gather for true concurrent execution
  • Follows existing test-int/mcp/ patterns and fixtures
  • All tests clean up created notes in teardown
  • Passes ruff check and ruff format

Motivation

Existing integration tests in test-int/mcp/ are sequential. The concurrency controls in FileService and entity_service deserve integration-level coverage to catch regressions that unit tests might miss.

@CLAassistant

CLAassistant commented Aug 3, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@FBISiri

FBISiri commented Aug 4, 2026

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

@FBISiri
FBISiri force-pushed the siri/concurrent-write-note-integration-tests branch from 514400e to 22447b4 Compare August 4, 2026 02:34
@FBISiri

FBISiri commented Aug 4, 2026

Copy link
Copy Markdown
Author

@CLAassistant check

1 similar comment
@FBISiri

FBISiri commented Aug 4, 2026

Copy link
Copy Markdown
Author

@CLAassistant check

@phernandez phernandez added the On Hold Don't review or merge. Work is pending label Aug 4, 2026

@phernandez phernandez left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @FBISiri for contributing this. MCP-level concurrency coverage is a worthwhile gap to close, and the proposed scenarios are clearly organized and easy to follow.

I checked the exact PR head 22447b4a6b974bee3ee7c1d7bdfebaf8844aae43 against the current repository test configuration. There are two substantive blockers plus one PR-metadata fix needed before we can merge this.

Reproduction

The normal repository invocation fails:

BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 \
  .venv/bin/python -m pytest -q \
  test-int/mcp/test_concurrent_write_integration.py --no-cov

.FFFF
4 failed, 1 passed

The failures are all:

RuntimeError: <asyncio.locks.Lock ...> is bound to a different event loop

Running the same exact file on a shared test loop succeeds:

BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 \
  .venv/bin/python -m pytest -q \
  test-int/mcp/test_concurrent_write_integration.py --no-cov \
  -o asyncio_default_test_loop_scope=session

5 passed

That isolates the first blocker to the interaction between the repository's function-scoped pytest loops and the cached local-ASGI preparation lock. Please make the file pass using the repository's normal test command; the inline comment has the two reasonable repair directions.

The second blocker is test-oracle quality. The current writes all use distinct file paths and distinct permalinks, so they exercise parallel disjoint writes but never enter the file-path/permalink conflict recovery that the PR says it protects. Please add at least one deterministic MCP-level collision case and assert both the API outcome and final persisted/indexed state. Using output_format="json" would make the action/permalink assertions more robust than parsing display text.

The static checks are otherwise clean:

  • ruff check: passed
  • ruff format --check: passed
  • ty check: passed

Finally, please rename the PR to an allowed semantic scope. integration is not in .github/workflows/pr-title.yml; test(mcp): add concurrent write_note integration coverage would fit the changed surface.

Thank you again for tackling this. Once the default test run is green and the suite actually exercises a collision/race recovery path, this will be useful coverage.

from fastmcp import Client


@pytest.mark.asyncio

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These function-scoped asyncio tests deterministically fail when the file runs under the repository's normal configuration. On this exact head, the first concurrent test passes and binds the cached local-ASGI preparation lock to its event loop; the next four tests receive new pytest loops and fail with RuntimeError: <asyncio.locks.Lock ...> is bound to a different event loop.

The same five tests pass with -o asyncio_default_test_loop_scope=session, which confirms the lifecycle cause. Please make the normal invocation pass. The smallest test-only option is to run this module on one explicit session-scoped loop. If Basic Memory is expected to reuse the global FastAPI app across multiple event loops, the stronger fix is to make the cached preparation lock loop-scoped or remove it after the last active client, plus retain a targeted regression for that lifecycle.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified fixed on the current head: the file now uses plain function-scoped @pytest.mark.asyncio with the per-function lock reset fixture, and all 8 tests pass locally under the repo's default pytest configuration (fork PRs don't run the suite in CI, so this was checked on a local checkout of the merged head).

},
)

results = await asyncio.gather(*(write_one(i) for i in range(note_count)))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This creates concurrent traffic, but it does not exercise the conflict/race recovery described in the PR. Every call has a distinct title, file path, and permalink, so EntityRepository.upsert_entity() never needs its IntegrityError file-path/permalink recovery. A regression that removed that recovery would still leave this test green.

Please add a deterministic collision case through the MCP layer—for example, simultaneous writes to the same directory/title with overwrite=False (assert exactly one creation and the expected conflicts), or distinct filenames that normalize to the same permalink (assert unique canonical suffix allocation). Then read the winning notes and search them to prove the file, database row, and index agree. Prefer output_format="json" so the test asserts structured action, permalink, and error fields.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified fixed: test_write_same_title_same_directory_collision, test_permalink_suffix_collision_recovery, and test_concurrent_same_title_collision now exercise real same-key contention (same title, same directory, permalink suffix recovery) rather than disjoint writes. This addresses the concern.

},
)

tasks = [write_extra(i) for i in range(6)] + [read_anchor() for _ in range(6)]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not establish read/write consistency: the anchor is unrelated to every write, it is never mutated, and there is no synchronization point proving any read overlaps a write's critical section. The test can pass if all writes finish before the reads run, and it would also pass if same-note overwrite atomicity were broken.

A meaningful oracle would coordinate an overwrite of the same note and assert that concurrent reads return either the complete old document or the complete new document—never partial content, mixed metadata/body, or another note. If that behavior is outside the intended contract, this case should be removed rather than presented as proof of concurrent read consistency.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially fixed: the reworked test now asserts a well-defined property — reads of an untouched, already-committed note return its original title and body while other writes run. That resolves the original 'anchor proves nothing' concern. However, the property still isn't exercised as written: the two sequential gather calls complete every write before any read starts (see the newer review thread at line 483). With that one-line interleaving fix, this thread is fully addressed.

@phernandez phernandez changed the title test(integration): add concurrent write_note MCP integration tests test: add concurrent write_note MCP integration tests Aug 5, 2026
@phernandez phernandez changed the title test: add concurrent write_note MCP integration tests core(test): add concurrent write_note MCP integration tests Aug 5, 2026
@FBISiri FBISiri changed the title core(test): add concurrent write_note MCP integration tests test(mcp): add concurrent write_note MCP integration tests Aug 5, 2026
FBISiri added a commit to FBISiri/basic-memory that referenced this pull request Aug 5, 2026
@FBISiri
FBISiri force-pushed the siri/concurrent-write-note-integration-tests branch from 3340e40 to 12ea3f7 Compare August 10, 2026 00:03
@FBISiri

FBISiri commented Aug 10, 2026

Copy link
Copy Markdown
Author

Hi @phernandez — thanks for the thorough review. I've pushed fixes for all three items:

  1. Event loop scoping: Removed loop_scope="session" markers; all tests now use plain @pytest.mark.asyncio with a per-function lock reset fixture, so they pass under bare pytest with the repo's default config.
  2. Test oracle quality: All write_note calls now use output_format="json" with structured dict assertions. Added test_permalink_suffix_collision_recovery for deterministic suffix verification and test_concurrent_same_title_collision for concurrent same-key race coverage.
  3. Read/write consistency: test_concurrent_write_and_read now writes an anchor note first, then asserts its content survives unchanged during concurrent writes via JSON-parsed reads.

Static checks (ruff check, ruff format) pass. Ready for re-review.

@phernandez

Copy link
Copy Markdown
Member

Thanks for the thorough rework, @FBISiri — the fix commit addresses all three review
threads, and since fork PRs don't run the full suite in this repo's CI, I verified locally
on your merged head:

  • test-int/mcp/test_concurrent_write_integration.py: 8 passed under the repo's
    default pytest configuration (no loop-scope markers needed) — thread 1 resolved.
  • The new test_write_same_title_same_directory_collision,
    test_permalink_suffix_collision_recovery, and test_concurrent_same_title_collision
    exercise real same-key contention rather than disjoint writes — thread 2 resolved.
  • The anchor test now asserts a well-defined property (reads of an untouched note stay
    uncorrupted during concurrent writes) — thread 3 resolved.

Two things before this can merge:

  1. DCO — none of the branch's commits carry Signed-off-by, so the DCO check fails.
    The sign-off is your attestation, so it has to come from you. This rebases your two
    commits with sign-offs and flattens the merge commit in one step:

    git remote add upstream https://github.com/basicmachines-co/basic-memory.git  # if not already
    git fetch upstream
    git rebase upstream/main --signoff
    git push --force-with-lease origin siri/concurrent-write-note-integration-tests
  2. One type diagnostic (would fail just typecheck, which fork CI didn't run):
    _parse(mcp_result) -> dict at line 56 needs type arguments — -> dict[str, Any]
    (with from typing import Any).

After the force-push I'll re-verify and this should be good to go.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 38ad644cd4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +419 to +422
search_results = await asyncio.gather(*(search_one(i) for i in range(note_count)))
for index, payload in enumerate(search_results):
titles = {result["title"] for result in payload["results"]}
assert f"Searchable Note {index}" in titles, (

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Assert search only after index convergence

When write_note returns before search-index publication, these immediate searches can legitimately omit the new notes because search rows are eventually consistent; the assertion therefore treats temporary projection lag as a lost update and will reject a valid asynchronous indexing implementation. Drive an existing index/reindex pass and assert convergence instead of requiring read-after-write search consistency.

AGENTS.md reference: AGENTS.md:L161-L166

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and this is worth fixing so the test doesn't couple CI to an implementation detail: search rows are eventually consistent by design in this codebase (AGENTS.md, Consistency Model), so asserting search hits immediately after write_note returns would reject a valid async-indexing implementation even though it happens to pass today. @FBISiri: rather than a strict read-after-write assertion, wrap the search in a small bounded poll (retry a few times over a couple of seconds until the expected notes appear, fail after the deadline). Same assertions, just convergence-based.

Comment on lines +482 to +483
write_payloads = await asyncio.gather(*write_tasks)
read_payloads = await asyncio.gather(*read_tasks)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Interleave anchor reads with the writes

When this test is relied on to catch failures that occur only while writes are in flight, the first gather completes every write before the second gather schedules any read; merely constructing the read coroutines does not start them. The test can therefore pass even if anchor reads are inconsistent during active writes, so schedule both groups together and then separate their results.

AGENTS.md reference: AGENTS.md:L136-L138

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed against the code — this is correct. await asyncio.gather(*write_tasks) at line 482 completes every write before line 483 schedules a single read, so the docstring's 'reads while writes are in flight' never happens and the test can't catch what it claims to. @FBISiri: one-line fix while you're rebasing for DCO — results = await asyncio.gather(*write_tasks, *read_tasks) then split with write_payloads, read_payloads = results[:6], results[6:].

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4e71bfa384

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +271 to +272
assert read_payload["title"] == "Race Note", read_payload
assert read_payload["permalink"].endswith(base), read_payload

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate the canonical body after same-title races

If overlapping same-path writes preserve the entity metadata but truncate or interleave the canonical Markdown, this test still passes because the read-back checks only the title and permalink. Since each writer supplies a known body and canonical note bytes are the protected state, assert that the resulting content matches one complete writer body rather than merely that the row remains resolvable.

AGENTS.md reference: AGENTS.md:L156-L160

Useful? React with 👍 / 👎.


read_results = await asyncio.gather(*(read_one(i) for i in range(note_count)))
for index, payload in enumerate(read_results):
assert f"Volume body {index}" in payload["content"], (

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Match the complete high-volume body

When a concurrency regression assigns note 10–19's body to note 1, this assertion still succeeds because, for example, "Volume body 1" in "Volume body 10." is true. Include the terminating period or otherwise compare an unambiguous complete body so the stress test detects cross-wired content for single-digit indices.

AGENTS.md reference: AGENTS.md:L136-L138

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

On Hold Don't review or merge. Work is pending

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants