Skip to content

feat(gain): add median ratio, concentration and above-ceiling to summary - #3511

Open
juxtapo9090 wants to merge 1 commit into
rtk-ai:developfrom
juxtapo9090:feat/gain-median-concentration
Open

feat(gain): add median ratio, concentration and above-ceiling to summary#3511
juxtapo9090 wants to merge 1 commit into
rtk-ai:developfrom
juxtapo9090:feat/gain-median-concentration

Conversation

@juxtapo9090

Copy link
Copy Markdown

Closes #3507.

rtk gain reports totals and averages only. Command output sizes are long-tailed, so a mean describes almost none of the population.

On my install — 10,376 commands, 90-day window — the headline reads 96.5% saved, while the median command is 0.97x and 53% come back byte-for-byte unchanged. Both are true; only the first was visible.

What it adds

Three lines in the summary block, using the existing print_kpi helper:

Median ratio:      0.97x (53% unchanged)
Concentration:     top 8 commands = 92.1% of total savings
Above ceiling:     396 / 10376 commands
  • Median ratio — median of output_tokens / input_tokens, skipping rows with zero input. Empty input reports 1.0 (unchanged) rather than 0.0, so "no data" never reads as "perfect compression".
  • Concentration — share of total savings held by the largest 8 commands, with the count actually used so it stays honest on small datasets. This exists because a single recursive grep can dominate the total: on my data one command is 69% of it.
  • Above ceiling — commands whose raw output exceeded CONTEXT_CEILING_TOKENS (7500, ~30k chars at ~4 chars/token), roughly where agent CLIs truncate. Makes the tail visible without changing anything that is stored or how savings are computed.

Scope

Additive only — 149 insertions, 0 deletions. No existing output line changes, no stored value changes, no new config key or CLI flag. GainSummary gains five fields, appended; nothing reordered.

Verification

  • cargo fmt --check clean
  • cargo clippy clean
  • cargo test — 2585 passed, 9 of them new

The new tests cover the empty set, a single value, even and odd counts for the median, zero savings, fewer commands than the top-N window, and the single-outlier shape the concentration metric exists to expose.

Numbers were cross-checked against an independent implementation reading the same database; median, unchanged percentage, concentration and above-ceiling count all agree.

One thing worth flagging

Two tests fail on develop before this branch — hooks::rewrite_cmd::tests::unattestable_passthrough::{test_plain_command_still_rewrites, test_fd_dup_redirect_still_rewrites}. I confirmed they fail identically with this change stashed, so they are pre-existing and unrelated, but CI will be red until they are addressed. Happy to open a separate issue if that is useful.

Note on process

The display code was drafted with an AI assistant and the statistics implementation, tests, and verification are mine; every line was reviewed and built locally before pushing. Flagging it since the project seems to work with agents openly.

rtk gain reports totals and averages only. Command output sizes are
long-tailed, so a mean describes almost none of the population: on a
10,376-command install the headline reads 96.5% saved while the median
command is 0.97x and 53% are returned byte-for-byte unchanged.

Adds three lines to the summary block:

  Median ratio:      0.97x (53% unchanged)
  Concentration:     top 8 commands = 92.1% of total savings
  Above ceiling:     396 / 10376 commands

Concentration exists because one recursive grep can dominate the total.
Above-ceiling counts commands whose raw output exceeded a constant that
approximates where agent CLIs truncate, so the tail is visible without
changing what is stored.

Existing output and stored values are untouched; the change is additive.

Closes rtk-ai#3507
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gain: add median and concentration to the summary — means alone hide what the tool is actually doing

2 participants