feat(gain): add median ratio, concentration and above-ceiling to summary - #3511
Open
juxtapo9090 wants to merge 1 commit into
Open
feat(gain): add median ratio, concentration and above-ceiling to summary#3511juxtapo9090 wants to merge 1 commit into
juxtapo9090 wants to merge 1 commit into
Conversation
rtk gain reports totals and averages only. Command output sizes are long-tailed, so a mean describes almost none of the population: on a 10,376-command install the headline reads 96.5% saved while the median command is 0.97x and 53% are returned byte-for-byte unchanged. Adds three lines to the summary block: Median ratio: 0.97x (53% unchanged) Concentration: top 8 commands = 92.1% of total savings Above ceiling: 396 / 10376 commands Concentration exists because one recursive grep can dominate the total. Above-ceiling counts commands whose raw output exceeded a constant that approximates where agent CLIs truncate, so the tail is visible without changing what is stored. Existing output and stored values are untouched; the change is additive. Closes rtk-ai#3507
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3507.
rtk gainreports totals and averages only. Command output sizes are long-tailed, so a mean describes almost none of the population.On my install — 10,376 commands, 90-day window — the headline reads 96.5% saved, while the median command is 0.97x and 53% come back byte-for-byte unchanged. Both are true; only the first was visible.
What it adds
Three lines in the summary block, using the existing
print_kpihelper:output_tokens / input_tokens, skipping rows with zero input. Empty input reports1.0(unchanged) rather than0.0, so "no data" never reads as "perfect compression".CONTEXT_CEILING_TOKENS(7500, ~30k chars at ~4 chars/token), roughly where agent CLIs truncate. Makes the tail visible without changing anything that is stored or how savings are computed.Scope
Additive only — 149 insertions, 0 deletions. No existing output line changes, no stored value changes, no new config key or CLI flag.
GainSummarygains five fields, appended; nothing reordered.Verification
cargo fmt --checkcleancargo clippycleancargo test— 2585 passed, 9 of them newThe new tests cover the empty set, a single value, even and odd counts for the median, zero savings, fewer commands than the top-N window, and the single-outlier shape the concentration metric exists to expose.
Numbers were cross-checked against an independent implementation reading the same database; median, unchanged percentage, concentration and above-ceiling count all agree.
One thing worth flagging
Two tests fail on
developbefore this branch —hooks::rewrite_cmd::tests::unattestable_passthrough::{test_plain_command_still_rewrites, test_fd_dup_redirect_still_rewrites}. I confirmed they fail identically with this change stashed, so they are pre-existing and unrelated, but CI will be red until they are addressed. Happy to open a separate issue if that is useful.Note on process
The display code was drafted with an AI assistant and the statistics implementation, tests, and verification are mine; every line was reviewed and built locally before pushing. Flagging it since the project seems to work with agents openly.