SerpApi.Org
A proxy tool called rtk — widely shared among Claude Code developers as a way to cut token costs by 60 to 90 percent — turned out to raise session costs by a median of 7.6 percent in a controlled benchmark published by JetBrains on July 20, 2026. The finding is the second in a series of independent evaluations from JetBrains researcher Denis Shiryaev that has now systematically dismantled two of the most prominent cost-saving claims in the AI coding tools ecosystem.
The number that makes the result particularly striking is not the cost increase itself — it is what rtk's own analytics reported at the same time. Across the same set of trials where the measured bill went up, rtk's built-in rtk gain command logged 96.2 million tokens saved. That gap — between a self-reported saving of tens of millions of tokens and a measured cost that rose — points to the central problem the benchmark exposed: the tool measures against a counterfactual that does not exist.
What rtk Does
rtk, short for "Rust Token Killer," is a lightweight CLI proxy that intercepts shell commands issued by Claude Code and returns compressed summaries to the model instead of raw output. The premise is technically sound: when a developer's coding agent runs git status, rtk intercepts it, runs the real command, and hands the model a compact representation — something like * master / M a.txt / ?? b.txt — in place of eleven lines of porcelain output. For Python test runners, it replaces 19-line pytest output with a one-line summary naming the failing test. A Claude Code PreToolUse hook handles the interception transparently, so the model has no awareness of the compression layer.
The tool's marketplace listing claims it reduces Claude Code token consumption by 60 to 90 percent through intelligent compression of Git status, Docker logs, test runners, and file operations. The project's own benchmark documentation describes a 30-minute session where 118,000 tokens of command output became 24,000 tokens — a figure that contributed to the tool's rapid spread in Claude Code communities throughout 2026.
How JetBrains Structured the Test
Denis Shiryaev used JetBrains' SkillsBench suite — an open benchmark for evaluating AI coding agents on realistic workloads — and ran four paired arms: a 10-task smoke test, the same 10 tasks at three repetitions, the full 86-task set at low reasoning effort, and the full 86-task set at high reasoning effort. Both arms pinned Claude Code 2.1.201 and claude-sonnet-5. Total experiment: 425 billed trials at approximately USD 320 in compute.
A key feature of the design was instrumented adoption tracking. Every trial with rtk active persisted the hook's audit log and its analytics database, allowing the team to distinguish "rtk saved nothing" from "rtk never ran" — two very different explanations. Across runs, the hook rewrote between 33 and 50 percent of Bash calls it saw.
Before committing any compute, Shiryaev replayed 83 existing baseline transcripts and asked how much of a real agent session rtk could have touched. The answer was structural: Claude Code reads files with its built-in Read and Grep tools, which bypass the Bash hook entirely; roughly half of what agents run in a shell are commands rtk does not cover; and about a sixth of Bash calls use pipes, heredocs, or substitutions that rtk deliberately refuses to rewrite. What remained — approximately one-third of Bash calls — carried just under 20 percent of tool-result characters. And tool results themselves are only a fraction of what a session bills as input, because the same context gets re-read on every agent turn at cached rates.
That analysis put the theoretical maximum savings at roughly 3 percent of input tokens — a number computed from replaying prior transcripts, at zero additional cost, before a single paid trial ran. It proved to be an accurate prediction.
Costs Went Up, Not Down
Across 80 clean paired task comparisons at low reasoning effort, the arm running rtk came out a median 7.6 percent more expensive per task (p=0.004), completed 13.8 percent more turns (p=0.03), and generated 14.3 percent more cache reads (p=0.008). The one metric where rtk had to deliver its entire benefit — uncached "new input" tokens, the only token class the hook actually compresses — moved 3.2 percent, a statistically flat null (p=0.23).
The cost increase had a traceable, if thin, cause. The more Bash calls the hook rewrote, the larger the penalty on those tasks. Transcript analysis traced it to one genuinely broken rewrite — compound find predicates got turned into usage errors, triggering retries — along with a handful of compression-induced re-reads and ordinary variance on extreme pairs. The paper describes it as a "thin, systematic tax rather than a dramatic failure."
At high reasoning effort, the penalty did not replicate. Median paired delta was +0.1 percent (p=0.99), with turn count flat (p=0.74). But at no point in either condition did rtk produce a measurable saving. The tool was cost-neutral at high effort and cost-additive at low effort.
Quality Was Not the Issue
The benchmark tracked task quality separately from cost. Across both effort levels, the arms were statistically indistinguishable on outcomes. At low effort, scores landed at 5 better, 4 worse, and 71 tied; at high effort, 5 better, 4 worse, and 62 tied. Sign test p-values were 1.0 in both conditions.
The failure modes documented in rtk's own issue tracker — over-filtered test output, masked exit codes, compressed text fed into pipes — barely appeared in practice. A forensic pass over the most divergent transcript pairs found exactly one broken rewrite and one instance of the agent deliberately bypassing the hook across roughly 150 Bash calls. The filters the tool applies are, as Shiryaev's report puts it, "real and often elegant." The problem is not that rtk breaks things. It compresses the fraction of the session that does not drive most of the cost.
The Scoreboard Problem
Perhaps the most instructive finding concerns rtk gain — the tool's built-in analytics command, which logs how many tokens it estimates it has saved for each operation.
Across the low-effort full run, rtk gain reported 96.2 million tokens saved, representing 99.8 percent of everything it touched — while the measured bill for the same trials went up. Three mechanisms explain the divergence.
First, rtk counts the full raw output as its counterfactual. One cat of a 1.2 MB CSV file logged 320,000 tokens "saved" — but Claude Code truncates tool results before they reach anywhere near that threshold, so the agent would have received a few thousand tokens in either case. The full run logged 190 such large reads at an average of approximately 506,000 "saved" tokens each. Second, rtk estimates token counts as characters divided by four at the moment of execution, while most of a session's input cost arrives as cached re-reads billed at one-tenth the price of fresh tokens. Third, the hook never sees most of the session's context at all.
The result is a scoreboard that grades its own homework against a counterfactual that the actual billing system never applies.
A Pattern Across the Series
JetBrains has now published two consecutive evaluations of high-profile token-saving tools that delivered a fraction of their claimed benefit — or worse — under controlled conditions. The Caveman skill promised 65 percent output-token savings and delivered 8.5 percent on real agent work. The rtk proxy promised 60 to 90 percent and added cost.
Both studies surface the same methodological trap: a small initial single-trial sample produced results dramatic enough to appear significant, which then dissolved when sample size grew. In the first smoke test for rtk, the with-rtk arm looked 35 percent more expensive — alarming, until the team noted that identical attempts at the same task in the same arm differed by 22 percent in cost anyway. At three repetitions, the scare evaporated into noise. The lesson the series has now demonstrated twice is that k=1 evaluations of agentic tools are unreliable as a category.
Both tools also share a structural counterfactual problem: marketing benchmarks measure what the tool touches, not what actually drives costs across a complete session. Chat-style benchmarks compress a portion of the output stream that simply does not dominate agentic billing.
For developers evaluating any tool in this space, Shiryaev's closing observation from the rtk study states the principle plainly: a tool's self-reported savings are a claim about its counterfactual, not about your bill. Measuring the paired billed cost — not the tool's own diff — is the only evaluation that answers the question developers actually need answered. JetBrains has invited the community to nominate the next tool for the same benchmark ladder.
Frequently Asked Questions
Does installing rtk actually reduce what I pay for Claude Code?
Based on JetBrains' controlled benchmark across 425 billed trials, it depends on reasoning effort setting — and neither outcome is a saving. At low reasoning effort, the median cost per task rose 7.6 percent (p=0.004) compared to stock Claude Code. At high reasoning effort, the difference was effectively zero (+0.1%, p=0.99). The tool's own rtk gain counter reported 96.2 million tokens saved in the same session where costs increased — because the counter measures against a theoretical maximum output that Claude Code's built-in truncation prevents in practice.
Why doesn't rtk's compression reach the tokens that actually drive Claude Code costs?
Structurally, Claude Code reads most files with its built-in Read and Grep tools, which bypass the Bash hook rtk relies on. Roughly half of what agents run in a shell are commands rtk doesn't handle (like python3), and a sixth of Bash calls use pipes or heredocs that the tool deliberately avoids rewriting. What the hook can reach carries less than 20 percent of tool-result characters — and tool results themselves are only a fraction of session input cost, because the same early context gets re-read on every agent turn at cached rates (billed at one-tenth of fresh input token prices). The JetBrains analysis estimated the theoretical ceiling at roughly 3 percent of input tokens before running a single paid trial.
Does rtk affect the quality of what Claude Code produces?
No measurable impact was found across either effort level. The benchmark ran 86 tasks at both low and high reasoning effort; sign tests in both conditions returned p=1.0, meaning the arms were statistically indistinguishable on task outcomes. The feared failure modes — over-filtered test output, masked exit codes — barely appeared in practice. The compression is technically real; it just doesn't reach the tokens that move the bill.
How can developers tell if any token-saving tool actually works?
The JetBrains benchmark series has established a methodological standard: measure the paired billed cost across multiple trials, not the tool's own savings counter. A tool's self-reported savings estimate its counterfactual — what the session would have cost without any compression — against theoretical maximums rather than what the billing system actually charges. Single-trial evaluations are unreliable because identical agentic tasks in the same arm routinely differ by 20 percent or more in cost due to inherent session variance. The minimum credible evaluation involves multiple paired runs across a representative task set, with cost measured from the actual billed invoice rather than from the tool's internal diff.
Related Articles
