Removing dead code: the CodeLaser engine versus Claude Code

In short

What the experiments show

• What the CodeLaser engine did. Given the four projects and a list of what the outside world reaches, it removed everything unreachable from each one — from twenty declarations in the smallest to about three and a half thousand in the largest — in between one and a half and six and a half minutes per project, without compiling anything to decide. Every result builds, and none of the checks run afterwards found a problem in any of the four.
• The agent takes between 15 and 54 times longer than the engine, and spends tokens, which the engine does not. The CodeLaser engine took between 1m 36s and 6m 25s per project; a single agent attempt took between 30 and 121 minutes, 15 to 54 times longer, and consumed between 10.7 and 47.1 million input tokens. The engine is a compiled program and consumes none.
• The agent is non-deterministic. It does not do the same thing twice. Three attempts on Caffeine, from an identical starting point and an identical instruction, removed 18, 40 and 231 declarations. Three on Timefold Solver removed 1,706, 1,478 and 1,317. Each attempt wrote its own analysis program and no two were alike — one wrote none at all and used the compiler as its oracle. None of them reached what the engine does: the same removals on every run, and a rule for each of the ways a declaration stays alive without any Java code referring to it.
• When the agent runs out of time, there is nothing to show for it. Three of the eight agent attempts never finished. On Trino the agent spent 196 minutes across two attempts and 35.6 million input tokens, and removed nothing at all: the time went on writing and repairing an analysis program rather than on removing code. The engine finished every project it was given.
• The agent deletes declarations it was told in writing are reachable. Both sides were handed the same list of everything the outside world reaches. On QuestDB the agent deleted six entries from that list, among them a test method and five declarations the project’s own authors marked as deliberately uncalled. The engine deleted none, on any project.
• The agent deletes code in ways that still compile and are still wrong. Four agent attempts left classes with no declared constructor, so Java silently supplies a public one and a class that could not be instantiated now can be. Agent attempts deleted fields whose initialising expression runs code, up to 128 in one attempt. One agent attempt emptied a list of named options that a configuration file selects by name, disabling a feature at run time. Every one of these passes a full build. None of the engine’s runs did any of it.

Every count was taken by parsing the source before and after each run, with the same parser for both sides. The sections that follow set out the evidence.

Removing dead code is worth doing, and an agent is a good thing to have driving it. What these experiments set out to establish is where the work should sit. Solving a problem by writing code for it is what an agent does, so an agent handed this task starts by building a dead-code analysis of its own. That is the natural first move and it is the wrong one: an analysis that resolves every name in a Java project is far more than can be built in two hours, and these eight attempts spent most of their time and tokens finding that out.

Give the same agent the CodeLaser engine and the work divides the right way round. The engine does the code mechanics: resolving every name, working out what nothing reaches, and making the removal. The agent does what an agent is good at — deciding what to tackle, judging the findings, explaining the result. Its tokens go on reasoning and on talking to you, not on rediscovering how Java works.

The setup

Experimental setup

Dead code is code that nothing can reach: a method nobody calls, a class nobody names, a field nobody reads. Removing it is useful, tedious and risky, because "nothing reaches this" is a claim about the whole program.

Two approaches were compared on the same task. The first is the CodeLaser engine, running its dead-code operation: it parses the project, builds a graph of what refers to what, and deletes what cannot be reached. The second is Claude Code, the same coding agent a developer would use interactively, working autonomously with full access to the project and a shell. Version 2.1.252, on Claude Opus 5, the same build and model in all eight attempts. Where this document says the agent, that is what it means.

Claude is on both sides of this comparison. The engine’s side uses a Claude instance as well: it produced the entry-point list both sides were given, and the short script that drives the engine. So this is not a tool measured against a rival. It is the same agent, asked in one case to drive a program built for the job and in the other to build that program from nothing.

The tool

How the CodeLaser engine removes dead code

Four things about the CodeLaser engine explain the results that follow.

First, the task is two problems rather than one. Finding the dead code is the easier of the two; taking it out is where the difficulty is.

Finding dead code has a cheap approximation. Counting references and walking outward from the entry points gets most of the way, which is why every agent attempt wrote code to do exactly that. What a reference count cannot find is a declaration whose reachability is decided outside the Java source. Two of the four silent failures below are of that kind: an enum whose constants a configuration file selects by name, and six declarations the agent deleted although the entry-point list it had been given named them as reachable.

Removing dead code has no approximation at all. Deleting a declaration means repairing every place that referred to it, and following the consequences: methods that only it called are now dead too, imports and overrides and signatures have to be corrected, and the code left behind can often be simplified further. Each deletion can make another declaration dead, so the work repeats until a pass finds nothing. The other two silent failures are here: the code really was unreachable, the agent was right to find it, and deleting it still changed what the program does.

The engine resolves every name before it decides anything. The project is parsed once and every name is resolved to what it denotes: which declaration a call reaches, which type a name means. Reachability is computed over that, outward from the entry points. What it costs to work from compiled output instead is set out under Four repairs caused by working from compiled classes.

The engine knows where reachability is decided outside the code. A name in a configuration file, a class loaded from a string, an enum constant selected by text at run time: none of these is a reference the compiler can see, and counting references will not find them. The engine carries a rule for each such mechanism. One of them is set out in full in the fourth example below: a call to valueOf or values() marks every constant in that enum as used, because which one will be asked for is not decided until the program runs. That single rule protects the ten strategy classes an agent attempt deleted.

The engine refuses rather than guesses. Where it cannot show that a removal is safe, it declines to make it and reports why. The clearest case is a field whose initialising expression calls a method: showing that the call only computes a value means following every call it makes, through code that may not be available to read. The engine reports such a field as unused and leaves it in place. That caution costs it real removals — four on Caffeine, where the expressions turned out to be harmless. The third example below is the same situation going the other way, where an agent attempt deleted 128 such fields and broke the tests that depended on them.

The engine does not need to compile anything to decide. It determines reachability from the source directly. The build afterwards is verification, not part of the method, which is why the engine’s builds run figure is zero on all four projects while the agent’s is between 7 and 25.

Results

What each run removed and what it cost

One table per project follows. Counts are of declarations — individual types, methods, constructors, fields and enum constants — and were measured independently by parsing the project before and after each run, not taken from either side’s own report of what it did. The five kinds sum to the total in every row.

Observation

The agent costs far more, in time and in tokens

On Caffeine the CodeLaser engine finished in 1m 36s. The agent took between 30 and 74 minutes. On Timefold Solver the engine took 2m 14s; the agent took 34 minutes at best, and in one attempt used the entire two-hour cap without finishing. Timefold’s three attempts are both ends of the range quoted above: 15 times the engine at best and 54 times at worst. It is not explained by the agent doing more work: on Timefold its best attempt removed slightly less than the engine.

On QuestDB the CodeLaser engine took 4m 10s and removed 3,531 declarations. The agent took 90 minutes and removed 991: twenty-one times longer, for under a third of the result. That is its best showing, the one attempt that finished because it had nothing left to find.

Put the two sides on one axis, project by project, and the shape of the benchmark is the whole argument. The engine’s bar is the shorter one every time, it costs no tokens, and it is the only one that ends clean. The agent is shown at its best on each project; where there were three attempts, the other two were worse.

On Trino the agent removed nothing, while the CodeLaser engine finished in 6m 25s and removed 1,478 declarations. Claude spent 196 minutes and 35.6 million tokens across its two sessions. Late in the second it found it had deleted a method that is named only in a string, undid all of its removals, and was applying them again when the two-hour cap stopped it.

Every time in this report includes the project’s first build. For both sides the clock starts when that build starts and stops when the tool finishes its own work. What the harness does afterwards to check the result — compiling it again, counting it, running the invariants — is excluded from every figure. Engine times are given to the second; agent times run to tens of minutes and are rounded to the minute.

Most of the engine’s 6m 25s on Trino is not analysis. Working out what nothing reaches, across eleven thousand files, takes 14.2 seconds. The rest is the project’s own first build, resolving the entry points, and writing the edits. In the order they happen:

Deciding what is dead is the cheap part. Resolving every name in the program, and then making the edits, is where the engine spends its time. An agent has to do both as well. It starts from the source files alone, with no resolved representation of the program.

The agent’s method is to compile and look at what broke. It removes some code, rebuilds the project, reads the errors, and repeats: between 7 and 25 builds per attempt on Caffeine and Timefold Solver. Without a representation of what reaches what, trial and error is the only method available. It is also the part that grows worst with project size. Trino’s build takes about ninety seconds, and a loop that rebuilds after every round pays that every round.

Tokens show the same pattern. A single attempt consumed between 10.7 and 47.1 million input tokens; the engine consumed none. That is the cost of one attempt at one project. Because one attempt does not predict the next, getting a usable result can take several, each at that cost.

Three of the eight attempts did not finish. One on Timefold was still working when the cap stopped it, and the code it had produced by then does not compile. The Trino effort ended with nothing removed. In each case the time was spent and nothing usable came of it. Only the QuestDB attempt stopped because a pass found nothing more to remove.

Caffeine is 160,000 lines and Timefold Solver 355,000. Trino and QuestDB are over two million lines each. QuestDB has more lines with half as many files, because its files average 379 lines against Trino’s 181.

Neither side’s time grows in proportion to size. QuestDB has fourteen times as many lines as Caffeine; it took the engine 2.6 times as long and the agent’s best attempt 3.0 times as long. The chart below fits a curve to each side’s runs, and both come out close to the square root of project size: the engine’s time grows as size to the power 0.45, the agent’s as size to the power 0.44. The two sides grow at much the same rate, so the gap between them holds at fifteen to thirty times. The agent’s curve is the looser fit of the two, because its attempts scatter so widely: its three on Caffeine took between 30 and 74 minutes. What size does change is whether the agent reaches an answer at all. On Trino its two sessions together ran past the cap for a single attempt, and it removed nothing.

Observation

The agent is not reproducible

Three attempts were run against Caffeine and three against Timefold Solver, from an identical starting point and an identical instruction. They did not produce the same answer, and they did not use the same method. Trino and QuestDB were not repeated. Measuring the spread on projects of that size was not considered: a single effort on Trino already took more than three hours. For those two the spread is unknown, which is not the same as zero.

On Caffeine the three attempts removed 18, 40 and 231 declarations, a factor of thirteen between the smallest and the largest. On Timefold Solver they removed 1,706, 1,478 and 1,317. The engine was run twice on QuestDB, three days apart, and removed the same 3,531 declarations both times. It performs the same computation on the same input, so it gives the same answer.

The variation comes from the agent choosing a different approach each time. Across the six attempts, three distinct strategies appeared:

• Working by hand. Read files, judge each case, edit directly. Slowest per declaration and the most conservative: on Caffeine this attempt removed the fewest.
• Writing an analysis program. Four of the six attempts wrote their own tooling: a program that reads every source file, builds a map of which declaration refers to which, walks outward from the supplied entry points, and lists what was never reached. Some attempts went further and wrote a second program to perform the deletions, a third to tidy up the leftover imports, and a script to run the whole cycle repeatedly until nothing more was found. One attempt also kept a list of declarations it had learned not to touch, after a previous deletion broke the build.
• Using the compiler as the oracle. One attempt wrote no analysis program at all. It removed code that looked unused, rebuilt, read the errors, and put back whatever the compiler complained about — seventeen times.

The agent chose among these methods itself; the instruction did not name one. So a result from one run says little about what the next run will do, and the quality of the outcome depends on which method the agent happened to pick. On Caffeine the same instruction produced both the smallest removal, 18 declarations, and the largest, 231.

Observation

Errors in the agent’s output

Three of the checks run over every result are the subject of this section. The first asks whether anything on the supplied entry-point list was deleted. The second asks whether a surviving class lost every constructor it declared. The third asks whether a field was deleted whose initialising expression calls a method. All three are cheap and specific. None of them is an audit: each looks for one known failure pattern and says nothing about anything else.

Those three checks alone found problems in every agent attempt that removed anything, and in none of the four engine runs. What follows are examples of places where an attempt is known to have gone wrong.

    @Test
    public void testBadJsonExtract() throws Exception {
        testBadJsonExtract(ColumnType.BOOLEAN, "false");
        testBadJsonExtract(ColumnType.SHORT, "0");
        testBadJsonExtract(ColumnType.INT, "null");
        ... eleven column types in all ...
    }

public static final ParameterSpec keySpec = ParameterSpec.builder(kTypeVar, "key").build();

strategy = [
  simple,
  correlation,
  trust-region-ewma,
  ...
]

public ImmutableSet<HillClimberType> strategy() {
  return config().getStringList("hill-climber-window-tiny-lfu.strategy").stream()
      .map(strategy -> strategy.replace('-', '_').toUpperCase(US))
      .map(HillClimberType::valueOf)
      .collect(toImmutableEnumSet());
}

public enum HillClimberType {
  SIMPLE((_, config) -> new SimpleClimber(config)),
  CORRELATION((_, config) -> new CorrelationClimber(config)),
  TRUST_REGION_EWMA((_, config) -> new TrustRegionEwmaClimber(config)),
  … six more, ending
  INDICATOR(IndicatorClimber::new);

public enum HillClimberType {
  ;

Observation

The agent’s real task is to build a dead-code engine under a clock

The instruction was to remove dead code. The work that filled the time was writing a program to decide what dead code is. The task cannot be done without such a program. The engine is one; the agent had to write its own.

On Trino the agent wrote nineteen files before removing anything: a reader for compiled Java classes, two scanners built on the compiler’s own parser, an index of every library on the classpath, a reachability pass, two planners, a text editor that removes a declaration along with the comment above it, an import fixer, a snapshot-and-rollback, a repair loop that reads compile errors, and an outer loop that repeats the cycle until a pass finds nothing. More than an hour of the effort went into building that, and another 47 minutes into repairing it, before a single declaration was removed.

Conclusion

Codebase-wide changes need an engine behind the agent

An AI coding agent is good at writing new code within a bounded scope: a function, a feature, a test, a fix. These experiments show that too. Most attempts wrote their own analysis programs without being asked to — readers for compiled classes, reachability passes, planners, repair loops.

Removing dead code is a different kind of task. It changes the whole codebase at once, every decision in it depends on the rest of the program, and it has to be right in thousands of places. On that task the experiments show four problems:

• It is slow and costly. Claude took 15 to 54 times as long as the engine on Caffeine, Timefold Solver and QuestDB, and consumed 10.7 to 47.1 million input tokens per attempt. See The agent costs far more.
• It does not reach an answer on the largest codebase. On Trino, two million lines, it spent 196 minutes and 35.6 million tokens and removed nothing.
• It gives a different answer each time. Three attempts on Caffeine, from the same starting point and the same instruction, removed 18, 40 and 231 declarations. See The agent is not reproducible.
• Its mistakes pass the build. It deleted a test that was on its own entry-point list, made classes publicly instantiable, deleted fields whose initialiser set up the tests, and emptied an enum whose constants a configuration file selects by name. Every one of these compiles. See Errors in the agent’s output.

These problems do not come from a careless agent. On QuestDB it worked the way a careful engineer would, and still removed under a third of what the engine removed. They come from asking it to build, within reasonable time, what a codebase-wide change depends on: every name in the program resolved, and a rule for each way Java code stays alive without being referenced. See The agent’s real task is to build a dead-code engine under a clock.

That is what the CodeLaser engine provides. It also gives the same answer every time: run twice on QuestDB, it removed the same 3,531 declarations.

Used together, each does the part it is suited to. An agent that calls the engine leaves the code mechanics to it — resolving names, working out what nothing reaches, making the edits — and spends its own time and tokens on deciding what to tackle, judging the findings and explaining the result.

The complete record of every run, from diffs and build logs to session transcripts, is available to anyone evaluating CodeLaser.