Apache Ignite · Reducing the dependency loop inside ignite-core from 2 476 classes to 483 using CodeLaser

In short

1. The loop reduced from 2 476 classes to 483 and one module separated out

• The goal. ignite-core is one artifact. Anything that uses any part of it compiles against all of it. The goal was to find out how far the core can be separated into modules that build in a fixed order, without changing what a node does. The second goal was to do as much of it as the code allows.
• The obstacle. 2 476 classes in the core depend on each other in a loop. A module has to be built before whatever uses it. Inside a loop there is no first, so no boundary can be drawn anywhere in it. Every one of those classes lives in core/main.
• What was done. The loop was reduced in 65 commits, from 2 476 classes to 483. Most of them are small: a class looked up by name, a constant moved to the class that owns it, an interface declared for what a caller uses. One is large: the kernal context was split into a lower interface and an upper one, which took 27 commits and removed 985 references from the lower half of the core into the cache engine.
• One module separated. The management command tree, 520 classes, is now ignite-management, a Maven module built after the core. 299 files moved without changing package, and the core shrank from 5 632 to 5 112 classes. Step 3 gives the bill.
• What the code now allows. With the kernal context split, the core has no reference from its lower half into the cache engine. Measured on the whole of core/main when the split closed, five build units follow from that boundary, every dependency between them pointing downward. They are measured, not yet cut. Section 9.
• What changed in the public packages. No public type changed package. Sixteen declarations in public packages changed, all of them signatures that already named an internal type. Two are extension points that an implementation outside the repository would have to follow. Section 8.
• How it was checked. Every commit builds the whole reactor with Ignite’s checkstyle at zero violations in all 43 modules. Every change was tested by the suites that exercise it, and most were tested a second time with the change broken on purpose, to prove the suite can tell. The whole suite was not run. Section 10 says exactly what was and was not covered.
• How it was done. The CodeLaser engine holds a model of the source. It answers questions about it with counts, prices a proposed change before it is made, and applies changes as operations. Claude Code works from those measurements, predicts what a step will do to the loop, and measures again after the step lands. Every step’s prediction is on the record beside its measurement.
• Where it ends up. 483 classes in the loop, and a priced road onward: thirteen further subsystem moves that would take it to about 260, each paying on its own. Section 9.

The setup

2. The project setup

Four components were involved: the Ignite source, the CodeLaser engine, Claude Code, and the Maven build.

The Ignite source. A clone of apache/ignite at master commit a5b6b0dc44, version 2.19.0-SNAPSHOT, on its own branch. The upstream repository was never written to.

The CodeLaser engine. A refactoring and modernization engine. It reads the whole project once and builds a model of it — every type, every member, every reference from one to another. It answers questions from that model, and it changes code through operations.

Claude Code. Drives the work. It puts questions to the model, receives measurements and ranked options back, chooses the next step from them, and issues the change as an operation. It records what it expected each step to produce, then what was measured.

The Maven build. Ignite’s own build, unchanged, on JDK 17. After each step it compiles the whole reactor with Ignite’s checkstyle, and runs the tests that exercise the change.

The core has 796 generated source files — message serializers, system-view walkers and data-transfer-object serializers, produced by Ignite’s own annotation processors. A build treats them as compiled output. This work treats them as source, because a generated class can hold a dependency as real as any hand-written one. The loop measured with generated code as source is 2 476 classes; without, it is 2 266. Every figure in this document uses the first, except the few marked as measured without the generated code. Appendix B says why it matters.

The tool

3. How the CodeLaser engine works

This section describes what CodeLaser adds to an editor and a compiler.

It holds a model of the code, not the text. The project is parsed once: 57 source sets and 10 962 types. What CodeLaser then holds is its structure — the types, what each one declares, and how the type hierarchies fit together. It also holds every reference from one place to another: calls, constructions, field accesses, annotations, imports. The dependency graph follows from that.

So do the questions worth asking of it. "How many places create this class by name?" comes back as a number. So does "which classes leave the loop if this reference goes", and "what would this boundary cost".

It prices a change before the change is made. Given a set of classes, CodeLaser reports which references keep them in the loop and how many classes would leave it if those references were cut. Given a proposed boundary, it reports how many references cross it and in which direction. Different sets can then be compared by their numbers.

It changes code as whole operations. Moving a set of static members to another class is one instruction. CodeLaser finds every place the change reaches and rewrites each one — imports and static imports included. One instruction in this work rewrote 420 call sites across 207 files.

It refuses what it cannot do correctly. An operation that would leave the project not compiling is declined before anything is written. After every write, the whole reactor is recompiled, and a failure stops the work there.

It rebuilds its model after every change. Every result in this document was taken from a fresh parse of the source after the change.

The first four change nothing; they answer a question. Most of the calls in this work were of that kind: 199 calls in 59 scripts, and the great majority of them measurements.

Here is one of those calls, from step 0. It asks what would happen to the loop if the root of the management command tree were taken out.

graph.cycleImpactWithout(
    types=["org.apache.ignite.internal.management.IgniteCommandRegistry"])

typesFreed          154
largestCycleAfter   2320

Without that one class, 154 classes would leave the loop entirely, and the largest loop left would be 2 320 classes. Appendix B comes back to the 154.

The starting position

4. What a build boundary requires, and why the core cannot simply be cut

Ignite builds with Maven and does not use the Java module system. Two of the module system’s three rules therefore do not apply: a module declares no exports, and a package may be split across two artifacts on one class path. One rule does apply, and it decides everything that follows.

A module is built before whatever uses it. Module B can use module A if A is built first. Two modules cannot depend on each other, in either direction, at any remove. So a set of classes can become a module only if every reference between the set and the rest runs one way.

That is what makes the core hard to cut. 2 476 of its classes reach each other in a loop: A uses B, B uses C, and C uses A again. No order puts any one of them first, so no boundary can be drawn between any two of them. All of the loop is in core/main. Measured without the generated code, it holds 76% of all the classes in the repository that sit in any loop. The next largest loop, 289 classes, is in the Calcite module and does not touch the core.

Moving files does not help. A class moved from one side of a proposed boundary to the other takes its dependencies with it, so the loop is the same size afterwards. The loop shrinks only when a dependency is removed. Every step below removes dependencies; one of them then moves files.

Ignite’s own maintainers had already started. The modules extracted under IEP-119 — commons, binary, thin-client, nio, unsafe — are out of the loop, and the 2.17.0 release measured at 3 262 classes in it where master measures 2 476. Each extracted module still carries a small loop of its own, because the dependencies inside it moved with the code. This work continues in the same direction, and its two first steps are on IEP-119’s path.

Method

5. The loop each step of the work went through

The work followed the same loop at every step: measure the code as it stands, get back a ranked set of possible moves, take one, and check the result against what was predicted.

1. Measure the source itself. Every figure in this document was derived from the Ignite source, on the branch the work ran on, at the moment it was needed. Nothing was carried in from elsewhere.
2. Establish that a step is possible before pricing it. Before any cost is estimated, CodeLaser is asked what holds a set of classes in the loop and which references would have to change. A set that is held by an extends, or by a public method’s return type, was not priced further.
3. Predict the outcome as a number, then compare. Each step was priced before it was made and scored afterwards. The prediction is the size of the loop after the step, and the classes that leave it.
4. Make the change. Through a CodeLaser operation where one exists for the shape, and otherwise as a scripted edit built on CodeLaser’s measurements. Section 12 says which was which.
5. Check with the tests that reach the change, and prove they can tell. Every commit compiles the whole reactor with checkstyle. The suites that exercise the changed code are then run. For most steps they were run a second time with the change broken on purpose, a class name misspelled or a service entry deleted. A suite which stays green either way is then known not to count.
6. Score the step by measuring again. The loop is re-measured from a fresh parse, and its members compared with the prediction one by one.

Step 5 is where this work differs from a whole-suite standard. Ignite’s suite is large, and the routine here ran the suites that reach each change rather than all of them. Section 10 gives the consequence.

What was done

6. Stepwise overview of the work

Steps 1 to 6 and 10 were each predicted to the class before they were built, and measured to the class afterwards. In two cases the measurement was higher than the prediction. In step 1 the one extra class is the new interface the step introduced. In step 4 the two extra classes are held by a helper that moved, and freeing them would change how peer class loaders are found.

Step by step

7. The eleven steps, in more detail

The trade-off

8. What changed in the public packages

No type in a public package changed package or name. Sixteen declarations in public packages changed their signature, and every one of them already named an internal type — a kernal context, an implementation class, an internal manager — in a public position. They fall into three groups.

Two further changes in public classes are not signature changes. The machine-derived defaults of IgniteConfiguration and the constants of IgniteSystemProperties moved to interfaces the two classes already implement, and stay readable under their old names. Ignition defines its exit codes from the new IgnitionLifecycle, and they remain compile-time constants.

⚠ Two of these changes can fail without a compile error. Widening a parameter is source-compatible for every caller and for every implementation inside the repository, which the compiler checks. An implementation outside the repository is the one case the compiler cannot see. A plugin that implements MessageMarshaller by hand would compile and would stop being called. So would a subclass outside the repository that overrides setIgnite on the multicast IP finder with its old parameter type.

The result

9. One separated module and five build units the code now allows

Assurance

10. How each step was checked

Three checks were applied at every commit, and one comparison at the largest step.

The whole reactor compiles, with checkstyle. Every commit builds all 43 modules with Ignite’s checkstyle profile at zero violations. A change that does not compile never landed.

The suites that reach the change pass. For each step, the suites that exercise the changed code were run. That means a node started with the changed component, a command sent over the wire, a thin-client handshake, a page written and read back. Section 7 names the behaviour each step touches. The record names each suite.

The suite can tell. For most steps the same suites were run a second time with the change broken on purpose: the class name misspelled, the service entry deleted, the wrong page format answered. A suite that stayed green with the change broken did not count, and another suite was used in its place.

The largest step was compared against its own starting point. For the kernal context split, 65 suites across six modules — 882 tests — were run on the finished split and on the commit before it, built the same way. Five tests fail on both. They are a system view that lists a directory in an unsorted order, a discovery test bound to the machine’s network interface, an upstream test that fails on purpose pending IGNITE-4706, and two page-compression tests that need Linux. Nothing fails on one side and not the other.

⚠ The whole test suite was not run. Ignite’s core test sources hold about 3 400 test classes. The routine here ran the suites that reach each change. That is a weaker standard than running the whole suite. Every changed behaviour has a test that passes and that was shown to fail when the behaviour is broken. No claim is made about behaviour no step touched.

Aside

11. Three things in Ignite’s code found along the way

None of them affects a result above. Each is worth knowing.

EventType and IgniteUtils initialise each other through Unsafe. EventType.EVTS_ALL is computed by IgniteUtils, whose static block reflects over EventType and, when the field is still null, writes it through putObjectVolatile. The two classes know about each other on purpose, and Unsafe repairs the initialisation order. A cleaner design has EventType compute its own list.

Components found by name are found by constructor signature. IgniteKernal, IgniteComponentType and CachePluginManager look a component up with getConstructor(GridKernalContext.class). A component whose constructor now takes the lower KernalContext keeps a constructor the lookup finds. Nothing at compile time checks this; only a node start does. It is written down because the next person to change a component’s constructor will not be told by the compiler.

The test harness shares one JVM across modules. The parent pom sets forkCount to 0, so the tests of every module in one Maven invocation share a JVM. Ignite’s serial filter is JVM-global, so the second module’s grid tests fail at class initialisation. Run one module per invocation and everything is green. One test, GridManagerMxBeanIllegalArgumentHandleTest, installs a failing mock in a static field that every later suite in the same JVM then inherits.

Effort

12. Effort required to compute and perform the work

Included because it shows what repeating the exercise would involve.

The work removed more lines than it added. The two largest deletions are the generated factory in step 2 and the 68 declarations that left GridKernalContext in step 9.

How the changes were made. Steps 1 to 6 were made through CodeLaser operations or through scripted edits built on CodeLaser’s measurements, with every write recompiled by CodeLaser before the next. Steps 7 to 10, including the kernal context split, were made through CodeLaser’s operations.

The predictions. In the first thirteen sessions every claim was written down before the run that would test it: 217 predictions, 168 held. Almost all of the refuted ones estimated the cost of a step: how many call sites or files it would touch. The size of the loop after every built step was predicted to within two classes. From the fourteenth session on, each step carried its prediction of the loop and its measured result without the formal tally.

⚠ Time is not reported. Most of the elapsed time was the build running, which depends on the machine it runs on.

Appendix

A. What was not done, and why

• The thin-client server side stays in the core. Section 9 gives the reason.
• The first target, a loop of 200 to 400, was set aside. When the loop stood at 1 190 classes, every arrangement the search found below about 500 had to cut one of three things. The first is the calls of Ignition into IgnitionEx, which is the facade’s whole purpose. The second is a built-in failure handler’s call to IgnitionEx.stop, which its javadoc names as its contract. The third is a configuration bean’s field holding its own sub-configuration. None of those is a change a reviewer should accept, so the target became about 555, the smallest loop reachable without them. The kernal context split reached 558, and step 10 then reached 483 without touching any of the three.
• Reflection was not introduced for the graph’s sake. Several one-reference changes were priced and refused: making PdsConsistentIdProcessor or a default failure resolver reflective would have freed a few classes each and served no design purpose. The by-name constructions that were made each follow a precedent in Ignite’s own code — IgniteComponentType, the platform processor, the thin-client service factory.
• Two changes were made that a maintainer may want to revisit. The statistics manager created by name in step 5 treats query statistics as a pluggable component. It is a separate commit so it can be dropped, and the REST change stands without it. The Java-object key serializer in step 8 moved a mutable global from one class to another rather than redesigning it. The commit says so.

Appendix

B. How to read the figures

Two loops were measured at the start, and this document uses the larger. With the 796 generated sources treated as a compiled library, the loop is 2 266 classes. Treated as source, it is 2 476. The first is what a build sees; the second is what the dependencies are. The difference matters because one generated class held 296 classes in the loop, and a model that cannot see it prices the separation of the command tree alone at 278 classes for its six references, where the true figure is 154. Every figure here is from the second model, except the few marked as measured without the generated code.

A prediction is the size of the loop after a step, and which classes leave it. Where a measurement differed from its prediction the difference is stated in the step: one class more in steps 1 and 7, each a new interface the step introduced; two more in step 4; nine fewer in step 8, from overrides the prediction had not been given.

The build units in section 9 have not been cut. They follow from giving every class in core/main to the side it transitively requires, and from checking that no reference then runs upward. The test sources, which Maven compiles inside the core, would have to be divided the same way, and the bill step 3 paid for one module is owed for each of them.

The road onward in section 9 is an estimate. Each of the fourteen moves was priced from the graph at the 558 state. The first has been built and measured exactly. The rest have not been built. The graph gives the number of references each would change. It does not say whether the change is a good design.

What the figures do not cover. The whole test suite was not run; section 10 gives the standard that was applied instead. Runtime behaviour that no test reaches is not claimed. Where this document says a change "stays readable under its old name", that was checked by compiling and, for the compiled constants of step 6, by comparing every value before and after.