Training a large model outside a data center has stopped being an experiment

On 6 November 2025 Bitcoin closed at $101,346, −2.4% against the previous day, while the day’s news was about the distributed training of artificial intelligence models.

What actually got finished

In November 2024 one group finished a model of 10 billion parameters trained on 112 cards spread over three continents, keeping the machines busy 96 percent of the time under the best conditions. In May 2025 the same group finished one of 32 billion with more than 100 nodes unlike each other, more than 400 hours of training, and the whole process public and repeatable by anybody.

That first model’s scores on the standard tests were no record — thirty-seven and a half out of a hundred on one, seventy-two on the other — and the people who made it never claimed otherwise. The result wasn’t the score: it was that volunteers scattered around the world, with machines coming into the training and dropping out of it while it ran, had carried the thing to the end using 400 times less bandwidth than the normal method.

Meanwhile a project running inside a network of subnetworks has more than 200 cards in production, has finished a model of 1.2 billion parameters and is aiming for 70. These are all figures declared by the people who produced them: checkable in the published work, not redone by us.

The wall was bandwidth, and how they lowered it

The first blow was to stop synchronizing at every step and to do it every 500: the machines work on their own for a stretch and realign afterwards, and that alone brings the communications down five hundredfold. The second was compressing what gets sent — a transform plus a reduction to a single bit — with reductions declared at between a thousand and 10,000 times against the standard methods.

The third is sharper still: synchronize only 0.1 percent of the parameters at each step, and the model converges all the same. In December 2024 one of these methods trained a model of fifteen billion parameters over eleven thousand steps with machines supplied by four different operators. The wall didn’t fall to a single idea: it fell to three tricks stacked on each other.

The other road: split the model, not the data

In September 2024 a group completed a training run of this kind for the first time across different participants and unequal machines: seven and a half billion parameters, three weeks, thirty-six billion words. A year later they showed a model of eight billion trained with its blocks in four different places joined by ordinary internet, with results on a par with a centralized setup — something that before that work was taken to be impossible.

The piece that makes the thing workable is the handling of failures. Another group built a system that looks at which computing routes are free and skips the slow or fallen nodes: it holds 93 percent of its output even when 50 percent of them go down. In a data center a node that vanishes is a fault; here it is a normal operating condition.

Why reinforcement learning falls well here

Out of it came a setup that separates three things that used to sit together: the training, the use of the model to generate answers, and the delivery of the updated weights. Every machine does its own round on its own account and delivers when it has finished, with a light check that verifies it really learned from what it saw rather than copying or inventing.

Another group put a three-beat mechanism on its test network: every node answers on its own, then the nodes criticize each other, then they settle on a common answer, and the reward depends on how close each one was to that agreement. It is a structure that looks more like a newsroom than a computing shed, and it works for the same reason: nobody has to trust anybody.

How you check that somebody is really working

There are four different answers going around, and they are worth keeping apart because they fail in different ways. The first is economic: whoever works puts down a bond, whoever cheats loses it, and anybody can report them. The second watches behavior: it checks that the corrections delivered are consistent with the data that machine saw. The third has validators vote on the quality of what arrives, and penalizes the validators who inflate their votes as well.

The fourth, not yet running, tries to make the work itself the proof: every answer generated carries a stamp showing which model produced it. None of the four is settled. One of these networks already came under a real attack over the 2024 holidays, and the defense is the usual chase between whoever attacks and whoever protects.

The four approaches to verification: economic, based on a bond put down that is lost by cheating; behavioral, which checks the consistency between the data seen and the corrections delivered; by vote, where validators judge the quality and whoever inflates the votes is penalized; by stamp, where every answer generated carries cryptographic proof of which model produced it.

The money inside it

Of live tokens, for now, there is essentially one: the token of the network of subnetworks, which in December 2024 touched $5 billion of market value and which since February 2025 has had separate tokens for each of its 118 subnetworks. All the others are awaited, and the people waiting for them say so openly: they are capital raises with a token promised later.

Capital raised by the groups named: nous research, $65m (declared valuation $1bn); gensyn, $43m (test network, no token); prime intellect, $20.5m (no token announced); pluralis, $7.6m (shares of the model, not a fixed wage).

The estimate going around all of this is that the artificial intelligence market will be worth $15 trillion by 2030. It is a partisan estimate, made by people who have put money into these projects, and it is worth what every five-year estimate is worth: it says what the people who wrote it are hoping for, not what will happen.

What doesn’t add up, and what to watch

The fourth is the most awkward and concerns real money: if the finished model’s weights are public, who pays for having trained it? One of the groups tries to solve that by giving the participants a share of the model instead of a fee, but that share is worth something only if the model generates revenue, and there is no guarantee at all that it will.

From outside there are four things to watch, and they are all accounting: how many cards actually take part — two hundred today in the project furthest along, and the useful threshold is fifty times that; how many training runs reach the end without being aborted; how much a unit of computing costs here against the traditional suppliers; and how far these models are from the best centralized ones. Bitcoin, in the middle of all this, closed at $101,346, -2.4 percent: the days when a piece of an industry moves are not the days when the price moves.

Put bluntly: in a data center a node that vanishes is a fault to repair; in a network of volunteers it is the normal condition, and all the research went into building on top of that rather than preventing it.