Every large language model spends most of its electricity on one operation: multiplying a matrix by a matrix. The two dominant ways to build a chip for that job are tensor cores and systolic arrays, and the difference between them is not raw speed but how operands move. One pulls values out of registers on each instruction; the other lets values ripple through a wired grid so each is read many times before it leaves the chip.

A tensor core executes a small fixed matrix tile — say 16 by 16 by 16 — in a single instruction from registers, while a systolic array hard-wires a grid of multipliers so each operand is fetched once from memory and then passed neighbour to neighbour across the whole array.

The choice is a bet about where the bottleneck sits. If it is the ability to issue and schedule work, tensor cores win. If it is the energy cost of dragging data from memory, systolic arrays win. Both compute the same mathematics.

Part One: What each design actually does

A tensor core is a specialised block inside a general purpose GPU. A warp of threads presents it with two small matrices held in registers, and one instruction performs all the multiplications and accumulations for that tile. The programmer still writes code, still schedules instructions, and still decides which tile goes where. The tensor core simply makes one common operation absurdly fast compared with doing the same multiplies one lane at a time.

That flexibility matters because real workloads are messy. Layer shapes vary, sparsity patterns appear, and new operators show up every year. A tensor core sits behind an instruction set, so a compiler can retarget it. When a model changes, the chip often keeps working because the software changes instead.

A systolic array takes the opposite approach. It is a fixed two-dimensional grid of processing elements, each connected only to its immediate neighbours. Data enters at the edges and flows through the grid in waves. A weight loaded at the top moves down; an activation loaded at the left moves right; where they meet, a multiply-accumulate happens. By the time a value exits the far side, it has participated in many operations without ever going back to memory.

In a systolic array each activation is reused across an entire column of multipliers and each weight across an entire row, so the ratio of arithmetic to memory traffic climbs with the square root of the array size rather than staying flat.

Part Two: The mechanism that decides the winner

The mechanism is the memory wall. Moving a number from off-chip memory costs roughly two orders of magnitude more energy than multiplying two numbers already on chip. So the design that reuses operands more often spends less energy per multiply and needs less memory bandwidth, which is the constraint that dominates large inference and training runs.

A systolic array is built for reuse. Once a tile is streaming, the grid is almost a closed loop: operands circulate, partial sums accumulate in place, and the outside world sees only a trickle of input and output. This is why large matrix multipliers can sustain very high utilisation without saturating memory bandwidth. The price is rigidity — the dataflow is fixed in silicon, and an operator that does not fit the grid wastes most of it.

A tensor core relies on the memory hierarchy instead. Registers and shared memory are close and fast, but a tile still has to be loaded before each instruction sequence. The GPU hides that cost with enormous bandwidth and deep batched scheduling: while one warp waits for data, others compute. It works brilliantly when there is enough parallel work to hide the waits, and less well when a workload is small or irregular.

Part Three: Where you feel it in real life

Where this shows up is the price and battery life of the device answering your request. A data centre running a popular chatbot cares about throughput per watt more than anything else, because electricity and cooling are the recurring bill. There, a well-fed systolic array with near-perfect reuse can deliver more tokens per joule than a general GPU, which is why several accelerator companies bet the whole company on the systolic approach.

A developer prototyping a new model cares about a different thing: can the hardware run tomorrow's operator? Tensor cores, exposed through a mature programming model, will usually adapt with a library update. A systolic design may need a new chip. That flexibility is worth a lot when the software ecosystem is still moving fast.

The market reflects the split. General purpose GPUs with tensor cores dominate training and research because the workload keeps changing. Fixed systolic accelerators find a home in stable, high-volume inference where the model is frozen and efficiency is king. The global AI accelerator market is measured in tens of billions of dollars a year, and the contest is largely over this one question of data movement.

Flexibility costs energy and efficiency costs adaptability, so the right design is decided not by which multiplier is faster on paper but by how stable your workload is and how expensive your bandwidth is.

There is also a middle path that blurs the line. Some designs keep a tensor-core-like programmable front end but add a deep on-chip buffer and a semi-fixed dataflow behind it, trying to capture some reuse without freezing the operator set. Others use chiplets so a systolic tile can be swapped for a different accelerator on a future product. The boundary keeps moving as models stabilise.

What is actually transferable

The transferable lesson is that in modern computing, moving data costs more than computing on it. Any design that cuts memory traffic beats one that merely adds arithmetic, which is why reuse — not raw FLOPS — is the number to watch when comparing accelerators.

The second lesson is about the cost of specialisation. A fixed design is efficient exactly while the workload matches it. When the workload changes, flexibility is worth paying for in wasted joules. This trade-off appears everywhere from storage engines to spreadsheets.

A tensor core and a systolic array can multiply the same matrix to the same answer, but one asks how fast you can feed it and the other asks how long the data can stay, and the shape of your workload decides which question matters.

SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.