Energy Stack Behind an AI Token

How to minimize the physical work required to produce a useful token?

The dominant architecture for AI inference today is to:

• store a large neural network in memory

• move its parameters into compute units

• execute billions of multiply-accumulate operations

• repeat the process for every generated token.

It works remarkably well. But also it’s extravagant energy-wise. So how do we look at the next generation of AI infra? It starts by looking at the following two questions:

1. What is the least amount of energy required to produce a useful token?

2. What is the least amount of model state that must be activated to produce it?

These questions lead to a much broader market than “we just need better GPUs”.

The answer could come from smaller models, conditional computation, lower precision, compute-in-memory, analog circuits, photonics, spiking neural networks, or software that avoids unnecessary work. It’s likely that it will come from combining several of them.

What’s the cleanest metric to measure this?

Raw joules per token is insufficient.

Why is it insufficient? Because a tiny model can consume little energy while producing useless output. If you want to compare, then you must hold the following things constant: model quality, task difficulty, context length, latency, and service level.

Keeping this in mind, the cleanest metric is perhaps:

joules per token per quality per task per context-length per latency per SLA

Rolls right off the tongue, doesn’t it? Maybe a better way to put it would be:

Let’s keep a bunch of things constant and THEN measure joules per token. This would be a fair comparison. And it would form a Pareto frontier where you can pick and choose a config depending on what you need.

A complete efficiency benchmark should therefore report:

• Joules per generated token

• Quality / accuracy at that energy level

• Input and output sequence length

• Time to first token and inter-token latency

• Batch size and hardware utilization

• Memory traffic per token

• Active parameters, operations or state transitions per token

Energy consumption can vary dramatically even on the same hardware. For example, a simple thing like task selection can produce energy differences of up to 25x. So “energy per token” is a property of the entire workload + system as opposed to just the model or chip.

The second metric that’s directionally useful is neurons activated per token. But it’s technically difficult.

A dense Transformer neuron, an MoE expert, an analog crossbar cell and a neuromorphic spike are not equivalent units of computation. So a better generalized metric is:

How many physical state transitions, parameter accesses, and arithmetic operations are required per useful token?

This captures the underlying objective of “how do we touch less of the machine to produce a token?”.

What’s the Theoretical Lower Bound?

There is a fundamental thermodynamic cost to irreversible computation. It’s called Landauer’s principle. And it states that at room temperature, erasing one classical bit requires at least 2.9 x 10^-21 joules. The principle has been experimentally demonstrated. Can’t go lower than this!

But this does not give us a meaningful “minimum energy per token” measure. Why? Because the real world is messy and we have to account for many other factors.

A generated token may encode several bits of information. But producing it requires storing state, moving signals, suppressing noise, correcting errors, synchronizing components and interacting with the outside world. Current computers operate many orders of magnitude above the Landauer limit.

There is similarly no universal minimum number of neurons required to produce a token.

A predictable token may require almost no reasoning. For example, I can always ask the system to output “cat” no matter what the user is asking. I’ll be super energy-efficient with this system but almost always produce the wrong answer.

A difficult mathematical step may require substantial computation. But the output will be very valuable.

So the “minimum” is conditional on the task, model quality, context and acceptable error rate. “Can inference reach the thermodynamic limit?” is not a useful question. It’s more along the lines of:

Which architectural path can eliminate the largest amount of unnecessary computation and data movement while preserving intelligence?

Okay so what avenues do we have to attack this problem?

There are 5 big ones we’ll discuss here:

1. Algorithm and Model Level

The first layer attacks the amount of work before changing the underlying machine.

Smaller models: Distillation, pruning, low-rank methods, weight sharing, and quantization reduce model size and arithmetic cost. The larger opportunity is to train models (e.g, binary, ternary networks) designed for much simpler hardware from the start.

Conditional computation: Most models spend roughly the same compute on every token. More efficient systems route easy tasks through less compute and reserve expensive models for harder ones.

Activation sparsity: Only a fraction of neurons may matter for a given token. Capturing the energy benefit requires co-design across models that produce structured sparsity, software that detects it cheaply, and hardware that can skip inactive computation.

Fewer model passes: Use techniques (e.g. speculative decoding, KV cache compression) to reduce the number of sequential inference steps.

The core idea is to use less model, activate less of it, and run it fewer times.

2. Architecture and Model Structure

The next category changes the structure of the model itself.

Sparse Transformers and MoE: Activate only selected experts, neurons, or attention connections. The benefit depends on keeping routing, communication, and memory movement low. Or else sparsity only shifts the cost.

State-space and recurrent models: Compress prior context into a fixed-size state instead of repeatedly reading a growing KV cache. This can lower inference cost, but may lose information that attention preserves.

Spiking neural networks: Compute only when discrete events occur. This works naturally for sparse sensory data, but language is not inherently event-driven. The larger opportunity is models trained natively for neuromorphic hardware (as opposed to converting transformers post facto).

Stochastic computation: Use stochastic computation instead of calculating every intermediate value precisely. They are most compelling where sampling, uncertainty, or approximate inference is already part of the task.

The core idea here is to build models around cheaper computational primitives rather than forcing every workload through dense, precise arithmetic.

3. Compute Paradigms

This layer attacks how the computation must happen. These approaches reduce the energy cost of moving and processing model weights.

Low-precision digital compute: INT8, INT4, FP8, and ternary arithmetic reduce costs while remaining compatible with existing chips and software. Mostly around memory, bandwidth, and compute. But weights still move between memory and processors, so the memory bottleneck remains.

Approximate computing: Neural networks can tolerate some numerical error. Hardware can therefore use lower precision or simpler arithmetic where exact computation does not change the answer.

Compute-in-memory: Perform matrix operations inside or next to memory rather than repeatedly moving weights into a processor. This could deliver major efficiency gains.

Event-driven compute: Process information only when the system’s state changes. This is well suited to sparse sensors and robotics, but requires models designed around sparse events rather than dense transformer execution.

Photonic compute: Use light for matrix operations and data movement. Photonics offers high parallelism and low latency. But the full system (including lasers, memory, conversion, and control) must also be efficient.

The core idea is to move fewer bits, use cheaper arithmetic, and eliminate data movement.

4. Hardware Architectures

This deals with different types of hardware we can use. Different chips trade flexibility for efficiency.

GPUs: Highly programmable with mature software, but carry overhead from supporting many workloads.

Specialized accelerators: Optimize tensor operations, memory movement, and dataflow for higher efficiency.

FPGAs: Sit between GPUs and custom chips. More adaptable than ASICs, but less efficient.

Workload-specific chips: LLM chips optimize decoding, quantization, KV-cache access, and expert routing. Neuromorphic chips target event-driven models. Photonic chips target communication and matrix operations. In-memory chips eliminate movement between storage and compute.

The core idea is to specialize the hardware around the model’s dominant operations.

These categories will likely converge. Future inference systems may combine digital control, sparse SRAM compute, analog matrix engines, and optical interconnects in one package.

5. System and Stack Level

A theoretically efficient chip can still produce an inefficient token.

Transformer inference has two different workloads:

1. Prefill: Processes the prompt in parallel and is usually compute-heavy.

2. Decoding: Generates tokens one at a time and is often limited by repeatedly moving model weights and KV cache data.

System design is therefore as important as chip speed. The main levers are:

• More on-chip memory

• KV cache compression

• Operator fusion

• Better batching and scheduling

• Power gating and dynamic voltage control

• Compilers that exploit sparsity

• Faster lower-energy packaging and interconnects

The exact energy breakdown varies by workload. But broadly, prefill is constrained by computation and decoding is constrained by data movement.

How Much Energy Does a Token Consume Today?

There is no single answer.

Recent GPU measurements show generated-token energy ranging from roughly hundreds of millijoules for smaller models to substantially higher levels for larger models and inefficient configurations.

It’s highly workload-dependent. The correct hierarchy is:

1. Energy per primitive operation

2. Operations and memory accesses per model step

3. Model steps per generated token

4. Generated tokens required to solve the task

Optimizing only the first layer can miss most of the system.

How Few Neurons Can Produce a Token?

No such thing. Technically you can use 0 neurons to produce a token. Obviously it will be a totally useless token, but it will be a token nonetheless.

Ok so what ranges are we looking at here?

• Dense models activate nearly their entire computational graph for every token

• MoE models reduce the number of active experts

• Structured sparsity reduces active channels

• Early-exit systems reduce active layers

• Recurrent models reduce the amount of historical state revisited

• Spiking systems reduce activity to discrete events.

The theoretical endpoint is not necessarily a model with fewer total parameters. It may be an enormous model that activates only a tiny, context-specific pathway for each token.

A plausible minimum-energy architecture would combine: a small persistent recurrent state, sparse retrieval of relevant external memory, a dynamically selected expert, low-bit computation, event-driven local updates, and a larger fallback model used only under uncertainty.

This would resemble a nervous system more than a conventional dense computer. It would have large total capacity, highly local memory, and very sparse activity.

Mapping the Path Toward Minimum Energy

The architecture landscape is not a sequence in which one technology replaces another. All these techniques will likely end up stacking on top of each other.

The progression is therefore better understood as 5 simultaneous reductions:

1. Reduce model work: smaller models, distillation and better architectures.

2. Reduce active work: routing, sparsity and early exits.

3. Reduce numerical work: quantization and approximate computation.

4. Reduce data movement: locality, compute-in-memory, and compressed state.

5. Reduce wasted system capacity: batching, scheduling, power management, and compilers.

So instead of achieving the minimum in any single category, the winning system will minimize the product of all five.

Key Trade-offs

Every route to lower energy introduces a constraint.

Lower precision can reduce accuracy. Sparsity creates irregular execution. MoE routing creates communication overhead. Analog systems introduce noise and calibration. Neuromorphic systems require new training methods. Photonics struggles with memory and electronic interfaces. Specialized ASICs sacrifice programmability. Aggressive batching improves energy efficiency but increases latency.

So when you design a system, you just choose the set of trade-offs you can live with.

What does the market look like today?

1. Model layer

• Liquid AI: Small, deployment-oriented Liquid Foundation Models

• Neural Magic: Quantization and sparsity for running open models on commodity hardware

2. Architecture layer

• AI21 Labs: Jamba hybridizes attention, state-space layers, and MoE

• Cartesia: Develops Mamba-based state-space models for efficient long-context inference

• BrainChip: Event-driven Akida neuromorphic architecture

• SynSense: Spiking neural networks and dynamic vision processing

• Innatera: Combines spiking, CNN, and RISC-V architectures

• Extropic: Thermodynamic architecture for probabilistic sampling

3. Compute layer

• EnCharge AI: Capacitor-based analog compute inside SRAM

• d-Matrix: Digital in-memory compute for generative-AI inference

• Axelera AI: Digital in-memory compute for edge AI

• Rain AI: Digital in-memory compute with low-precision formats

• IBM: Analog PCM and RRAM research, alongside the Spyre accelerator

• Lightmatter: Photonic matrix computation and optical data movement

4. Hardware layer

• NVIDIA: Blackwell GPUs and low-precision tensor hardware

• AMD: Instinct accelerators, including the MI350 series

• Google: TPU accelerators, including Ironwood

• AWS: Trainium and Inferentia accelerators

• Meta: MTIA inference accelerators

• Microsoft: Maia inference accelerators

• Intel: Gaudi accelerators and Loihi neuromorphic chips

• Qualcomm: Cloud AI 100 inference accelerators

• Cerebras: Wafer-Scale Engine processors

• Tenstorrent: RISC-V-based AI processors

• MatX: Specialized silicon for large language models

• Etched: Generative-AI-specific inference silicon

• Sony: Event-based vision sensors

• Prophesee: Neuromorphic image sensors that capture pixel changes

• Lumentum: Lasers and optical components for AI networks

• Coherent: Lasers, transceivers, and precision optical components

• GlobalFoundries: Silicon-photonics manufacturing

5. System layer

• Modular: MAX runtime and Mojo programming stack

• CentML: Compiler optimization and runtime memory management

• Fireworks AI: Optimized inference serving, kernels, batching, and caching

• Astera Labs: PCIe and CXL connectivity infrastructure

• Marvell: Networking, interconnect, and custom data-center silicon

• Broadcom: Tomahawk switches and AI networking infrastructure

• Enfabrica: Accelerated Compute Fabric networking

• Eliyan: Chiplet interconnect without silicon interposers

• MemVerge: CXL-based memory pooling and orchestration

• Celestial AI: Photonic Fabric connecting compute directly with memory

• Ayar Labs: Optical I/O chiplets for chip-to-chip communication

Where is the Opportunity for Startups?

Some companies will reduce model size or active parameters. Others will build sparse inference engines, memory-centric accelerators, analog arrays, neuromorphic processors, or photonic interconnects. Another category will build the compilers, measurement systems, and runtimes required to make these architectures usable.

The highest-value companies may be those that connect multiple layers. So the central opportunity is perhaps model-system-hardware co-design.

The ultimate objective is an intelligent machine that knows when not to multiply, which memory not to access, which neurons not to activate, and which tokens not to generate.

That is the path toward the minimum-energy token.

---

If you are getting value from this newsletter, consider subscribing for free and sharing it with 1 infra-curious friend: