How Is an AI Token “Made”? The Five-Layer Cost Stack Explains the Real Price
A token may look like a tiny unit of text, but producing it requires electricity, advanced chips, data-center infrastructure, inference software, and an application capable of turning computation into useful work.
Have you ever wondered what happens after you type a sentence into ChatGPT, Claude, or Gemini and receive a polished response seconds later?
The model does not simply “think” and write. GPUs inside data centers perform a rapid sequence of calculations. Your prompt, the context the model reads, and the answer it generates are divided into tokens—the basic units used to measure both processing and API usage.
But the token price shown on a provider’s pricing page is not the same thing as the cost of producing that token.
Every Token Creates at Least Three Different Bills
1. The production bill: electricity, GPUs, servers, racks, networking, cooling, and maintenance.
2. The provider’s price: what a model company charges for input tokens, output tokens, cached input, and batch processing.
3. The task bill: the total cost of completing a job, including long context windows, tool calls, retries, and multi-agent coordination.
One million tokens used for a straightforward summary are not economically equivalent to one million tokens used for complex coding, deep research, or a multi-agent workflow. To understand why, it helps to examine the five layers behind token production.
Layer 1: Energy—the Raw Input of AI Computing
Token production begins with electricity. GPU servers, network switches, storage systems, and cooling equipment all require continuous power. A data center also needs grid connections, power-distribution equipment, backup systems, and operational redundancy.
In the United States, electricity costs vary by state, energy mix, utility contract, location, and scale. The electric bill also extends beyond the servers themselves: cooling and power delivery consume additional energy.
The industry commonly tracks this overhead with Power Usage Effectiveness, or PUE. A facility with a PUE of 1.2 consumes about 1.2 kilowatt-hours at the facility level for every 1 kilowatt-hour used by its IT equipment. The difference supports cooling, power conversion, lighting, and other infrastructure.
Even after those costs are included, electricity is usually not the most expensive layer. Power is the raw material; the costly part is the machinery and infrastructure required to convert it into dependable AI output.
Layer 2: Chips—Where Much of the Cost Accumulates
If electricity is the raw material, AI accelerators are the core machinery on the token production line. Training and serving large models may involve NVIDIA H100, H200, B200, or GB200 systems, Google TPUs, AWS Trainium, and other specialized hardware.
These systems also depend on high-bandwidth memory, or HBM, which enables rapid access to model parameters and intermediate results. The accelerator alone does not represent the full bill: server components, integration, deployment, maintenance, and supply constraints all matter.
Chips must also be depreciated. New hardware can deliver better throughput and energy efficiency, making an older accelerator economically less competitive even when it still works. Providers therefore spread capital costs across the hardware’s expected useful life and utilization rate.
Layer 3: Infrastructure—A GPU Cannot Work Alone
Buying GPUs does not create a functioning AI service. Accelerators must be installed in servers and connected through high-speed networks. A production cluster also requires:
- CPUs, system memory, and local storage;
- high-speed interconnects and optical modules;
- switches, cabling, racks, and power equipment;
- air or liquid cooling systems;
- buildings, security, monitoring, and redundancy;
- operations, maintenance, and incident-response teams.
Networking is especially important in large AI clusters. When many accelerators serve or train one model, they must exchange data quickly. A networking bottleneck leaves expensive chips waiting, lowers utilization, and raises the effective cost per token.
An AI data center is therefore better understood as a tightly coordinated computing factory than as a warehouse full of GPUs.
Layer 4: The Model and Inference Stack
Once the hardware is in place, software determines how efficiently it can be used. This layer includes model architecture, quantization, inference engines, routing, batching, caching, parallelism, and scheduling.
Two providers using comparable hardware can produce tokens at very different costs. One may batch requests efficiently and reuse cached context. Another may lose capacity to long contexts, poor scheduling, or communication delays.
- Quantized models can require less memory and compute, although quality may change.
- Mixture-of-experts models activate only part of the network for a given token.
- Speculative decoding can increase generation speed under the right conditions.
- Caching can prevent repeated computation on identical or reusable context.
That is why GPU count alone reveals little about a company’s cost advantage. The more useful metric is how many reliable, task-relevant tokens the system produces from its available hardware.
Layer 5: The Application—Users Pay for Outcomes
At the top of the stack, the important question changes from “What does one token cost?” to “How many tokens does it take to complete the job?”
A simple answer may require only a short prompt and response. A more advanced workflow can involve:
- a long context window;
- multiple reasoning passes;
- web searches and document retrieval;
- code execution and external tools;
- multi-agent review and verification;
- automatic retries after errors.
An agent that repeatedly rereads the same material or asks several models for nearly identical answers can make a low token price expensive at the task level. A well-designed system can lower costs by selecting the right model, reusing cached context, limiting unnecessary history, and stopping once it has a sufficiently reliable result.
Why Token Price Is Not Token Cost
Published API pricing is a market price, not a direct display of physical production cost. It may need to cover:
- hardware and server depreciation;
- electricity, cooling, networking, and facilities;
- model training and research;
- engineering, sales, support, security, and compliance;
- idle capacity reserved for traffic spikes;
- profit margins—or temporary strategic subsidies.
Input, output, and cached tokens also have different economics. Output tokens are generally generated sequentially and can be more expensive to serve. Input can often be processed more efficiently, while cached input avoids some repeated computation.
Prices across models and providers are therefore not perfectly comparable. A cheaper model may offer lower quality, slower performance, or a smaller context window. A premium model may still be wasteful if it is used for work a smaller model could handle.
The Number That Matters Is Cost per Useful Task
The five-layer stack reveals a simple reality: a token is not a digital product created from nothing. Electricity sits at the bottom; chips, infrastructure, and inference engineering occupy the middle; useful customer outcomes sit at the top.
Evaluating an AI company’s cost advantage requires more than comparing price per million tokens. It also means asking:
- How high is accelerator utilization?
- How efficient is inference?
- How effectively are batching and caching used?
- How many tokens does a real task consume?
- Does the output create measurable value?
- Is a low price sustainable, or subsidized?
Note: Hardware prices, cloud rates, energy costs, utilization, and API pricing vary by provider, geography, workload, and contract. Figures and examples should be treated as illustrative rather than universal.