A particular kind of AI business advice has become common in Chinese-language media: rent a GPU server, run a few deployment commands, expose an API, and start selling tokens.
The opportunity is presented as accessible to “ordinary people.” That phrase is doing far too much work.
Inference is becoming a large market. There is nothing unreasonable about building a business in it. The problem is the jump from “there is demand for inference” to “a person with a colocated server has a viable business.” Capital, customers, production engineering, support, security, and regulatory duties disappear from the story. The server remains because it is the easiest part to sell to the reader.
Owning a truck does not create a logistics company. It creates a vehicle that starts costing money today.
Three different businesses share one label
“Selling tokens” can describe at least three businesses:
- Running an open-weight model on owned or rented GPUs and selling inference through an API;
- Buying upstream model APIs, then aggregating, routing, billing, or reselling access;
- Building model capability into a product and charging for a completed task or workflow.
The first is an infrastructure-heavy inference service. The second is a channel and platform business. The third sells an application.
Their capital requirements, technical risks, customer promises, and reasons to exist are different. Collapsing all three into one phrase makes a server look like the entrance to revenue. It is usually the entrance to fixed cost.
Some promotional articles really do reduce the plan to renting a GPU, running a few commands, supporting many models, and waiting for revenue. At WAIC 2026, companies describing themselves as token factories were talking about unified APIs, model routing, protection against compute-supply volatility, and enterprise delivery. The industry opportunity is real. The simplified side-hustle version leaves out the business that makes it work.
A data center is an infrastructure project
Even a conventional data center is not a room with a few racks.
Uptime Institute’s certification scope covers electrical and mechanical systems, on-site power, physical security, equipment maintenance, fire protection, and capacity management. A site also needs planned power sources, network paths, cooling, cabling, and safety controls.
Those systems exist because power fails, cooling equipment needs maintenance, network paths break, and customer workloads are expected to keep running.
AI clusters add higher power density and demanding interconnects. A multi-node deployment may involve high-speed adapters, switches, optics, cabling, RDMA, congestion control, and performance isolation. This is a system, not an expensive network card attached to a workstation.
Building an IDC is not a casual path into AI entrepreneurship. It is a capital and operations project.
Colocation removes only part of the problem
Colocation is a reasonable alternative to building a facility. Depending on the contract, it may provide physical space, power, cooling, and network connectivity.
It does not automatically provide:
- Model deployment, quantization, upgrades, and rollback;
- Compatibility across GPUs, drivers, runtimes, and inference engines;
- Capacity planning, routing, rate limits, and load balancing;
- Monitoring, alerts, logs, backup, and recovery;
- API credential management, tenant isolation, abuse control, and billing;
- Customer support, refunds, service commitments, and overnight incident response.
Open-source software can make a model answer a request. It will not take a support call or explain an outage to a paying customer.
Colocation lowers the facility barrier. It does not turn inference into passive income.
Fitting the weights is only a startup test
Many informal plans begin with a weight calculation. Take a model, apply 4-bit quantization, estimate the memory required, and choose enough GPUs to load it.
That proves the model may start. It does not establish useful serving capacity.
During generation, an LLM keeps attention state for previous tokens so it does not recompute the entire sequence at every step. That memory is commonly called the KV cache. Longer inputs, longer conversations, and more active sequences generally require more cache. The exact amount depends on the model architecture, precision, and inference engine, so there is no honest universal answer for “users per GPU.”
NVIDIA’s NIM configuration guide gives a useful example. On an 80 GB GPU with 60 GB occupied by model weights, its sample allocation leaves 12 GB for KV cache and 8 GB for intermediate results.
That gap is where the difference between loading a model and serving a workload begins.
Quantization can reduce weight memory and some compute cost. It does not make context free, remove queues, or let a hundred unrelated requests share one KV cache.
One hundred customers are not one hundred concurrent requests
The opposite exaggeration is also unhelpful. Moving from one user to one hundred does not produce a fixed orders-of-magnitude increase in required hardware.
One hundred registered users may send a handful of short requests across a day. They may also upload long documents within the same minute. Capacity depends on arrival rate, input and output lengths, latency targets, model behavior, batching, and cache reuse.
A public service needs to measure more than tokens per second from a single interactive session:
- Queue time before a request receives compute;
- Time to first token;
- Delay between generated tokens;
- Aggregate input and output throughput;
- KV cache utilization;
- Error and timeout rates under load.
The official vLLM Production Stack includes routing, distributed deployment, KV-cache management, monitoring, and autoscaling. Its metrics documentation tracks running and waiting requests, GPU cache use, token throughput, prefix-cache activity, and queue time.
A model producing an acceptable response for one person says very little about the service customers will receive during a burst.
The only useful capacity test uses a realistic request distribution. Without context lengths, output lengths, arrival rates, and service targets, a concurrency number is marketing.
The customer question comes before the GPU question
Consumer users already have free or inexpensive chat products. Professional users are not shopping for an unlabeled bucket of tokens.
Developers choosing Claude, Codex, or Cursor care about repository understanding, file edits, terminal execution, tools, version control, and the experience around the model. Teams add identity, permissions, audit, privacy, centralized billing, and support. Cursor’s product plans, for example, bundle model access with agent features, team administration, usage analytics, privacy controls, and enterprise security capabilities.
API buyers can also purchase directly from model providers. DeepSeek’s official API pricing distinguishes cache hits, cache misses, and output tokens while providing provider-scale concurrency. The exact prices will change. That is part of the competitive risk: an upstream price update can invalidate a small operator’s unit economics overnight.
Why would a customer move a production workload away from those options and onto a server maintained by one person?
“It speaks the OpenAI API format” is not differentiation. A customer may pay for lower regional latency, a specific model, data-residency requirements, reliable capacity, industry integration, or ownership of the delivered result. Without one of those advantages, the remaining lever is usually price.
Price is where the smallest buyer of hardware and electricity is least likely to win.
Fixed cost starts before demand
Hardware, colocation, power, network service, and labor accrue with time. Customers pay only when they send billable work.
An idle GPU still costs money. Spare capacity reserved for an evening burst may sit unused all afternoon. A new model may leave the hardware operational but commercially unattractive. During an outage, customers do not relax their recovery expectations because the provider is a small side business.
Simplified cost models often measure electricity and divide a short benchmark by one million tokens. The result looks precise while leaving out the questions that determine whether the business survives:
- What percentage of each month produces billable tokens?
- How much headroom is reserved to meet peak latency targets?
- What is the mix of input, output, and cache hits?
- Who pays for retries, failed requests, abuse, and promotional credits?
- How are hardware, colocation, bandwidth, tax, and labor amortized?
- What happens to the margin when a model provider cuts its price?
Without signed demand and representative load, a detailed spreadsheet can become a justification for buying hardware rather than a forecast.
Public access creates public responsibilities
Running a model for personal use and charging the public through an API carry different responsibilities.
China’s Interim Measures for the Management of Generative AI Services apply to services that provide generated text, images, audio, or video to the public in China. The definition of a provider includes organizations and individuals offering the service through an API.
The Measures for Labeling AI-Generated Synthetic Content, effective since September 1, 2025, cover visible and embedded labels, user agreements, logs, and related filing materials.
Whether a specific deployment requires a filing, security assessment, or other controls depends on its users, model, functionality, and delivery method. This article is not a substitute for legal review.
It is still clear that exposing a port and adding payments does not complete the risk assessment.
Small operators can still have a viable case
None of this means an individual or small team can never run a commercial inference service.
The conclusion changes when:
- Identified customers already have documented model, context, traffic, and latency needs;
- Privacy, location, latency, model choice, or industry delivery creates a real advantage;
- Existing equipment has usable spare capacity and a controlled marginal cost;
- Customers pay for a concrete outcome while tokens remain an internal cost;
- Rented compute or an upstream API validates demand before hardware is purchased;
- The operator can own upgrades, incidents, refunds, abuse, security, and compliance.
In that situation, infrastructure is a resource required by an existing business.
Buying the machine first, signing a colocation contract, and then searching for token buyers reverses the order. It creates perishable inventory with a monthly bill.
Eight questions to answer before buying hardware
Write down the answers before committing capital:
- Who are the first customers, and have they expressed a willingness to pay?
- Why will they not use an existing assistant, specialized product, or direct model API?
- What are peak concurrency, input-length, and output-length distributions?
- What queue time, time to first token, and recovery time will customers accept?
- What takes over when a model upgrade, GPU fault, or network outage occurs?
- Can the business survive several months of low utilization?
- Who handles overnight alerts, support, refunds, and abuse?
- Who owns filing, content-safety, data, and logging duties for public access?
Some estimates will be rough at the beginning. That is normal. If “Who is the customer?” can only be answered later, a long-term colocation contract should also wait.
The practical opportunity is above raw tokens
There will be successful token providers. Inference demand is real, and new suppliers will enter the market.
The misleading part is presenting a capital-intensive, operations-heavy service as a low-cost side hustle for ordinary people. The reader takes the hardware and cash-flow risk. The author takes responsibility for a headline.
A more practical path is to find a problem for which someone will pay and use existing models to deliver the result. Once customers, traffic, and constraints are known, self-hosting can be evaluated as a way to lower cost or meet requirements that existing services cannot satisfy.
Tokens are a unit of measurement. They are not customer demand.