How Large Language Models Work: Training, Finetuning & Inference Explained

Large language models move through three distinct stages: training (where the model learns language patterns from billions of text examples), finetuning (where it's adapted for specific tasks or behaviors), and inference (where it actually responds to your prompts). Understanding these stages helps you evaluate AI tools, decide whether to use off-the-shelf or custom models, and make smarter decisions about when to finetune versus prompt-engineer. Think of training as building the foundation, finetuning as customizing the house, and inference as living in it daily.

What Is LLM Training and How Does It Work?

Training is the massive, expensive stage where a model learns the statistical patterns of language from scratch. Companies like OpenAI, Anthropic, and Meta feed billions of tokens (roughly 750,000 words equals 1 million tokens) through neural networks over weeks or months using thousands of GPUs.

During pre-training, the model learns to predict the next word in a sequence. If you show it "The cat sat on the", it learns that "mat" or "floor" are statistically likely next words. Repeat this process across trillions of examples, and the model builds an internal representation of grammar, facts, reasoning patterns, and even some common sense.

GPT-3 was trained on roughly 300 billion tokens. GPT-4's training set is undisclosed but estimated at several trillion tokens. This stage typically costs between $2 million and $100 million depending on model size, and it happens once. The output is a "base model" with broad language understanding but no specific instruction-following ability.

You can't run training yourself unless you're a well-funded organization. A single training run for a 70-billion parameter model requires around 1,000 GPUs running for several weeks. That's why most developers start with pre-trained base models like Llama 3, Mistral, or GPT-4 rather than training from scratch.

What Is Finetuning and When Should You Use It?

Finetuning takes a pre-trained base model and adapts it for specific tasks, domains, or behaviors. Instead of learning language from scratch, you're teaching the model to follow instructions, adopt a particular tone, or specialize in medical terminology, legal documents, your company's product knowledge.

There are several types of finetuning. Instruction tuning teaches models to follow user prompts (this is how ChatGPT learned to respond helpfully instead of just completing text). RLHF (reinforcement learning from human feedback) trains models to prefer certain responses over others based on human ratings. Domain-specific finetuning adapts models to specialized fields like radiology reports or financial analysis.

Finetuning is dramatically cheaper than training. You might use 1,000 to 100,000 examples instead of billions, and it takes hours or days instead of months. OpenAI's finetuning API lets you customize GPT-3.5 for around $0.008 per 1,000 tokens of training data. A typical finetuning job with 10,000 examples costs roughly $80 to $200.

The key decision point: use finetuning when you need consistent behavior across thousands of requests that prompt engineering can't reliably achieve. If you're processing 50,000 customer support tickets monthly and need specific formatting, tone, and routing logic, finetuning makes sense. If you're experimenting or handling varied requests, prompt engineering is faster and cheaper. You can learn more about cost-effective finetuning alternatives that deliver similar results.

What Happens During Inference When You Use an LLM?

Inference is the runtime stage where a trained model actually generates responses to your prompts. When you type a question into ChatGPT, Claude, or your company's AI assistant, inference is what's happening behind the scenes. The model takes your input, processes it through billions of mathematical operations, and produces output tokens one at a time.

Here's what makes inference different: the model's weights (the billions of learned parameters) are frozen. The model isn't learning or updating itself during your conversation. It's applying what it learned during training and finetuning to generate the most statistically probable response given your prompt.

Inference costs scale with usage, not upfront. GPT-4 charges roughly $0.03 per 1,000 input tokens and $0.06 per 1,000 output tokens. A typical conversation with 500 input tokens and 500 output tokens costs about $0.045. If you're running 100,000 queries monthly, you're looking at $4,500 in inference costs.

Latency matters during inference. Smaller models like GPT-3.5 generate roughly 40 to 60 tokens per second, while larger models like GPT-4 generate 20 to 30 tokens per second. This is why many production systems use smaller finetuned models for speed-critical tasks and larger models only when reasoning quality justifies the wait. Understanding how to run multiple models efficiently becomes critical when you're deploying AI at scale.

How Do Training, Finetuning, and Inference Work Together?

The three stages form a pipeline, and understanding their relationship helps you make better decisions about which AI tools to use and when to invest in customization.

Start with a concrete example: Meta trains Llama 3 on trillions of tokens (training). A healthcare company takes that base model and finetunes it on 50,000 medical transcription examples (finetuning). Doctors then use the customized model to generate patient notes from voice recordings (inference). Each stage builds on the previous one.

Here's the cost breakdown that matters for planning. Training happens once and costs millions. Finetuning happens per use case and costs hundreds to thousands. Inference happens per request and costs fractions of a cent to dollars. Your total cost of ownership depends on which stages you control.

Most small businesses and teams will never train a model. You'll choose between using pre-finetuned models (ChatGPT, Claude) or finetuning an existing base model (Llama, Mistral) for your specific needs. The decision hinges on volume and consistency requirements.

When to Use Pre-Trained Models Without Finetuning

Use GPT-4, Claude, or other instruction-tuned models directly when you're handling diverse requests, low to medium volume (under 10,000 queries monthly), or still experimenting with use cases. Prompt engineering gets you 80% of the way there for most applications.

Examples: content generation, research assistance, brainstorming, code debugging, customer support for small teams. You're paying for inference only, typically $50 to $500 monthly depending on usage.

When to Invest in Finetuning

Finetune when you need consistent formatting, specialized domain knowledge, specific tone or brand voice. Also consider it for cost optimization at scale (finetuned smaller models often outperform larger base models for narrow tasks while costing 70% less per query).

Examples: processing thousands of legal documents with specific extraction requirements, generating product descriptions in your brand voice, routing customer inquiries with company-specific logic. Upfront cost of $500 to $5,000, ongoing inference costs 30 to 70% lower than using GPT-4. If you're considering this path, training models on your specific writing style offers a practical starting point.

When Training From Scratch Makes Sense

Almost never, unless you're building a foundation model company, have unique data that provides competitive advantage, or need complete control over model behavior and safety. Or you operate in a domain where pre-trained models fundamentally don't work (extremely specialized languages, proprietary notation systems).

The threshold is roughly $10 million in available budget and a clear path to 100x return on that investment. Companies like Bloomberg trained their own models because financial data and terminology differ significantly from general web text.

What Are Common Misconceptions About How LLMs Work?

The biggest misconception: models learn from your conversations during inference. They don't. When you chat with ChatGPT, it remembers context within that conversation, but it's not updating its underlying knowledge or weights. Each new conversation starts fresh (unless you're using features like custom instructions or memory, which store preferences separately).

Another confusion point: finetuning versus retraining. Finetuning adjusts existing knowledge; it doesn't erase and rebuild from scratch. Think of it like teaching a fluent English speaker medical terminology versus teaching someone English from birth. The base language understanding remains intact.

People also conflate finetuning with RAG (retrieval-augmented generation). RAG doesn't change the model at all. It fetches relevant documents at inference time and includes them in the prompt. RAG is faster to implement and easier to update than finetuning, but it can't change the model's fundamental behavior or tone. For complex document scenarios, understanding how RAG handles visual elements becomes essential.

Finally, the idea that bigger is always better. A well-finetuned 7-billion parameter model often outperforms GPT-4 on specific tasks where it was trained with quality examples. Llama 3 8B finetuned on customer support data will typically beat GPT-4 on routing accuracy and response consistency while generating responses 3x faster and costing 90% less per query.

How to Evaluate AI Tools Using Your Understanding of These Stages

Now that you understand the three stages, you can ask better questions when evaluating AI tools or planning implementations.

For off-the-shelf tools: What base model are they using? (GPT-4, Claude, Llama?) Has it been finetuned for your use case? (Customer support models should be finetuned on support conversations, not just general chat.) What are the inference costs at your expected volume? Get specific numbers for 1,000, 10,000, and 100,000 queries.

For custom implementations: Can you achieve your goals with prompt engineering, or do you need finetuning? Test with 50 to 100 examples first. If finetuning, what's your data collection strategy? You need 500 to 5,000 quality examples minimum. What's your inference infrastructure? Cloud APIs, self-hosted, edge deployment?

For vendor claims: Be skeptical of "proprietary AI" that's just a finetuned version of an open-source model with 10x markup. Understand whether you're paying for training costs (you shouldn't be, that's amortized), finetuning value (reasonable if it's genuinely specialized), or just inference with markup (shop around).

What's the Practical Path Forward for Different User Types?

If you're learning AI: Start by using ChatGPT or Claude extensively to understand inference behavior. Then experiment with prompt engineering to see how far you can push instruction-tuned models. Finally, try a simple finetuning project using OpenAI's API or Hugging Face's tools with 1,000 examples to understand the difference.

If you're implementing AI for work: Map your use cases to the stages. High-variety tasks (research, brainstorming, general assistance) stay at inference with good prompts. High-volume, consistent-format tasks (data extraction, classification, generation with specific rules) justify finetuning. Calculate the breakeven point: if finetuning costs $2,000 and saves $0.02 per query versus GPT-4, you break even at 100,000 queries.

If you're running a business: Build your AI strategy around inference costs and finetuning ROI, not training. Partner with vendors who use quality base models and can demonstrate finetuning value. Budget for iteration because your first finetuned model won't be perfect. Plan for 3 to 5 finetuning cycles at $500 to $2,000 each to reach production quality.

Look, here's the mental model that matters: training builds the foundation (you'll almost never do this), finetuning customizes for your needs (do this when volume and consistency justify it), and inference is where you spend most of your time and money (optimize this relentlessly). Understanding these three stages transforms AI from a black box into a system you can evaluate, budget, and implement strategically. Most AI implementation failures come from confusion about which stage solves which problem, so getting this framework right is genuinely the 80/20 knowledge that makes everything else make sense.