A guide to the real capabilities of large language models — what they're good at, what they fail at, the technology behind them, and the specific tasks and fields they're reshaping. Written for people who want intuition, not hype.
In early 2023, AI could answer trivia and write passable essays. In February 2026, it fixes 80% of real software bugs, answers PhD science at 94%, solves olympiad math, reads million-token codebases, and chains tools together for hours-long autonomous work. The following timeline maps not products but abilities — the moment each capability crossed from "not really" to "surprisingly yes."
Software engineering is where AI capabilities are most measurable, most adopted, and most economically consequential. It's also where the gap between hype and reality is most documented.
A frontier model, given access to the Django repository, will: navigate to the relevant migration files, identify the timezone handling bug in django/db/backends/postgresql/operations.py, write a patch that preserves existing behavior while fixing the conversion, and run the test suite to verify.
Claude Code scaffolds the OAuth flow, creates migrations, implements callback handlers, writes integration tests, and iterates when tests fail — often over 15-30 minutes of autonomous work.
At a Seattle meetup in January 2026, over 150 engineers packed a room to trade Claude Code use cases. One Google principal engineer publicly said Claude reproduced a year of architectural work in one hour. Microsoft — which sells GitHub Copilot — has widely adopted Claude Code internally across major engineering teams. Boris Cherny, Claude Code's creator, used multiple Claude agents running in parallel to ship 300+ pull requests in December — his most productive month in a year and a half at Anthropic.
AI coding tools fail predictably on large-scale architectural decisions (choosing between microservices and monolith), ambiguous requirements (when the bug report is incomplete), cross-system coordination (changes spanning multiple services), and subtle security bugs — researchers flag concerns about AI-introduced vulnerabilities in complex code. The remaining ~20% failure rate on SWE-bench concentrates in exactly these areas: problems that challenge experienced engineers too.
MMLU — 57 academic subjects from high school to professional — went from random guessing (2020) to saturated (2024) in four years. Its harder successor, GPQA Diamond, uses PhD-level questions designed to be un-Googleable: non-experts with internet access get 34%, domain PhD holders get ~65%. By February 2026, three labs score above 91%. The remaining errors likely reflect ambiguous questions more than knowledge gaps.
This single pair of facts tells the whole story of where AI is: LLMs correctly identify medical conditions in 94.9% of test scenarios when tested alone. But in a randomized study of 1,298 people, participants using an LLM identified conditions in fewer than 34.5% of cases — no better than people without LLMs at all.
Where LLMs are working in medicine: clinical documentation (reducing charting burden on doctors), medical education (generating realistic case studies for rare conditions — one NYU program uses them to simulate diagnoses residents rarely see in training), clinical trial enrollment (matching patients to studies), and radiology assist. Where they're not yet reliable: direct diagnosis, treatment planning, and anything where a wrong answer has immediate physical consequences.
When an LLM doesn't know something, it doesn't say "I don't know" — it generates a confident, plausible-sounding answer that may be completely fabricated. On questions flagged as uncertain by internal confidence scores, Gemini 3.1 Pro's factual error rate dropped from roughly 88% to 50% between model versions — real progress, but a coin flip on uncertain questions remains the reality. This is the single biggest barrier to high-stakes deployment, and the reason every AI-generated claim in medicine, law, or engineering still requires human verification.
The biggest surprise in AI over the past two years wasn't that models got bigger. It was that they learned to think — or at least, to do something that produces the same outputs as thinking.
François Chollet designed ARC (Abstraction and Reasoning Corpus) specifically to test whether AI could reason about things it had never seen before — visual puzzles requiring fluid intelligence, not memorization. For four years, it worked. The best models were stuck at 0–5%.
Then, in December 2024, o3 scored 87.5%. Not through memorization — through iterative reasoning at test time. The model generates chains of internal thought, evaluates them, backs up when they fail, and tries new approaches. ARC-AGI-2 was designed to be even harder. Within six months, Gemini 3.1 Pro had already reached 77.1%. Humans score ~95%.
When a "reasoning model" (o1, o3, Claude in thinking mode) tackles a problem, it doesn't just predict the next token. It generates sometimes thousands of internal tokens — exploring approaches, checking its work, reasoning through implications — before producing a response. You trade speed and cost for accuracy: a reasoning query might take 30–60 seconds and cost 10× more than a standard query, but it can solve problems that standard models fail at entirely.
GPT-4o scored 9.3% on the International Mathematical Olympiad qualifier in May 2024. Four months later, o1 scored 74.4%. By late 2025, multiple models achieved 100% on AIME. The entire arc — from barely functional to solved — took about 16 months. But FrontierMath — unpublished research-level problems — sits at roughly 40%. "Can solve hard problems with known answers" and "can do math that hasn't been done before" remain very different capabilities.
The shift from "assistant" to "agent" is arguably the most consequential development of 2025–2026. An assistant answers your question. An agent takes your goal, breaks it into steps, uses tools, handles errors, and delivers a result.
"Summarize the key points from this research paper." You copy-paste relevant sections into a chat window. The model summarizes what you gave it. If you want more, you paste more. Context limit: a few pages. You are the retrieval engine.
"Find the 5 most-cited protein folding papers from 2025, summarize their contributions, and identify which ones build on AlphaFold." The model searches the web, reads full papers, cross-references citations, and delivers a structured synthesis. 15-20 tool calls over 2-3 minutes. You define the goal; it executes.
Models with "computer use" capability can take screenshots, identify UI elements, click buttons, fill in forms, and navigate multi-step workflows — operating a computer the way a human would.
Agency requires memory. Context windows grew from 4,000 tokens (a few paragraphs, 2022) to over 1 million tokens (an entire codebase, 2026). But advertised capacity and usable quality diverge sharply: at 1 million tokens, Claude Opus 4.6 retrieves specific information 76% of the time while Gemini 3 Pro manages only 26%. The "usable context" is often far smaller than the spec sheet.
Current agentic systems fail on long sequences of dependent steps (error rates compound — 95% success per step becomes 60% after 10 steps), ambiguous goals (they do exactly what you said, not what you meant), and recovery from unexpected states (a dialog box they didn't expect can derail an entire workflow). The SWE-bench 80% success rate is for single, self-contained tasks. Multi-hour, multi-tool workflows are far less reliable.
This page has focused on text and code — where capabilities are most measurable — but frontier models in February 2026 are natively multimodal. They see, hear, and increasingly generate across media types.
Vision: Upload a photo of a rash and ask for a differential diagnosis. Photograph a math problem from a textbook and have it solved step-by-step. Share a screenshot of an error message and get debugging help with full context. Screenshot a whiteboard of architecture notes and have it converted to a structured document. Gemini leads on multimodal understanding benchmarks (MMMU-Pro); all three frontier labs support image input natively.
Voice: GPT-4o introduced real-time voice conversation in 2024 — low-latency, emotionally expressive, interruptible. This shifted the interface from typing to talking for a subset of use cases: hands-busy workflows, accessibility, language practice, and brainstorming. Claude and Gemini followed with voice interfaces in 2025.
Image and video generation: Distinct from LLMs but increasingly integrated with them. Text-to-image (DALL-E 3, Midjourney, Flux) is production-grade for marketing, concept art, and prototyping. Text-to-video (Sora, Veo) is emerging but not yet reliable for professional use. The quality ceiling rises monthly.
Document understanding: Models can now read PDFs, spreadsheets, slides, and handwritten notes natively. A common workflow: upload a 200-page contract, ask "what are the non-standard termination clauses?" and get a structured answer with page references. The accuracy depends heavily on document quality and model — it works well for clean typeset documents and poorly for low-resolution scans.
Now that you've seen what these systems do, here's how they do it. Every major AI model — ChatGPT, Claude, Gemini, Llama — is built on the same fundamental architecture: the transformer, invented at Google in 2017.
A language model is a function that takes a sequence of words (or "tokens") and predicts the next one. That's it. The remarkable thing is that, trained at sufficient scale on enough text, this simple objective produces emergent capabilities — reasoning, translation, code generation, mathematical proof — that weren't explicitly programmed.
The transformer's key innovation is the attention mechanism — it allows every word in a sequence to attend to every other word, weighted by relevance. When processing "The bank was steep along the river," attention connects "bank" to "river" rather than to "money." At scale, billions of these attention computations produce something that looks remarkably like comprehension — though whether it is comprehension remains an open question.
Each major jump in capability traces back to one of these breakthroughs:
| Paradigm | When | What Changed | In Plain English |
|---|---|---|---|
| Scale | 2020–23 | More parameters + data = predictably better | A bigger brain learns more |
| RLHF | 2022–23 | Human feedback makes models usable | Training the brain to be helpful, not just smart |
| Mixture of Experts | 2023–24 | Only part of the model activates per query | A team of specialists, not one generalist |
| Test-time compute | 2024+ | Model "thinks longer" before answering | Taking 30 seconds on a test question instead of 3 |
| Agentic tool use | 2025+ | Models browse, code, and act iteratively | Not just answering — doing |
In January 2024, the best open-source model trailed the best proprietary model by 8% on Chatbot Arena Elo. By February 2025, the gap was 1.7%. DeepSeek R1 — open-source, from a Chinese lab — matched frontier reasoning at roughly 90% less cost. Meta's Llama models run locally on consumer hardware. This isn't just a pricing story; it's an access story. The capabilities described on this page are increasingly available to anyone with a laptop, not just organizations with API budgets.
Stanford HAI's 2026 predictions converge on a theme: the era of AI evangelism is giving way to an era of AI evaluation. The question has shifted from can AI do this? to how well, at what cost, and for whom?
Claude Code: $2.5B run-rate. Microsoft uses Claude Code internally despite selling Copilot. 40-70% dev time reduction on standard features at early adopter companies.
MASAI trial: AI-supported mammography in 105,934 Swedish women. MedHELM: 120+ clinical scenarios benchmarked across 6 models.
GPT-4 passed the bar at the 90th percentile (2023). Real-world legal AI adoption focuses on document-heavy workflows, not advice.
NYU uses LLMs to generate clinical cases for rare conditions residents seldom see. Shen & Tamkin (2025): "How AI Impacts Skill Formation."
Epoch AI: task complexity doubling every 5-7 months. But researchers warn AI use may reduce code verification rigor.
Book releases tripled post-LLM; average quality declined. But top-1000 monthly showed higher quality than pre-LLM (Reimers & Waldfogel, 2026).
The economic story may be more consequential than the capability story. Capabilities that exist at $100 per query are research curiosities. Capabilities at $0.001 per query reshape industries.
Two overlapping trends are driving prices down. First, the frontier itself got cheaper: each new model version costs less per token than the last — roughly 7× cheaper at the same capability tier over three years. Second, smaller models caught up to what the frontier used to be — GPT-4o mini ($0.38/M tokens) roughly matches GPT-3.5 from 18 months prior. If you count the capability of a 2023 frontier model delivered at 2026 commodity prices, the effective decline is 100× or more. These are overlapping trends, not a single curve.
| If You Need... | What It Costs (Feb 2026) | What It Cost 2 Years Ago |
|---|---|---|
| GPT-3.5-level chat | Free (Llama 3.1 8B, local) | $2/M tokens |
| GPT-4-level analysis | $0.55/M (DeepSeek V3, open-source) | $30+/M tokens |
| Frontier coding (80% SWE-bench) | $8–18/M (Gemini 3.1 Pro, Opus 4.6) | Didn't exist at any price |
| Maximum reasoning (PhD science, math) | $58/M (GPT-5.2 Pro) | Didn't exist at any price |
The consequence: AI capabilities that were research demos two years ago are accessible to any developer with a credit card, and many users for free. The gap between "what AI can do in a lab" and "what AI can do for you" has collapsed.
The failures are as instructive as the successes. These aren't edge cases — they're systematic limitations that follow directly from how these systems work.
When an LLM encounters something it doesn't know, it doesn't say "I don't know." It generates a confident, plausible-sounding answer that may be completely fabricated — a citation to a paper that doesn't exist, a coherent account of an event that never happened. This is called hallucination, and it's the defining limitation of the technology. On questions where models are uncertain enough to hedge, frontier systems still get it wrong roughly half the time. Closely related: they express the same confidence whether right or wrong. When humans are uncertain, we usually recognize it. LLMs cannot reliably do this. This makes them dangerous in domains where knowing the limits of your knowledge — medicine, law, engineering safety — matters more than the knowledge itself.
LLMs process text as tokens, not as grounded concepts. Ask "how many r's in strawberry?" and it may get it wrong — even as it solves differential equations. Ask it to predict what happens when you push a stack of blocks, and it struggles — it learned from text, not from interacting with the physical world. Despite million-token context windows, performance degrades over extended interactions; a model that's brilliant in minute one may subtly drift by minute thirty, forgetting constraints or contradicting itself. Agentic tasks that chain many dependent steps see compounding error rates — 95% per-step success becomes 60% after ten steps.
Trained on internet text, LLMs absorb the internet's biases — demographic, cultural, linguistic. Alignment reduces but doesn't eliminate this. More fundamentally, the Nature Medicine study is worth repeating: LLMs identified conditions at 94.9% alone, but humans with LLMs did no better than humans without. The bottleneck isn't the model's capability — it's the human-AI interface, the workflow design, the user's ability to evaluate AI output critically. Tools are only as good as the process around them. And right now, we're early in learning how to build those processes.
In February 2026, AI is extraordinary at structured tasks with clear evaluation criteria — coding, test-taking, pattern matching, text synthesis — and mediocre at tasks requiring judgment, factual reliability, physical reasoning, and sustained coherent thought over long horizons.
The technology behind the improvements is real: scale, human feedback, test-time compute, and agentic tool use each contributed measurable, documented gains. The capabilities are real. The economic trajectory is real.
What remains genuinely uncertain: whether the current rate of progress continues or plateaus. Whether productivity gains spread broadly or concentrate in software and a few other domains. How labor markets adjust — and how quickly. Whether "reasoning" constitutes understanding or the most sophisticated simulation of understanding ever built. And how to construct systems that reliably know the limits of their own knowledge — because until they can, every high-stakes application requires a human in the loop who can.