…or if you like the good/old one: The future is here it’s just not evenly distributed — and this time it’s on AI (or SI as from last week).

There is a growing disconnect in the AI/SI conversation, and it’s not because one side is wrong. It’s because people are having radically different experiences with the same technology.

Andrej Karpathy has been pointing at this for months, but his latest framing makes the problem particularly clear: the gap between what people think frontier AI can do and what it can actually do is widening, not closing. He describes a steep capability funnel, with most people near the wide end and a tiny group at the narrow end experiencing something that looks almost like a different technology.

Consider the distribution.

Around six billions people have barely touched AI. Another billion or two use free chat interfaces for search, writing, brainstorming, and the occasional questionable answer. For them, ChatGPT AI is useful, sometimes impressive, but hardly revolutionary.

Then there are roughly 20 million people, many working in technical fields, using paid frontier models and agentic tools on serious, multi-day projects. They are handing agents entire codebases, complex mathematical problems, and substantial research tasks that once required weeks of focused human effort. Increasingly, they return to find much of the work done.

At the very narrow end are perhaps a few thousand people with (free!) access to the latest internal systems. They are experimenting with fleets of agents working for days or weeks on substantial software and research projects, including security-sensitive work such as vulnerability discovery and exploit development.

These numbers should be understood as estimates illustrating the distribution, not a precise census. The important point isn’t whether the boundaries are exactly right. It’s how dramatically the experience changes as you move down the funnel.

And that’s where things get interesting.

The Same AI, Two Completely Different Realities

For most people, the benchmark is still a chatbot that occasionally gets a straightforward question wrong. They ask it to summarize an article, write an email, or explain a concept, and judge its capabilities accordingly.

For a smaller group of engineers, researchers, and builders, the experience is fundamentally different. They can give an agent a complicated objective, provide access to the necessary tools and context, and let it work through a problem that would otherwise consume hours or days. The result might not be perfect, but the amount of work completed can be substantial.

Both experiences are real. The mistake is assuming that one invalidates the other.

This creates a peculiar problem for anyone trying to understand where AI is heading. Someone who has only used a free chatbot can reasonably conclude that the technology is overhyped. Someone building with frontier models, coding agents, persistent context, and automated evaluation can reasonably conclude that the nature of software development is changing faster than most organizations realize.

They aren’t necessarily disagreeing about the technology.
They’re operating at different points in the capability funnel.

Three forces explain much of the difference.

First, the size of the problem matters. A simple question doesn’t reveal much about a system’s ability to plan, use tools, iterate, and solve a complicated problem. Larger tasks create opportunities for those capabilities to compound, provided the agent has sufficient context, tools, time, and a way to check its work.

Second, capability is jagged. AI isn’t uniformly good at everything. It can perform remarkably well in one domain and struggle with something that appears trivial in another. Tasks involving code, mathematics, formal reasoning, and other digitally verifiable work have benefited particularly from training approaches that reward measurable outcomes. When success can be checked reliably, systems can be trained and evaluated against that signal. Open-ended tasks with ambiguous definitions of success are harder to optimize.

Third, access is unequal. People aren’t just using different interfaces; they may be using different models, different levels of compute, different tool integrations, and different workflows. Those differences matter, and they compound. Someone who can run long-lived agents, inspect their work, refine the process, and repeat the cycle has an advantage over someone asking isolated questions in a browser tab.

The result is a widening gap between the public perception of AI and the capabilities that experienced practitioners encounter every day.

Security Is One of the Places Where the Gap Matters Most

For those of us building software and security infrastructure, this isn’t an abstract debate about benchmarks or predictions. It’s an operational change.

Software development has many characteristics that make it particularly suitable for agentic systems: the environment is digital, the work can be decomposed into manageable steps, and the results can often be tested. A coding agent can modify a codebase, run tests, inspect failures, make corrections, and repeat the process. That feedback loop allows it to do considerably more than generate a plausible-looking answer.

Cybersecurity has similar properties. Agents can help analyze large codebases, identify suspicious patterns, investigate vulnerabilities, generate tests, and work through remediation tasks. Some offensive security workflows can also be accelerated, including vulnerability discovery and the development of exploit paths. The implications extend to defenders, who must increasingly account for attackers operating with comparable automation.

Compliance is another interesting example. Much of the work involves gathering evidence, checking configurations, mapping controls to technical systems, identifying gaps, and verifying remediation. These are precisely the kinds of structured, repeatable activities where agents can reduce manual effort. The hard part isn’t simply generating a checklist or writing a policy; it’s connecting requirements to the actual environment, validating that controls are working, and maintaining that state over time.

Of course, none of this means that an agent can reliably run an entire security program without oversight. Real environments contain exceptions, incomplete information, legacy systems, ambiguous requirements, and consequences for getting things wrong. Agents also make mistakes, and a workflow that cannot detect those mistakes is a liability, not an innovation.

But the economics are changing. When more of the routine analysis and execution can be automated, human attention shifts toward defining objectives, handling exceptions, evaluating risk, and deciding what should happen next.

That shift is already important, even without assuming that today’s agents are autonomous experts.

What Builders Should Notice

There are three practical lessons for anyone building products or evaluating AI capabilities.

1. Stop evaluating AI only on small tasks.

If your test suite consists of writing a function, summarizing an email, or answering a trivia question, you’re measuring only a narrow slice of what these systems can do. Evaluate them on meaningful, end-to-end work: investigate a real bug, implement a feature across a codebase, trace a compliance gap to its technical root cause, or propose and verify a remediation plan.

Measure the quality of the result, the amount of human intervention required, the time to completion, and the cost of getting there. Include failures, not just successful demos. A system that completes 80% of a task but requires an engineer to spend hours untangling the remaining 20% may be less useful than it first appears.

2. Choose domains where results can be verified.

The fastest path to useful automation isn’t necessarily the most impressive demonstration. It’s finding a valuable workflow where success can be measured and errors can be caught. Code that passes meaningful tests, configurations that meet defined requirements, and evidence that satisfies a specific control are better starting points than vague objectives such as “make the system more secure.”

This doesn’t mean AI is limited to deterministic tasks. It means that reliable feedback loops make it easier to build systems that improve, and safer to delegate work without blindly trusting the output.

3. Treat the workflow as part of the product.

Access to a powerful model is not, by itself, a competitive advantage. The real leverage comes from combining models with the right context, tools, permissions, memory, evaluation, and orchestration. An agent that understands the environment, can execute approved actions, checks the outcome, and escalates uncertainty is far more useful than a chatbot that simply recommends what someone should do.

This is also where iteration speed matters. Teams that continuously improve their agent workflows can accumulate advantages in reliability, cost, and execution. The model is one component; the surrounding system determines how much of its capability becomes useful work.

The Bottleneck Is Moving

The most important shift may be happening somewhere less glamorous than the model itself: the human review process.

For years, the bottleneck in many technical workflows was producing the work. Engineers had to write the code, analysts had to investigate the findings, and compliance teams had to collect and reconcile the evidence. As agents take on more of those activities, the bottleneck increasingly becomes determining whether the output is correct, safe, complete, and worth shipping.

That creates a new set of engineering problems. How do you evaluate thousands of agent-generated changes? How do you limit what an agent can access or modify? How do you preserve an audit trail? How do you decide when an agent should continue, retry, or stop and ask for help? And how do you prevent the cost of reviewing automated work from consuming the savings it was supposed to create?

These aren’t reasons to avoid agentic systems. They’re the next layer of the problem.

The organizations that benefit most won’t necessarily be those with access to the most powerful model. They’ll be those that build the best systems around it: clear objectives, constrained permissions, reliable verification, measurable outcomes, and sensible human oversight.

The Question Worth Asking

You don’t have to believe the most aggressive AI predictions to recognize what’s happening. You only need to look at the tasks that agents can already complete, the conditions under which they succeed, and how quickly the surrounding workflows are improving.

The useful question is no longer simply, “Does it hallucinate?” Of course it does.
The question is: For the work that matters to your business, how much can an agent reliably complete, how much does it cost, and what must change in your process to take advantage of it?

Start with one real workflow.
Measure the baseline.
Give an agent the context and tools it needs, define what success looks like, and build a way to verify the result.

Then compare the quality, cost, and human effort against the existing process. You’ll learn more from that experiment than from another month of arguing about AI benchmarks.

Most people are still judging it from the wide end of the funnel. A smaller group is already building with capabilities that look very different. The gap between those experiences is real, and it has consequences for how software, security, and compliance get delivered.

The question isn’t which group has the more convincing opinion.
It’s whether you’re testing what the technology can do today on the problems that actually matter to you.

Be strong.