Anthropic Is Blocking Live Internet Access in Some AI Agent Evals. Here’s Why That Matters
Anthropic’s reported decision to cut off live internet access in some internal AI-agent evaluations highlights a basic problem: an agent that can browse, click, and take actions is harder to test safely than a chatbot that only produces text. Restricting internet access can reduce risk during evaluation, but it cannot prove an agent is safe once connected to real services.
What is the issue?
An AI agent does more than answer a question. Depending on its tools, it can search the web, open pages, inspect files, call software functions, or carry out a sequence of tasks. That extra reach is useful, but it creates a different kind of safety problem: a mistake may affect something outside the conversation.
The report named in the supplied headline says Anthropic is cutting off its internal evaluations from the live internet because it cannot reliably control its AI agents. That is the central claim available here; without the original report or more detail, it would be careless to claim which specific agents, tests, or incidents prompted the change. The important distinction is between a reported testing decision and a general conclusion about every Anthropic model.
Why does live internet access matter? Because the web is not a tidy test fixture. Pages change, services have their own permissions, and content can contain misleading instructions. A browser-enabled agent may encounter information that was not part of the test designer’s plan. If the agent can also use tools, the consequences could extend from reading a page to sending a message, changing a setting, or triggering a transaction.
I treat an agent as software with delegated authority, not as a clever chatbot with a fancy interface. That is a less glamorous description, but it puts attention where it belongs: on permissions, side effects, and recovery. Anthropic’s own research on agentic misalignment is relevant to this wider concern: researchers need to study how models behave when given tools and consequential objectives, not just whether they can produce a polished answer.
Cutting off the live web is one possible safety boundary. It can prevent an evaluation from interacting with real services or unknown pages. It also changes what the test measures. An agent that behaves well in a sealed environment has not automatically passed a test of how it handles the open web. Isolation is a brake, not a certificate.
Core benefits and importance
There are good reasons to restrict internet access while evaluating agents. Some are practical; others are about protecting people and services from a test that gets out of hand.
- Containment: A disconnected or tightly filtered environment can keep an experimental agent from reaching real accounts, public websites, or services. This reduces the chance that an evaluation creates unintended side effects.
- Repeatability: The public web changes constantly. A controlled test environment can hold pages and data steady, making it easier to compare model behavior across runs. That helps analysts understand whether a change in the model or test setup caused a different result.
- Privacy: Evaluations can involve prompts, files, or user-like records. Restricting outside connections makes accidental disclosure less likely, though it does not replace careful data handling inside the test system.
- Clearer attribution: When a test runs inside a defined environment, logs can help show which tool call, page, or instruction preceded an action. Without those records, blaming the model—or the test setup—becomes guesswork.
- Lower operational risk: If an agent is still being assessed, there is little reason to give it broad access to live services by default. Use the minimum authority needed to answer the research question.
Here’s the kicker: the safer test may also be less representative. A sealed sandbox cannot fully reproduce an open web service’s changing content, login flows, rate limits, or unexpected responses. If an agent is intended to work online, evaluations must eventually test the relevant online risks—but they should do so through a controlled setup, not by casually pointing an experimental system at the public internet.
I would separate two questions that often get mashed together. First: can a model complete a task in a test environment? Second: can the whole system complete it safely with real tools and real consequences? The second question includes the model, software wrappers, permissions, monitoring, and human oversight. Anthropic’s Responsible Scaling Policy offers useful context for why model risk controls and deployment decisions should be treated as related, but not interchangeable, work.
This is also a matter of stewardship. Powerful tools should serve people without treating their data, accounts, and time as disposable test material. That principle is not anti-innovation; it is a reason to build safeguards before handing a system more authority.
Step-by-step: how to evaluate agents without being reckless
A responsible evaluation does not have to choose between a toy simulation and unrestricted internet access. The better approach is to increase realism in measured stages, while keeping the consequences bounded.
- Write down the intended task. Be specific. “Use the web” is not a useful evaluation goal. “Find the current return policy on this test site and report the relevant paragraph” is clearer. Define what counts as success, what the agent must not do, and which actions would count as a failure.
- List every permission the task needs. Does the agent need to read pages, download files, submit forms, or use an account? These are not equivalent powers. Start with read-only access if that is enough. Do not grant email sending, purchases, account changes, or access to private data just because the agent might find them convenient.
- Begin with a contained environment. Use a local mock site, a captured page set, or another controlled test setup. The point is not to pretend the web is simple; it is to learn basic behavior without exposing external services. Record what the agent sees and which tools it can call.
- Add realistic hazards deliberately. Test ambiguous instructions, unexpected page layouts, irrelevant material, and content that tries to redirect the agent. Keep the scenario bounded. A useful evaluation probes how the system handles pressure and conflicting instructions without letting a test instruction trigger real-world damage.
- Use a staged connection policy. If the research question requires internet access, allow only the destinations and actions needed for that test. A monitored proxy or allowlist can be more informative than an all-or-nothing switch. Set time limits, request limits, and clear stop conditions.
- Separate reading from acting. An agent that may inspect a page should not automatically be allowed to submit a form. Require human confirmation for consequential actions, and make sure the confirmation explains what will happen. “Approve?” is weak if the person cannot see the action and its likely effect.
- Log the full chain. Keep records of prompts, observations, tool requests, results, and human interventions. Protect those records because they may contain sensitive material. Without useful logs, teams cannot reproduce failures or tell whether a safeguard worked.
- Test the stop button. Revoke access during a task and confirm that the agent cannot keep acting through a lingering session or separate tool. A stop mechanism that only closes the visible window is not much of a stop mechanism.
- Review failures before widening access. Look for repeated patterns, not just headline-grabbing incidents. If an agent misunderstands a limit, follows a hostile instruction, or attempts an action beyond the task, identify whether the fix belongs in the model, the tool layer, the test design, or the permission policy.
When I assess tool-enabled systems, I care less about a smooth demo than about whether the system stays inside its assigned boundary when something unexpected happens. A staged method gives teams more useful evidence than a single yes-or-no question about whether the model is “safe.” For broader guidance on structuring risk work, the NIST AI Risk Management Framework is a sensible reference point; it emphasizes risk management across the system’s use, rather than treating one test result as the whole story.
Comparison: isolated evaluations versus live-web evaluations
The supplied report does not provide numbers, named test suites, or a detailed comparison of Anthropic’s methods with a competitor. So the honest comparison is between the two evaluation approaches that the headline raises: testing without live internet and testing with it. That distinction is more useful than inventing a scorecard.
| Evaluation approach | What it can show | What it cannot establish by itself | Main risk to manage |
|---|
| Isolated or offline test | How an agent behaves with a known set of pages, data, and tools; whether behavior is repeatable | How it responds to changing public websites, real services, and unplanned online content | A test that is too artificial may miss important online behaviors |
| Live-web test | How an agent encounters real pages, changing content, and service responses | That the agent will behave safely in every site, account, or future situation | Unintended access, data exposure, or actions with real consequences |
| Staged, restricted test | How behavior changes as carefully chosen real-world features are introduced | That a restricted result guarantees safe use outside those boundaries | Misconfigured permissions or gaps in monitoring and revocation |
Frankly, neither column wins in every circumstance. Offline tests are easier to control, repeat, and audit. Live-web tests offer realism but create more ways for the test itself to cause trouble. A staged test can balance those needs if its limits are real—not just a policy document that leaves every tool enabled.
The meaningful measure is not “internet on” versus “internet off.” It is whether the test matches a clearly stated risk question and whether the agent’s capabilities are restricted to what the evaluation requires. A company can run an offline test that tells it very little, or a carefully limited online test that answers a narrow question responsibly. The method matters more than the label.
This is where product claims need a bit of skepticism. A benchmark score does not tell you which tools the agent had, what it could reach, how many runs were checked, or how failures were handled. Ask what the agent was allowed to do, what was blocked, and how the team knows. If those answers are missing, the score is only part of the story.
Common mistakes to avoid
The first mistake is treating disconnection as proof of safety. It is evidence about behavior under specific conditions. It cannot establish what happens when the same model receives different tools, new instructions, or access to live services. Reported access restrictions may be prudent; they should not be inflated into a claim that the underlying control problem is solved.
The second mistake is assuming that a realistic test must be unrestricted. It does not. A team can create realistic tasks, use controlled accounts, constrain destinations, and require confirmation before important actions. Realism is not the same thing as handing an experimental agent the keys.
Third, do not focus only on the model. Agent behavior depends on the whole system: the instructions, browser or API tools, permission settings, session management, data stores, and people supervising it. If the agent can perform an action that the task does not require, the system has a design problem even if the model usually declines to use that power.
Fourth, beware of tests that reward task completion while ignoring how it happened. An agent that completes a request by exceeding its permissions has not passed. Evaluation criteria need to include boundary-respecting behavior, not just speed or a correct final answer.
Fifth, do not treat a human approval prompt as magic. People often approve requests quickly, especially when alerts are vague or repetitive. Show the proposed action, destination, and likely consequence; ask for confirmation at the point where it matters. Keep the number of prompts low enough that users will still read them.
Sixth, avoid one-off testing. A single successful run says little about variation across prompts, page content, or system state. Repeat tests, examine failures, and document changes to the model and tool setup. The goal is not to manufacture a reassuring demo. It is to find weaknesses before they affect users.
Finally, do not hide uncertainty behind confident language. If a test only covers a limited environment, say so. If internet access has been blocked, explain what that boundary does and does not tell you. The NIST Generative AI Profile provides further guidance for thinking about risks across generative AI use; it is a useful reminder that safety depends on context, deployment, and oversight, not a single switch.
Frequently asked questions
Why would Anthropic block live internet access during agent evaluations?
The supplied headline says Anthropic is cutting off its internal evaluations from the live internet because it cannot reliably control its AI agents. Restricting access can reduce exposure to changing websites and unintended interactions while researchers test behavior. The headline does not specify which evaluations are affected or the precise technical controls involved, so claims beyond that should be treated cautiously.
Does an offline evaluation prove an AI agent is safe?
No. It can show how an agent behaved within the tested environment, with the tools and limits provided. It cannot by itself establish how the same system will act on live websites, with different permissions, or under conditions the test did not cover. Useful evidence needs a clearly defined scope and, when relevant, additional staged testing.
Are AI agents more dangerous than chatbots?
Not automatically, but agents can have more ways to affect the world. A text-only chatbot mainly produces responses; a tool-enabled agent may browse, call software, or carry out actions. Risk depends on what it can access, what it is asked to do, and whether its actions can be reviewed or reversed. More capability calls for tighter permission design.
Can companies test agents with internet access responsibly?
Yes, if the connection is justified and bounded. Teams can use test accounts, limit destinations and actions, monitor activity, set clear stop conditions, and require approval for consequential steps. Some questions still need isolated testing, and others need controlled online tests. The right setup depends on the risk being measured.
Final Thought
The useful takeaway is not that AI agents should never touch the internet. It is that web access is a capability with consequences, and should be granted as deliberately as access to a payment account or a company inbox. A system that can only answer in a chat window has different failure modes from one that can browse, inspect documents, and act through connected tools.
The reported decision deserves attention because it recognizes a real testing problem: researchers cannot assume they have reliable control simply because an agent is running inside an evaluation. Cutting off access may be a sensible containment measure. It also leaves an honest question for the next stage: how will teams gather evidence about online behavior without exposing real people and services to avoidable risk?
The answer is disciplined testing, not bravado. Start with the smallest permission set, log what the agent does, make important actions reviewable, and prove that access can be revoked. Then introduce realism only when it answers a specific question. A careful boundary can feel less exciting than an unrestricted demo, but it is far more useful to the people who must live with the outcome.
Technology is best judged by what it lets people do—and by the damage it prevents when something goes wrong. That standard makes room for useful AI while insisting that convenience is not a license to ignore privacy, consent, or human judgment. If agent builders can show not just what their systems accomplish but how they keep them within bounds, users will have a sounder basis for trust.