The Wrong Cooks in the AI Regulation Kitchen
Something significant is happening in the artificial intelligence industry, and I think it deserves more attention than it's receiving. Anthropic Chief Executive Officer Dario Amodei recently proposed a framework for strengthening AI oversight, including giving independent evaluators ongoing access to the internal operations of leading AI companies. Rather than conducting periodic assessments from the outside, these evaluators would work inside the organizations, with access comparable to internal risk teams and the ability to publish their findings without corporate interference.
Anthropic has committed to implementing this first step independently, without waiting for broader industry agreement or government intervention. OpenAI Chief Executive Officer Sam Altman subsequently expressed support for the approach, as did Elon Musk and representatives of Google DeepMind. That level of agreement is significant in an industry that has frequently faced criticism over transparency and accountability. Independent evaluators with meaningful access to internal systems could help identify risks that might otherwise remain hidden from public scrutiny.
But while I welcome the direction of this proposal, I believe it leaves several fundamental questions unanswered. Who selects the evaluators? Who determines what they examine? Whose experiences count as evidence? And what happens when they discover something that requires immediate intervention? Because there's a difference between allowing someone to observe a system and giving that person the authority to protect the people affected by it.
The Restaurant Inspection Problem
Consider how we evaluate restaurant safety. A health inspector enters a restaurant, examines food storage temperatures, checks sanitation procedures, reviews expiration dates, and determines whether the establishment complies with applicable standards. Those inspections serve an important purpose, but imagine an inspection process in which the inspector never considers whether anyone who ate at the restaurant became sick. The kitchen might appear compliant with every documented procedure while customers continue experiencing harm.
Now imagine placing that inspector inside the restaurant permanently. They have a desk, access to records, and the ability to publish their findings. That improves visibility into the restaurant's operations, but it doesn't automatically resolve the question of whether the inspection process is capturing the experiences of the people the safety standards are supposed to protect. And what happens if the restaurant selects the inspector, determines their compensation, and establishes the boundaries of what they're allowed to examine? Even an evaluator acting in good faith may be operating within a structure that limits their independence.
That's the distinction I see in the emerging conversation about AI oversight. Embedded evaluators may gain unprecedented visibility into how companies develop and test their systems. But if their assessments focus primarily on technical processes and internal safety standards, they could miss important evidence from people experiencing the consequences of those technologies. An inspection can be technically thorough and still fail to answer the most important question: Is the system actually protecting the people it's supposed to serve?
The Harm Isn't Always Somewhere in the Future
When the technology industry discusses AI risk, the conversation frequently centers on what might happen as systems become more capable. Those questions deserve attention, but they shouldn't overshadow the harms associated with automated decision-making systems already operating across employment, housing, education, healthcare, and public services.
Consider someone applying for rental assistance whose application is flagged for additional review without a clear explanation, or a Medicaid recipient who receives a denial notice without understanding which information influenced the determination. A parent may discover that a school screening tool flagged their child but struggle to obtain an explanation of how the assessment was made. A job applicant may be evaluated by automated screening technology before a hiring manager reviews their qualifications, while an unemployment claimant may be directed into a fraud review process that delays access to benefits. These systems vary considerably in how they operate, and not all use the same forms of artificial intelligence. But they raise a common question about accountability: What happens when a consequential decision is made through an automated process and the person affected cannot meaningfully understand, challenge, or correct it?
For individuals without substantial financial resources or access to legal representation, navigating these systems can become an enormous burden. They may have documentation supporting their position, records of repeated attempts to correct an error, and evidence of the consequences they've experienced. But if the oversight structure doesn't recognize those experiences as relevant evidence, their documentation may never influence how the system is evaluated. And that's a serious design problem, particularly when the people most affected by these technologies may also have the fewest resources to challenge them.
Who Gets to Define Compliance?
There's another issue embedded in the proposal for independent AI evaluators: Who establishes the standards against which the companies are evaluated? Consider a school that creates its own grading rubric and then hires an outside reviewer to confirm that teachers applied it correctly. The reviewer might conduct a thorough, independent assessment and accurately conclude that every procedure was followed. But that doesn't establish that the rubric itself measures what students actually need to learn. The evaluation can be technically sound while the underlying standard remains inadequate.
A similar problem can arise when AI companies establish internal safety frameworks and independent evaluators assess whether those frameworks have been followed. Compliance with an internal standard doesn't necessarily establish that the standard adequately protects the public. An evaluator may be professionally independent while still operating within a framework shaped by the company under review, particularly if that company influences the scope, resources, or conditions of the evaluation.
This is why independence must involve more than access to internal systems. Who appoints the evaluators? Who pays them? What protections exist against retaliation or termination? Can they expand the scope of their investigations when they identify concerns outside the original assessment? And who decides whether the company's safety standards are sufficient in the first place? These questions need to be addressed before an oversight model becomes widely adopted, because technical compliance should never become a substitute for meaningful accountability.
The Missing Evidence: People Who Experience the Harm
If I could change one thing about the emerging approach to AI oversight, it would be how evidence from affected individuals is incorporated into the evaluation process. Technical testing is essential, but it shouldn't be the only source of information. The experiences of people living with the consequences of automated decisions can reveal problems that aren't necessarily visible through internal testing or compliance reviews.
Consider a parent whose child has been repeatedly flagged by an automated screening system. She has correspondence with the school, records of the decisions, and documentation showing that she has attempted to challenge the results. Under an oversight structure focused primarily on internal technical compliance, that documentation might remain outside the formal evaluation process. It could be treated as an individual customer service complaint rather than evidence of a broader system failure. Meanwhile, the evaluator could conclude that the company followed every established procedure without ever examining whether those procedures produced harmful outcomes.
I believe that distinction needs to change. When individuals document potential harms associated with automated systems, their evidence should have a defined pathway into independent review. Companies should be required to respond to substantiated concerns within clear, enforceable timeframes, and those responses should inform public accountability reporting, with appropriate protections for privacy and sensitive information. Technical evaluators should examine how systems were designed and tested, while evidence from affected individuals should help establish how those systems operate in practice. An inspection that examines the manual is not the same as an inspection that also examines the complaints. We need both.
What Happens When an Evaluator Finds Something Dangerous?
There's another fundamental question about independent oversight that deserves greater attention: What happens after an evaluator identifies a serious risk? Imagine a bridge inspector discovering a significant structural defect. The inspector documents the problem, prepares a report, and makes the findings public. But what if that inspector has no authority to close the bridge, restrict traffic, or require immediate action? The inspection may have been conducted perfectly, and the findings may be accurate, but vehicles continue crossing the bridge because nobody has been given the authority to intervene.
The same principle applies to AI safety. If an evaluator discovers a serious problem, there must be a clear process for determining what action follows. That process could involve mandatory remediation, escalation to an independent regulator, restrictions on deployment, or other interventions proportionate to the severity of the risk. Evaluators also need protections that allow them to report serious findings without fearing professional or financial retaliation, and the standards governing those interventions should be established in advance rather than negotiated after a problem emerges.
Otherwise, oversight risks becoming an exercise in documentation rather than prevention. Transparency is valuable, but transparency without enforcement can tell us that something is wrong without giving anyone the authority to make it right. An inspector with access but no enforcement authority is still just someone with a clipboard. And when we're discussing technologies capable of affecting millions of people, that isn't sufficient.
Corporate Cooperation Is Not Public Accountability
I don't want to dismiss the significance of leading AI companies expressing support for independent evaluation. Greater access to internal systems could help researchers identify risks that would otherwise remain hidden, and the ability to publish findings without corporate interference is an important principle. But voluntary corporate cooperation is not the same as enforceable public oversight, particularly when companies retain influence over the scope and conditions of the evaluation.
Technology companies face pressure to release products, attract investment, increase revenue, and compete with other developers. Those incentives don't automatically make companies irresponsible, but they create circumstances in which independent accountability becomes particularly important. A credible oversight structure must therefore address more than whether evaluators can enter a company's offices. It must establish who determines their authority, how their independence is protected, what evidence they're required to consider, and what happens when serious problems are identified.
And the people affected by AI systems need meaningful opportunities to participate in that process. Not simply as consumers submitting complaints, but as stakeholders whose experiences can reveal risks that technical assessments alone may overlook. Public accountability requires more than companies agreeing to be observed. It requires mechanisms through which the people affected by technological decisions can influence how those decisions are evaluated and corrected.
Why the Timing Matters
We're at an important moment in the development of AI governance. The practices established today could influence future industry standards, regulatory frameworks, and public expectations. Once a particular oversight model becomes widely accepted, its assumptions can become difficult to challenge. That's why the design of these systems matters now, while the frameworks are still being developed and there is an opportunity to address weaknesses before they become embedded in institutional practice.
Historically, many safety regulations have emerged in response to harm that had already occurred. Building codes, transportation protections, and workplace safety standards have all evolved through lessons learned from preventable injuries and failures. Artificial intelligence presents an opportunity to establish stronger accountability structures before some of the most serious risks materialize. But that opportunity will be diminished if the conversation remains confined to technology companies, technical researchers, and government officials.
Workers, communities, public-interest organizations, and individuals affected by automated decisions also have expertise that belongs in the process. Their participation should not be an afterthought, particularly when they may recognize problems that are invisible to the people designing or evaluating the technology. If today's voluntary oversight arrangements become templates for future regulation, then the absence of public participation today could become a structural weakness in tomorrow's laws.
What Meaningful AI Oversight Should Look Like
I believe a credible approach to independent AI evaluation should rest on three principles: genuine independence, meaningful public evidence, and enforceable consequences. Evaluators need access to systems and records, protection from interference, and the ability to publish findings without corporate approval. Their authority should not depend entirely on the preferences of the organizations they're examining, and their work should be subject to standards that extend beyond internal corporate policies.
Oversight must also include mechanisms for incorporating documented harms from workers, consumers, and communities affected by automated systems. Technical compliance cannot be the only measure of whether a technology is operating responsibly. When people report harmful outcomes, those reports should be examined as potential evidence of systemic problems rather than dismissed as isolated complaints. And when serious risks are identified, there must be clear procedures for remediation, escalation, and intervention. Publishing a report cannot be the final step when people remain exposed to preventable harm.
These principles aren't arguments against innovation. They're conditions for developing technology that can earn public trust. As artificial intelligence becomes more deeply embedded in institutions that shape people's lives, meaningful accountability will require more than sophisticated technical assessments. It will require structures capable of connecting what happens inside technology companies with what happens to people outside them.
The Public Cannot Be Missing From the Design
Anthropic's proposal for embedded independent evaluators deserves serious consideration. It could improve transparency and provide greater visibility into how leading AI companies manage risk. But access alone isn't accountability. An evaluator can have a desk inside a company, access to internal systems, and permission to publish findings while still operating within a framework that excludes the experiences of people affected by the technology. And an oversight process can document serious problems without possessing the authority to ensure they're corrected.
The ultimate purpose of AI safety shouldn't be to demonstrate that companies followed their own procedures. It should be to protect people from preventable harm. As the industry develops new approaches to independent evaluation, we need to ensure that the public has a meaningful role in determining what safety means, what evidence counts, and what consequences follow when standards aren't met.
Otherwise, we risk building an extraordinarily sophisticated inspection system that can tell us everything about how AI companies operate while telling us far too little about what their technologies are doing to people. And that would be a missed opportunity, particularly when the frameworks being established today could influence the laws and institutions governing artificial intelligence for years to come.
The question isn't simply whether independent evaluators should be allowed inside AI companies. It's whether the people living with the consequences of artificial intelligence will finally have a meaningful place in the oversight process.
Because the public deserves more than an inspector with a clipboard. It deserves accountability with the authority to act.