AI, Automation, and Autonomy

CORE PROPOSITION: Use AI to interpret. Use automation to execute defined obligations. Permit autonomy only inside bounded authority. Keep consequential commitments attached to identifiable policy, deterministic enforcement, observable execution, and accountable people or institutions.

Practice-based research synthesis | Systems architecture | AI governance

This paper is intended to support system design and governance. It does not determine legal liability, certify compliance, or replace sector-specific legal, safety, security, or regulatory review.

Abstract

Organizations are rapidly adding generative and agentic AI to systems that can search, draft, classify, call tools, modify records, transact, deploy software, influence physical operations, and affect rights or safety. The dominant governance question is often framed around the model: which provider is approved, how accurate the model is, whether a human is “in the loop,” or how much autonomy is permitted. This paper argues that the more durable unit of governance is the consequential state transition - the point at which interpretation or discretion becomes an authoritative commitment.

The paper separates five system properties that are often conflated: intelligence, automation, autonomy, authority, and accountability. It proposes a consequence-based authority allocation doctrine: probabilistic AI should be used for interpretation, search, prediction, synthesis, and proposal; deterministic automation should own defined obligations and exact state transitions; autonomy should be bounded by preauthorized corridors whose scope contracts as consequence and irreversibility rise; accountable humans should resolve legitimate exceptions and high-impact ambiguity; and institutions should retain policy ownership, authority assignment, deployment responsibility, appeal, and remedy.

The position is supported by established security architecture, human-factors research, current AI reliability evidence, banking model-risk governance, defense autonomy controls, aviation safety-assurance practice, biomedical AI guidance, the EU AI Act, and current U.S. administrative examples that continue to locate formal responsibility in natural persons and recognized legal entities rather than AI models. These sources do not all prescribe the architecture proposed here. They independently converge, however, on several measurable properties: risk-proportionate controls, explicit authority, validation, monitoring, human capability to intervene, traceability, lifecycle governance, and outcome verification.

The paper therefore advances “deterministic before probabilistic” as an authority-allocation rule, not as a rejection of AI. High levels of automation are compatible with low levels of autonomy. Powerful AI can provide substantial intelligence without owning commitment authority. The objective is not to prevent machines from acting; it is to prevent operational authority from becoming detached from the people and institutions that remain responsible for the consequences.

THESIS IN ONE SENTENCE The decisive boundary is not human versus machine; it is proposal versus commitment - and the authority to cross that boundary must remain explicit, bounded, attributable, and provable.

Keywords: AI governance; automation; autonomy; accountability; authority; high-risk systems; human oversight; model risk; safety assurance; deterministic systems; agentic AI; auditability.

1. Executive Argument

The current AI-policy debate often begins with capability: how powerful a model is, which tools an agent can call, what benchmark it achieves, or whether a provider labels a feature “autonomous.” Accountability begins one layer lower. Before deciding how intelligent or autonomous a system may become, an organization must decide what the system is authorized to make true.

A model can be highly capable, well-tested, and useful while still being the wrong component to own payment, access, production release, record deletion, legal status, safety interlocks, regulated attestations, or other consequential commitments. The problem is not that AI is uniquely dangerous. The problem is that capability, discretion, authority, and responsibility are distinct system properties - and modern software can accidentally collapse them.

This paper proposes a default architecture for consequential organizational systems: use probabilistic intelligence where ambiguity creates value; automate exact obligations wherever rules can be validated; permit autonomy only within explicit and recoverable corridors; retain accountable human judgment for exceptions that cannot legitimately be reduced to fixed rules; and preserve institutional ownership over policy, deployment, control-plane changes, appeals, and remedy.

WORKING DOCTRINE AI for interpretation. Automation for defined obligations. Bounded autonomy for constrained discretion. Humans for legitimate judgment and exceptions. Institutions for policy and accountability.

1.1 The central risk is authority leakage

AI risk is often described as an output-quality problem: hallucination, bias, unsafe content, prompt injection, or low accuracy. Those risks matter, but a wrong answer becomes an operational failure only when the surrounding system accepts it, grants it authority, or represents it as settled truth. The architectural failure is authority leakage: business rules, permissions, and commitment logic migrate from governed software and accountable roles into prompts, model behavior, ad hoc reviewer judgment, or opaque agent plans.

Authority leakage appears in ordinary shortcuts: a model selects a final price because it “understands” a price book; an agent decides whether a customer is paid; a summary replaces authoritative workflow state; a provider acknowledgement is treated as completion; an administrator’s technical access is treated as policy authority; or an AI agent can modify the permissions or evidence rules intended to constrain it. In each case the organization has transferred operational control without establishing a reproducible decision owner.

1.2 Automation and autonomy are not synonyms

A system can automatically execute thousands or millions of steps without choosing its own business goals or inventing missing rules. A payment engine can calculate exact balances, a scheduler can allocate resources under validated constraints, a manufacturing engine can create deterministic bills of material, and a retention service can delete records after approved periods and legal-hold checks. These systems are automated; they need not be autonomous.

Autonomy begins when the system receives discretion to select among actions, plans, strategies, tools, or intermediate objectives beyond a fully predetermined rule path. That discretion can be useful. The paper does not seek to eliminate it. It seeks to place it where the consequence of a wrong choice is bounded, observable, reversible, and still governed by an authority outside the autonomous component.

1.3 The legal asymmetry

Operational authority can be delegated to software much faster than legal and institutional responsibility can be delegated away from people. Today’s AI systems may negotiate, recommend, schedule, purchase, communicate, deploy, or manipulate operational state, but current legal and regulatory structures continue to assign formal responsibility to natural persons, recognized legal entities, public authorities, licensed professionals, providers, and deployers. In U.S. tax administration, for example, an EIN application generally requires a natural-person responsible party; U.S. patent inventorship likewise remains limited to natural persons [24], [25]. In the EU AI Act, the regulated provider and deployer roles are assigned to natural or legal persons, public authorities, agencies, or other bodies [18].

The implication is not that architecture determines legal liability. It is that an organization creates avoidable accountability risk when it delegates consequential discretion to a component that cannot itself absorb the legal or institutional responsibility that remains with the organization and its people.

ACCOUNTABILITY QUESTION If responsibility remains attached to people and institutions, why would the architecture allow consequential authority to become detached from them?

2. Research Question, Method, and Evidence Standard

2.1 Research question

How should intelligence, deterministic automation, autonomous discretion, human judgment, and institutional authority be allocated within consequential systems so that organizations can obtain the benefits of AI while preserving reproducible control, traceable responsibility, meaningful human authority, and evidence sufficient for accountability?

2.2 Method

The paper uses a design-science and systems-synthesis method. It identifies a recurring operational problem - authority leakage - then integrates established security principles, human-factors findings, model-risk practice, safety-assurance guidance, AI governance requirements, and practitioner system design into an implementable placement doctrine. The contribution is normative and architectural: it proposes where different kinds of computation and discretion should sit in a governed system and how the boundary to consequential state should be enforced.

2.3 Evidence hierarchy

Because organizational AI discourse is saturated with vendor claims, launch announcements, and corporate positioning, this paper applies a strict source hierarchy. Corporate press releases and marketing claims are not treated as proof of safety, adoption, compliance, or system performance.

Where a source is a recommendation rather than a binding requirement, the paper labels it accordingly. Where evidence is measured only in a bounded study or system, claims remain inside that boundary. The goal is not rhetorical caution for its own sake; it is to prevent evidence from being enlarged into claims it cannot support.

2.4 The “80/20” posture

Practitioner experience suggests that the doctrine is a useful default for the broad majority of high-consequence organizational workflows: environments where duties are governed by policy, people or institutions remain accountable, and incorrect commitments can create financial, safety, regulatory, legal, contractual, privacy, or rights-related harm. This “80/20” characterization is a design heuristic, not a measured prevalence statistic. The burden of justification should increase when an organization departs from the default and grants broader autonomous authority over consequential state.

2.5 Relationship to related practitioner architecture

The author has separately developed practitioner architectures for governed execution and evidence, including TAG. Those systems informed the operational questions that motivated this paper, but TAG is not the subject of this paper and is not required for the thesis. A separate paper will address that architecture. Patent status, implementation history, or internal benchmarks are not used here as proof of the general doctrine.

3. Definitions: Intelligence, Automation, Autonomy, Authority, and Accountability

A central governance error is treating AI, automation, autonomy, capability, authority, and accountability as interchangeable. They are not. The paper separates them so that each can be governed explicitly.

3.1 Capability does not confer authority

A model capable of calling a payment API is not thereby authorized to issue payments. A service account that can edit a policy table is not thereby entitled to change policy. An administrator who can reach a database is not automatically permitted to modify regulated records. Technical possession of a capability must therefore be separated from legitimate permission to exercise it. This is an old security principle applied to a new execution surface [3], [4].

3.2 Determinism is not truth

A deterministic rule can consistently enforce a bad policy, use stale data, encode discrimination, or contain a defect. Determinism improves reproducibility, testability, attribution, and change control; it does not replace governance, validation, ethics, domain expertise, or outcome monitoring. A reproducible error is still an error.

3.3 Probabilistic does not mean random

A fixed model, seed, and configuration may produce repeatable output. The relevant distinction is not whether bytes repeat; it is whether the output is learned inference whose semantic correctness is guaranteed by the governing system. Repeatability alone does not convert a model inference into an authoritative fact.

3.4 Consequential state

The framework applies most strongly when a system can change or represent consequential state: money, pricing, access, safety, eligibility, legal rights, production release, regulated records, deletion, privacy, public attestations, or other states whose error is costly, difficult to reverse, or difficult for an affected person to detect.

4. Formal Placement Model

The placement question should not be answered by asking whether AI can perform a task. It should be answered by asking what uncertainty the task contains, what state it can change, how reversible the action is, whether the organization can define the rules, and what evidence must exist afterward.

4.1 Where AI belongs

• Transcribing notes or conversations into drafts for confirmation.

• Extracting candidate fields from documents while retaining source references and uncertainty.

• Semantic search over authorized material.

• Drafting explanations, communications, summaries, and support responses from approved facts.

• Classifying or triaging information to prioritize human or deterministic review.

• Suggesting anomalies, similar cases, missing evidence, counterexamples, or scenarios.

• Simulation and decision analysis that does not itself change authoritative state.

The model’s value is interpretive leverage. Its output should remain identifiable, source-bounded where possible, and optional until validated. Abstention and escalation should be successful outcomes when evidence is inadequate.

4.2 Where deterministic automation belongs

• Identity, authentication context, authorization, tenant scope, field-level access, and segregation of duty.

• Money arithmetic, pricing, margin floors, tax, units, dimensions, rounding, and ledger state.

• Buildability, bills of material, production release, quality gates, and safety interlocks.

• Workflow state machines, prerequisites, expiries, and no-false-progression semantics.

• Retention, legal hold, deletion eligibility, export scope, consent state, and privacy-right execution.

• Idempotency, transactional behavior, bounded retry, timeout states, reconciliation, rollback, and disaster recovery.

• Policy evaluation, approval validation, evidence generation, signing, release gates, and audit replay.

4.3 Where bounded autonomy belongs

Autonomy is most defensible where the objective is already authorized, the action space is explicit, the system cannot alter rights or policy, failure is reversible or cheaply recoverable, outcomes are observable, and a kill or containment path exists. An autonomous component may choose how to search approved sources, prioritize internal work, simulate plans, select among approved tools, or optimize reversible resources. It should not be able to enlarge its budget, add destinations, weaken approval, alter safety policy, redefine success, delete evidence, or approve its own exception.

BOUNDED-AUTONOMY RULE An autonomous component may exercise discretion inside its authority. It must not determine the extent of its own authority.

4.4 Where humans and institutions belong

Humans should not be retained merely as liability absorbers. Their role is to exercise legitimate judgment that should not be silently reduced to a fixed rule, and to govern the rule system itself. Institutions remain responsible for purpose, policy, risk tolerance, prohibited uses, authority allocation, exception policy, release, monitoring, appeals, and remedy.

4.5 The consequence-discretion relationship

Permissible machine discretion should generally contract as consequence, irreversibility, uncertainty, and evidentiary weakness increase. This is a design relationship rather than a literal mathematical law:

CONCEPTUAL RELATIONSHIP Permissible autonomous discretion decreases as consequence × irreversibility × uncertainty increase; deterministic constraints, explicit authority, validation, reconciliation, and evidence should increase in the opposite direction.

5. Govern Consequential State Transitions, Not Merely Models

A model does not create operational consequence merely because it produces an incorrect answer. Consequence occurs when the surrounding system accepts, authorizes, executes, or represents that output as authoritative state. Two organizations can deploy the same model and create radically different accountability properties.

5.1 The consequential state transition

A consequential transition moves an authoritative condition from one state to another: unpaid to paid; unauthorized to authorized; draft to binding commitment; candidate price to final price; not released to released; retained to deleted; development to production; requested to completed. The governance question is not simply which component participated. It is what conditions allowed the transition to become authoritative.

5.2 Proposal-commit separation

A proposal may contain an interpretation, extracted fact, prediction, recommended action, plan, generated communication, or requested tool call. A commitment changes authoritative reality. These should remain separate events.

5.3 Accuracy does not determine authority

Model accuracy is useful but insufficient as an authority rule. Aggregate accuracy does not reveal the severity of the remaining errors, whether those errors are reversible, whether they can be detected before commitment, or whether the affected person has a remedy. A less accurate model behind strong authority containment may create less operational risk than a highly accurate model with unrestricted execution privileges. The relevant question is not only “How often is the model correct?” but “What can happen when it is wrong?”

5.4 Confidence and structure are not permission

Confidence scores, explanations, citations, and structured output can improve interfaces and validation, but they do not constitute authority. A well-formed JSON object remains model output. The receiving system must distinguish candidate values from authoritative values, evidence from inference, and recommendation from permission.

5.5 No false progression

Operational systems frequently collapse attempted action into completed outcome. A payment request is represented as paid; an email provider acceptance is represented as delivered; a deployment is initiated and represented as deployed; a deletion is queued and represented as deleted. These are different states. A responsible system distinguishes proposed, validated, authorized, attempted, accepted, externally observed, reconciled, complete, failed, unknown, and reversed as appropriate to the domain.

NO-FALSE-PROGRESSION RULE A system must not claim more progress than it can prove. When the outcome is unknown, the honest state is unknown or pending - not success.

5.6 Unknown and abstention are valid states

A missing policy should not trigger invention. A stale record should not silently authorize action. An unavailable provider should not create fake completion. A conflicting source should not be silently resolved by a model when the result is consequential. High-integrity systems intentionally support abstain, escalate, request evidence, quarantine, defer, deny, simulate-only, review-required, unresolved, and safe shutdown states.

5.7 Control-plane isolation

The component being governed must not possess unilateral authority to rewrite the rules governing itself. Policy changes, model allowlists, approval thresholds, signing identities, evidence retention, and bypass permissions are themselves consequential transitions and require their own versioned authorization, testing, rollout, rollback, and custody path.

6. Evidence Supporting Consequence-Based Authority Allocation

The thesis is not derived from a single benchmark or regulation. It is supported by converging evidence from model reliability, human factors, security architecture, auditing, and high-risk governance. The sources differ in domain and do not all prescribe the same implementation. Their convergence is in the properties they demand: explicit context, bounded authority, validation, effective challenge, monitoring, human capability to intervene, traceability, and lifecycle responsibility.

6.1 Probabilistic intelligence remains epistemically fallible

Generative AI can be extraordinarily useful while remaining fallible as a source of settled fact. NIST’s Generative AI Profile treats generative-AI risk management as a system lifecycle problem [2]. Kalai and colleagues’ 2026 Nature analysis provides a deeper mechanism: next-word training and common accuracy-based evaluation incentives can reward guessing over abstention, helping explain why plausible falsehoods persist even in advanced models [12]. TruthfulQA historically demonstrated that fluent language modeling can reproduce common misconceptions rather than truth: in the original benchmark, the best evaluated model was truthful on 58% of questions versus 94% for human respondents [11].

The operational conclusion is not “never trust AI.” Retrieval, tools, model selection, calibration, and evaluation can improve reliability. The conclusion is that trust in model output should not itself be the authorization control for consequential state.

6.2 Human review does not automatically neutralize machine error

Human-in-the-loop language often assumes that a reviewer will catch bad advice. Human-factors research does not justify that assumption. Bansal and colleagues found that explanations increased acceptance of AI recommendations regardless of whether the recommendation was correct, without increasing complementary team performance [8]. Buçinca and colleagues, in an experiment with 199 participants, found that cognitive-forcing interventions reduced overreliance, while the interventions that reduced overreliance most were also rated least favorably [7]. In clinical decision support, Gaube and colleagues found diagnostic accuracy was significantly worse under inaccurate advice; the study reports task experts and non-experts performing 40.10% and 37.53% better, respectively, with accurate rather than inaccurate advice [9].

These findings echo decades of automation research. Bainbridge described the “ironies of automation”: designers can leave humans responsible for abnormal conditions while removing the routine engagement needed to intervene effectively [5]. Parasuraman and Riley formalized misuse, disuse, and abuse of automation and linked overreliance to failures of monitoring and decision bias [6]. Green later argued that human-oversight policy can provide false assurance when institutions fail to justify and evaluate how oversight actually works [10].

CONTROL IMPLICATION A reviewer is not a safety control merely because an approval button exists. The reviewer needs competence, independent evidence, time, authority to refuse, and an enforceable stop or escalation path.

6.3 Security architecture already separates capability from authorization

The separation of capability and authority is not unique to AI. Saltzer and Schroeder’s classic security principles include fail-safe defaults, complete mediation, least privilege, and separation of privilege [4]. NIST Zero Trust Architecture similarly emphasizes least-privilege, per-request access decisions and policy decision/enforcement points rather than implicit trust [3].

Applied to AI-mediated action, the implication is straightforward: model output and tool requests should be treated as untrusted requests that must traverse an independent authorization layer. Prompt safety can reduce risk, but it is not a substitute for authority containment. OWASP identifies prompt injection and insecure output handling as major LLM-application risks [21]. The UK NCSC warns that prompt injection is structurally different from SQL injection because current LLM systems do not provide a reliable separation between instructions and untrusted content [22].

6.4 End-to-end accountability is a lifecycle property

Raji and colleagues’ end-to-end framework for internal algorithmic auditing argues that accountability cannot be reduced to a one-time model assessment; it requires governance across the development and deployment lifecycle [13]. NIST’s AI RMF similarly organizes risk work across GOVERN, MAP, MEASURE, and MANAGE [1]. ISO/IEC 42001 establishes an AI management-system approach rather than a single-model test [23]. These frameworks support the paper’s emphasis on named owners, context of use, release evidence, monitoring, incident response, and change control.

6.5 Safety engineering focuses on control, not component optimism

Systems-safety engineering has long recognized that accidents can emerge from interactions among components even when individual components behave as designed. Leveson’s systems-theoretic safety work treats unsafe control actions, constraints, feedback, and organizational context as central [26]. This reinforces the paper’s state-transition framing: a highly capable component can still be embedded in an unsafe authority structure.

7. What High-Risk Systems Actually Do

The strongest support for consequence-based authority allocation is not corporate messaging about AI adoption. It is the control structure visible in actual regulatory, supervisory, and safety documents. Mature high-consequence domains do use advanced computation and, in some cases, autonomy. They do not treat capability as unrestricted authority.

7.1 Banking: materiality before novelty

The April 2026 joint model-risk guidance from the Federal Reserve, FDIC, and OCC states that model use in banking continues to grow in complexity and scope and is important to competitiveness, while also recognizing substantial risk from decisions based on model output [17]. The guidance is especially useful for this paper because it explicitly distinguishes deterministic rule-based software from statistical models and ties oversight to inherent risk, exposure, purpose, and materiality. Models with higher business impact warrant more comprehensive and rigorous oversight; “effective challenge” requires objective expertise, independence, and organizational standing; governance benefits from clear roles and well-defined accountability [17].

The guidance does not cover generative or agentic AI models, which it describes as novel and rapidly evolving. Rather than weaken the argument, this is instructive: the agencies expressly state that a bank’s risk-management and governance practices should guide the controls applied to tools outside the document’s scope [17]. The durable control logic is therefore risk-and consequence-based, not vendor-or model-generation-based.

7.2 Defense: autonomy is not the absence of authority

DoD Directive 3000.09 is narrower than general defense AI policy: it governs autonomous and semi-autonomous weapon systems. Within that high-consequence context, the directive requires systems to allow commanders and operators to exercise appropriate levels of human judgment over the use of force, undergo rigorous hardware/software V&V and realistic developmental and operational T&E, provide understandable interfaces and transparent status feedback, and provide clear procedures to activate and deactivate functions [16]. Specified autonomous systems require senior approval; legal reviews address consistency with domestic and international law; post-fielding monitoring and additional testing are required when changes may alter risk [16].

This is not an AI prohibition. It is evidence of a stronger principle: as consequence approaches the use of force, autonomous capability is embedded inside explicit authority, verification, testing, legal review, and accountable human judgment.

7.3 EU AI Act: oversight proportional to autonomy and risk

Article 14 of the EU AI Act requires high-risk AI systems to be designed so natural persons can effectively oversee them. The oversight measures must be commensurate with risk, level of autonomy, and context of use. Assigned overseers must be able, as appropriate, to understand limitations, recognize automation bias, correctly interpret outputs, decide not to use the system, disregard or reverse outputs, intervene, and stop the system in a safe state [18]. Article 9 requires an iterative lifecycle risk-management system, and Article 12 imposes record-keeping requirements for high-risk systems [18].

The Act does not adopt this paper’s exact architecture, but it independently validates the distinction between nominal human presence and technically enabled human authority.

7.4 Biomedical and aviation practice: context and lifecycle

The 2026 FDA/EMA Good AI Practice principles explicitly call for a risk-based approach with proportionate validation, risk mitigation, and oversight based on context of use; documented provenance and analytical decisions; complete-system performance assessment including human-AI interaction; and lifecycle monitoring and reevaluation [19]. The FAA AI Safety Assurance Roadmap similarly emphasizes AI-specific assurance methods, conventional safety frameworks, safe integration, workforce readiness, and research [20]. Both domains treat AI as a component inside a safety and quality system.

7.5 The convergence

Across these domains, the recurring pattern is not “AI banned” or “AI trusted.” It is consequence-based governance: define context, identify accountable roles, constrain authority, validate before or around use, monitor performance, preserve evidence, and increase rigor with impact. That convergence supports the paper’s default placement doctrine even though each domain implements it differently.

8. Legal and Institutional Accountability

Architecture does not determine legal liability. Liability depends on jurisdiction, statute, contract, professional duty, facts, and sector. Architecture can, however, preserve or destroy the information and control relationships necessary for legal and institutional accountability to function.

8.1 The accountability chain

ACCOUNTABILITY CHAIN Institutional purpose → policy → delegated authority → authoritative facts → proposal or decision → authorization → execution → observed outcome → evidence → challenge and remedy

A consequential action should be reconstructable across this chain. If policy cannot be identified, the decision lacks a governing basis. If authority cannot be identified, capability may have substituted for permission. If facts cannot be reconstructed, review becomes narrative rather than evidence. If execution is conflated with outcome, the record may be false. If there is no owner of remedy, traceability has documented a failure without establishing accountability.

8.2 AI is not presently a substitute locus of formal responsibility

Current U.S. administrative rules provide concrete examples. IRS EIN administration generally requires the “responsible party” to be an individual natural person rather than another entity [24]. USPTO guidance states that only natural persons can be named as inventors, even when AI tools materially assist the inventive process [25]. The EU AI Act assigns provider and deployer obligations to natural or legal persons, public authorities, agencies, or other bodies rather than to AI models as independent regulated principals [18].

These examples are not a comprehensive theory of AI legal personhood. They demonstrate the present operational asymmetry: software may exercise broad technical discretion while formal obligations continue to attach to recognized people and organizations. That asymmetry is a reason to keep consequential authority technically connected to accountable institutions rather than allowing responsibility to terminate at “the AI decided.”

8.3 Institutional responsibility cannot be delegated by interface design

Placing a human at the last click does not erase organizational responsibility for system purpose, training data, authority design, deployment choices, workload, monitoring, appeals, or vendor integration. Likewise, procuring an AI service does not move the organization’s local authority decisions into the provider’s model card. The institution remains responsible for what its application permits and what consequences it treats as authoritative.

8.4 The objective is legal defensibility, not automated liability allocation

The framework seeks to preserve evidence sufficient for later legal, regulatory, audit, contractual, or internal review. A defensible record identifies policy ownership, system ownership, decision authority, source facts, model contribution, approval, exact execution, observed outcome, exceptions, and remedy. It does not pre-decide who is legally liable; it prevents the system from making that inquiry impossible.

9. Meaningful Human Authority

Human oversight is a system property, not the presence of an approval button. A design that routes every exception to an overloaded operator, shows only the model recommendation, penalizes disagreement, or makes refusal operationally impossible has not preserved meaningful human authority.

9.1 Escalation is not failure

A high-integrity system treats escalation as correct behavior when evidence is inadequate or policy does not cover the case. The alternative - asking AI or a rushed operator to invent an answer - converts an acknowledged unknown into an unauditable commitment.

9.2 Overrides are governed actions

An override is not a hidden escape hatch. It requires permission, reason, identity, timestamp, original failure evidence, affected scope, expiry where appropriate, and follow-up review. Repeated overrides are evidence that policy, data, or workflow may be wrong and should produce a governance signal rather than disappearing into ordinary operations.

10. Runtime Accountability Operating Model

10.1 The control compiler

Organizations often maintain disconnected layers: board policy, risk register, system prompt, provider configuration, application workflow, and audit export. Accountability requires a traceable compilation path across them:

CONTROL COMPILER Named obligation → versioned policy → deterministic rule or autonomy corridor → runtime allow/deny/review decision → bounded execution → external receipt → reconciled outcome → evidence packet → review and improvement

A policy statement that cannot be traced to enforcement behavior is incomplete. An enforcement decision that cannot be traced back to approved policy is unauthorized. An evidence packet that cannot identify the exact policy, input snapshot, model configuration, approval, and outcome is insufficient for replay.

10.2 Authority classes

10.3 Minimum runtime controls

10.4 Provider independence

Provider safety controls, model cards, and contractual commitments are governance inputs, not substitutes for local authority. A structured proposal interface and deterministic authority layer let an organization compare or replace models without moving business rules into prompts. The model remains a component inside a governed system rather than the system of record.

11. Measurement, Validation, and Release Gates

Traditional model metrics remain useful, but they do not answer whether an incorrect output escaped into a commitment, whether policy was bypassed, whether the reviewer could refuse, whether an external operation completed, or whether the organization can reconstruct the event. Accountability metrics therefore cover the whole system.

11.1 Release evidence

A responsible release should bind the system to an exact code version, model/provider configuration, policy version, data contract, test corpus, known limitations, and rollback path. Stale evidence should block release. Reviewers should approve the exact evidence package, not a generic statement that the system “usually passes.”

• Unit and property tests for formulas, units, permissions, state invariants, and policy constraints.

• Golden and historical edge cases with approved expected outcomes.

• Adversarial, malformed-input, prompt-injection, stale-state, replay, duplicate, and control-plane tests.

• Human-factors tests under realistic workload, urgency, uncertainty, and consequence.

• Provider failure, timeout, partial success, unknown-outcome, and reconciliation tests.

• Backup, restore, migration, rollback, incident, and evidence-custody tests.

• Post-deployment monitoring, sampled review, complaint/appeal analysis, and outcome validation.

11.2 Suggested critical release gates

These are engineering targets, not universal regulatory thresholds. For critical actions, a conservative release posture includes zero unexpected allows in the approved critical-action catalog; complete policy/version/evidence linkage; zero stale-policy authorizations; no AI write path to governance assets; successful idempotency and unknown-outcome tests; successful reconciliation; demonstrated human refusal and escalation capability where required; and explicit approval of residual risks and non-claims.

12. Boundary Conditions and Counterarguments

A serious position should survive its strongest objections without dissolving into false balance. The following points do not negate the doctrine; they define where its claims begin and end.

12.1 “Deterministic systems can be wrong.”

Correct. Determinism is not truth. It makes errors stable enough to test, reproduce, audit, compare, and change. A bad rule can systematically harm people at scale. The framework therefore requires provenance, domain validation, monitoring, appeals, change control, and outcome review. AI may be useful for discovering rule gaps, but it should not silently rewrite governing policy in production.

12.2 “Humans are biased and inconsistent.”

Correct. The framework does not treat humans as infallible. In low-ambiguity tasks, deterministic automation may be preferable to manual discretion precisely because humans are inconsistent. Human judgment is reserved for legitimate exceptions and value-laden decisions that the institution has not reduced to validated policy, and those human decisions remain subject to authority, evidence, reasons, appeals, and audit.

12.3 “Autonomous agents need freedom to create value.”

They need freedom inside a problem space, not unlimited authority over the organization. Agents can search, plan, simulate, draft, prioritize, and request actions with substantial initiative while deterministic controls constrain targets, budgets, data, destinations, side effects, and control-plane access. The restriction is on unilateral commitment and self-expanding authority, not on intelligence or initiative.

12.4 “This adds friction.”

Some controls add latency. The relevant comparison is net operational value after duplicates, incidents, rework, audit, remediation, support burden, and loss of trust. Exact low-risk actions can be heavily automated through preauthorized corridors. Friction should increase with consequence, irreversibility, ambiguity, and evidentiary weakness rather than being applied indiscriminately.

12.5 “Future AI may become more reliable than humans.”

The framework does not depend on permanent human superiority. If future models become dramatically more accurate, that can justify broader roles and wider corridors. It does not by itself answer who is authorized to change consequential state, who owns policy, how an exception is appealed, or who remains legally responsible. Capability improvements can change the optimal allocation of tasks without eliminating the need for authority architecture.

12.6 “Not every domain can be reduced to deterministic rules.”

Agreed. The framework explicitly reserves human judgment and bounded autonomy for legitimate uncertainty. When an organization cannot specify an authoritative rule, the correct response is not to pretend the rule exists inside a prompt. The case should remain a judgment, be escalated to an accountable role, or be kept non-authoritative until governance defines the rule.

12.7 Trust assumptions

Any enforcement boundary governs only actions that pass through it. It cannot protect endpoints, administrators, networks, or external systems that bypass the insertion point. Evidence can be falsified if roots of trust, signing identities, hosts, or storage are compromised. High-assurance deployments may therefore require hardware-backed identity, independent custody, multi-party control, external monitoring, and independent validation. Those are implementation trust assumptions, not reasons to move authority into the model.

13. Research Agenda and Falsifiable Hypotheses

The doctrine becomes scientifically useful when its claims are translated into controlled comparisons and measurable system properties. The following hypotheses are intentionally falsifiable.

13.1 Practical evaluation sequence

1. Select one consequential workflow and define the exact commitment, authority owner, failure state, and proof obligation.

2. Implement a baseline architecture and a consequence-based authority architecture while holding the model and task constant.

3. Run routine, malformed, adversarial, stale-state, duplicate, provider-failure, and reviewer-workload cases.

4. Measure productivity, model quality, authority escapes, reviewer correction, evidence completeness, and outcome accuracy.

5. Publish the scenario catalog, policy versions, claim boundaries, and reproduction commands so results can be challenged.

6. Repeat in another domain before generalizing beyond the original context.

14. Conclusion

AI governance will remain incomplete as long as it focuses primarily on what a model may say or how capable an agent appears. The decisive governance question is what the surrounding system permits an AI, human, service, device, or automated process to commit.

The evidence reviewed here supports a strong default: use AI where ambiguity, language, prediction, and search create value; automate exact obligations wherever rules can be validated; permit autonomous discretion inside bounded and recoverable corridors; retain accountable human authority for exceptions and high-impact judgment; and keep institutional control over policy, permissions, release, appeals, and remedy. As consequence and irreversibility rise, machine discretion should contract while validation, deterministic constraints, explicit authority, reconciliation, and evidence should become stronger.

This is not a call to slow AI adoption. It is a way to adopt more of it without confusing intelligence with authority. A bank can use models extensively without making model output sovereign. A defense system can employ autonomy while retaining human judgment, V&V, legal review, and explicit approval. A medical-development system can use AI while requiring context, provenance, validation, and lifecycle monitoring. An organization can automate nearly everything it can define without turning the business itself into an unaccountable autonomous actor.

FINAL PROPOSITION A system may act automatically without acting autonomously. It may use intelligence without surrendering authority. Accountability is preserved when every consequential commitment remains governed, attributable, observable, challengeable, and provable.

Appendix A. Placement Decision Tree

1. Does the task require flexible interpretation of language, images, incomplete evidence, or unfamiliar patterns? If no, prefer deterministic software.

2. Will the output remain informational, simulated, or easily reversible without changing authoritative or external state? If yes, AI assistance may be appropriate with provenance and labeling.

3. Can the output change money, rights, safety, access, production, retention, legal status, public claims, or another consequential state? If yes, it must traverse explicit authority and validation.

4. Are the commitment rules exact, validated, and approved? If yes, automate them deterministically. If no, block or route to accountable judgment; do not ask AI to invent the governing rule.

5. Does autonomy add value? If yes, define an explicit corridor: actions, targets, tools, data, budget, time, prohibited states, evidence, reversibility, and stop conditions.

6. Is the action irreversible, high-impact, exceptional, or legally sensitive? Require stronger evidence, explicit approval, dual control where appropriate, and a recovery or appeal path.

7. Can the reviewer independently evaluate the case, refuse, and stop execution? If no, oversight is not meaningful; redesign or block.

8. Can execution be idempotent, bounded, observed, and reconciled? If no, do not represent success and do not grant autonomous execution.

9. Can the system preserve exact facts, versions, policy, approval, receipt, outcome, and limitations? If no, accountability is not sufficient for the action.

Appendix B. Model AI Policy Clauses

The following language is a starting point for internal policy and should be adapted to the organization’s legal, security, privacy, labor, safety, and sector obligations.

Purpose

The organization uses AI to augment interpretation, research, drafting, classification, retrieval, prediction, and recommendation. AI capability does not by itself create operational authority.

Non-authoritative output

AI output is treated as a proposal or evidence input. It does not become an authoritative fact, decision, approval, or completed state solely because it is fluent, confident, structured, or generated by an approved provider.

Consequential actions

Financial, legal, safety, rights-relevant, access, production, retention, deletion, compliance, and other materially consequential actions must pass through approved authority and validation controls.

Bounded autonomy

Autonomous components operate only within explicit action, target, data, tool, budget, time, destination, reversibility, and evidence corridors. They may not expand their own authority.

Fail-closed uncertainty

Unknown, malformed, stale, conflicting, expired, unapproved, or policyless requests are denied, quarantined, or held for review. The system must not invent a plausible default for an authoritative rule.

Control-plane isolation

AI systems may not unilaterally modify their own permissions, governing policies, approval records, signing identity, evidence requirements, audit lineage, or release gates.

Human authority

Required reviewers must be trained, sufficiently equipped, able to inspect independent evidence, able to refuse or halt action, and supported by workload and escalation controls.

Execution and reconciliation

Actions must be bounded, monitored, duplicate-safe where feasible, and followed by outcome verification. Attempted, accepted, delivered, settled, deployed, and completed states must not be conflated.

Evidence and challenge

Consequential actions preserve a replayable record of request, sources, versions, policy, decision, approvals, execution receipt, external outcome, exceptions, and claim boundary. Appropriate challenge or appeal paths are maintained.

Change and incident management

Material model, provider, prompt, policy, tool, data, or workflow changes require versioning, evaluation, approval, release evidence, monitoring, rollback, and incident response proportionate to risk.

Appendix C. Minimum Evidence Packet

Appendix D. Claim-to-Evidence Map

This map makes the paper’s evidentiary structure explicit. A source is used only for the claim class it actually supports; the final column identifies the principal inference that should not be drawn from that evidence.

References

[1] National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, Jan. 2023. doi: 10.6028/NIST.AI.100-1.Read original source

[2] C. Autio et al., Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, July 2024. doi: 10.6028/NIST.AI.600-1.Read original source

[3] S. Rose, O. Borchert, S. Mitchell, and S. Connelly, Zero Trust Architecture, NIST SP 800-207, Aug. 2020. doi: 10.6028/NIST.SP.800-207.Read original source

[4] J. H. Saltzer and M. D. Schroeder, “The Protection of Information in Computer Systems,” Proceedings of the IEEE, vol. 63, no. 9, pp. 1278-1308, 1975. doi: 10.1109/PROC.1975.9939.Read original source

[5] L. Bainbridge, “Ironies of Automation,” Automatica, vol. 19, no. 6, pp. 775-779, 1983. doi: 10.1016/0005-1098(83)90046-8.Read original source

[6] R. Parasuraman and V. Riley, “Humans and Automation: Use, Misuse, Disuse, Abuse,” Human Factors, vol. 39, no. 2, pp. 230-253, 1997. doi: 10.1518/001872097778543886.Read original source

[7] Z. Buçinca, M. B. Malaya, and K. Z. Gajos, “To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-Assisted Decision-Making,” Proc. ACM Human-Computer Interaction, vol. 5, CSCW1, Art. 188, 2021. doi: 10.1145/3449287.Read original source

[8] G. Bansal et al., “Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance,” CHI 2021. doi: 10.1145/3411764.3445717.Read original source

[9] S. Gaube et al., “Do as AI say: susceptibility in deployment of clinical decision-aids,” npj Digital Medicine, vol. 4, Art. 31, 2021. doi: 10.1038/s41746-021-00385-9.Read original source

[10] B. Green, “The flaws of policies requiring human oversight of government algorithms,” Computer Law & Security Review, vol. 45, Art. 105681, 2022. doi: 10.1016/j.clsr.2022.105681.Read original source

[11] S. Lin, J. Hilton, and O. Evans, “TruthfulQA: Measuring How Models Mimic Human Falsehoods,” ACL 2022, pp. 3214-3252. doi: 10.18653/v1/2022.acl-long.229.Read original source

[12] A. T. Kalai, O. Nachum, S. S. Vempala, et al., “Evaluating large language models for accuracy incentivizes hallucinations,” Nature, vol. 653, pp. 1047-1051, 2026. doi: 10.1038/s41586-026-10549-w.Read original source

[13] I. D. Raji et al., “Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing,” FAccT 2020, pp. 33-44. doi: 10.1145/3351095.3372873.Read original source

[14] S. Amershi et al., “Guidelines for Human-AI Interaction,” CHI 2019, pp. 1-13. doi: 10.1145/3290605.3300233.Read original source

[15] National Academies of Sciences, Engineering, and Medicine, Human-AI Teaming: State-of-the-Art and Research Needs. Washington, DC: National Academies Press, 2022. doi: 10.17226/26355.Read original source

[16] U.S. Department of Defense, DoD Directive 3000.09, Autonomy in Weapon Systems, Jan. 25, 2023.Read original source

[17] Board of Governors of the Federal Reserve System, Federal Deposit Insurance Corporation, and Office of the Comptroller of the Currency, Supervisory Guidance on Model Risk Management, SR 26-2, Apr. 17, 2026.Read original source

[18] European Parliament and Council of the European Union, Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Official Journal of the European Union, 2024; consolidated text accessed Aug. 26, 2026.Read original source

[19] U.S. Food and Drug Administration and European Medicines Agency, Guiding Principles of Good AI Practice in Drug Development, Jan. 2026.Read original source

[20] Federal Aviation Administration, Roadmap for Artificial Intelligence Safety Assurance, 2024.Read original source

[21] OWASP Foundation, OWASP Top 10 for LLM and Generative AI Applications, 2025 edition, including LLM01 Prompt Injection.Read original source

[22] UK National Cyber Security Centre, “Prompt injection is not SQL injection (it may be worse),” Dec. 8, 2025.Read original source

[23] ISO/IEC 42001:2023, Information technology - Artificial intelligence - Management system, 2023.Read original source

[24] Internal Revenue Service, “Responsible parties and nominees,” stating that an EIN responsible party must be a person, not an entity, except for government entities; accessed Aug. 26, 2026.Read original source

[25] U.S. Patent and Trademark Office, “Revised Inventorship Guidance for AI-Assisted Inventions,” examination guidance, 90 Fed. Reg. 54636-54637, Nov. 28, 2025.Read original source

[26] N. Leveson, Engineering a Safer World: Systems Thinking Applied to Safety. Cambridge, MA: MIT Press, 2011.Read original source

Source-access note

Official and research sources were checked as of 26 August 2026. Corporate media releases were not used as primary evidence for safety, adoption, system performance, or compliance claims. Where current law or regulatory interpretation matters, readers should verify the latest consolidated text and sector-specific requirements before relying on this working paper.

KEEP A COPY

Download the complete paper.

Enter your email to download the original PDF. No account required.

Email address You may contact me about TYGGO research and future papers. (Optional)

Website

We record your email and requested paper to provide this download. Further contact is optional. Privacy policy.

Get the PDF