Black-Box Prompt Optimization: Accountability Gaps and the Ethics of Automated LLM Behavior Control

The first time I encountered Prompt Optimization described as automatic differentiation via text, I recognized the engineering elegance immediately. A system that propagates textual feedback through a pipeline, adjusting instructions the way gradient descent adjusts weights, is a genuine intellectual achievement. But elegance in mechanism can obscure what a mechanism does at scale. The more I read about how these tools work, the more one question kept surfacing — a question the benchmark papers never answer: what happens to human accountability when the behavior of AI systems is determined by an optimization loop rather than a human decision?

What Gets Lost When No One Writes the Prompt

There is something philosophically significant about a prompt that no human authored. In the traditional model — a person writes an instruction, a model follows it, an outcome emerges — the chain of intent is at least traceable. The human made a choice, however imperfectly. That choice can be examined, contested, revised. The person who wrote it can be asked to explain it.

Tools like Opro, which uses the LLM itself as an iterative optimizer over candidate instructions, and Textgrad, which backpropagates textual feedback through compound AI systems and was published in Nature (TextGrad Paper), are not marginal experiments. OPRO demonstrated gains of up to 50% on Big-Bench Hard compared to human-designed prompts (OPRO Paper). DSPy, which compiles declarative pipeline specifications into self-improving prompt sequences, has accumulated 25,000+ GitHub stars and is in production at companies like Cursor, Databricks, and Mistral — though it is under rapid development, with breaking changes introduced in recent releases (DSPy Docs). The ecosystem extends further: Prompt Compression reduces token footprint algorithmically, Structured Output Prompting enforces response shapes through JSON Schema constraints, and Constrained Decoding guarantees syntactic compliance at the generation layer. Automation removes human authorship from the instruction.

The practical effect is that the prompt — the most consequential governance layer in an LLM-based system — is increasingly assembled by processes that neither the deployer nor the end user initiated, read, or endorsed. It simply exists, optimized, as the effective policy of the system.

The Efficiency Argument, Made Honestly

The case for automating prompt design is not trivial, and dismissing it too quickly would be a failure of intellectual honesty. Human-authored prompts are inconsistent, difficult to maintain across system updates, and chronically under-specified. When a system handles a wide range of query types, the premise that a human can craft optimal instructions for each is a fiction sustained by convenience, not evidence.

Automated tools address real constraints. They find instruction patterns that generalize more reliably than intuition-driven prompts. They make Prompt Versioning tractable — enabling organizations to track which instruction changed, when, and whether the change improved measurable outcomes. They enable systematic Prompt Testing And Evaluation at a scale no human review process can match. Automation encodes more care than intuition. An automated approach subjected to adversarial testing might produce more equitable outcomes than an intuitive one, precisely because it forces the assumptions into the open.

I want to hold this position genuinely, because collapsing it into a strawman defeats the purpose of this argument. The organizations adopting these frameworks are not being reckless. They are solving a resource constraint that human-only prompt engineering cannot solve at enterprise scale.

The Hidden Assumption Inside Every Benchmark

And yet the steelman for automated optimization rests on an assumption so deeply embedded that most practitioners never articulate it: that the metric being optimized is an adequate proxy for the goal we actually care about.

OPRO optimizes for benchmark performance. TextGrad optimizes for whatever loss function the engineer defines. Each framework is rigorous within its own frame — and invisible outside it. Prompts are not neutral mechanisms. Research presented at ACM FAccT 2025 tested six commercial LLMs across fifty demographic groups and found that system prompts create representational and allocative biases that users cannot detect or counteract (ACM FAccT 2025, “Position is Power”). The paper’s conclusion — that “system prompt analysis must be incorporated into AI auditing processes” — is a direct response to this invisibility. Hidden prompts allow bias to accumulate not through malicious intent, but through the structural gap between what a benchmark measures and what the world requires.

The Prompt Injection attack surface points to the same structural issue from a security direction. When attackers exploit the gap between what a system believes it is executing and what it is actually doing, the mechanism mirrors the bias problem: an instruction layer the user cannot inspect is determining an outcome the user is responsible for. OWASP ranks prompt injection as the number one risk in its LLM Top 10 for 2025–2026, noting that each organization in the AI supply chain can add prompts that alter model behavior downstream — often without the awareness of deployers or users (OWASP Gen AI). Automated optimization does not introduce this gap. It inherits and deepens it, because the optimized prompt is more persuasively constructed and less legible to human oversight.

Governance Without Governors

History offers a familiar shape for this problem. The governance scholar knows that some of the most consequential rules are not the ones that parliaments vote on but the ones that accrue through administrative procedure, technical standards, and implementation choices made by people who believed they were making only local decisions. The effective policy is often not the written rule — it is the default behavior that everyone comes to rely on without anyone having decided to authorize it.

Automated prompt optimization is performing this function at scale and at speed. The system prompts governing AI behavior in enterprise environments are increasingly the output of optimization loops, not authorial choices. They have performance records but no authors. They have version histories — when prompt versioning practices are even in place — but no sponsors who can be asked to justify what the text encodes. Each organization in the supply chain adds layers of instruction that propagate downstream, often invisible to the deployers and users whose experience they determine. Policy accrues without deliberation. This is governance without governors.

The gap is not that the tools are sophisticated. It is that the institutional infrastructure for oversight has not caught up with what the tools can do.

The Accountability Gap Is Structural

Thesis: When prompt behavior is determined by optimization rather than human judgment, the ethical responsibility for that behavior becomes structurally untraceable — and no technical improvement in prompt testing and evaluation alone can close that gap, because the problem is institutional, not architectural.

This claim is uncomfortable because it implies the problem is not one engineering iteration away from resolution. Better evaluation frameworks narrow the scope of unknowns; they do not resolve the question of who bears liability when a well-evaluated, well-versioned, properly tested prompt produces harm at scale. GDPR Article 22 establishes the value of human review for automated decisions that materially affect individuals — a principle rooted in the philosophy of autonomy, not in a specific claim about AI. The EU AI Act, in force since August 2024, extends transparency obligations into high-risk deployment contexts. Neither framework was designed with the specific structure of automated prompt optimization in mind — a process that is not a decision in the legal sense but that shapes every decision the system makes afterward. The regulatory trajectory names a direction. The destination has not been built.

The accountability question is not abstract. Someone will be denied a loan, misrouted in triage, or given incorrect guidance — and the chain of decisions that produced the outcome will terminate in a prompt that an optimizer found and nobody chose.

Questions Worth Sitting With

I am not arguing against automated prompt optimization as such. The efficiency case is real, and prohibiting effective tools rarely improves the outcomes for the people those tools affect. What I am proposing is that practitioners, organizations, and regulators ask a different set of questions when evaluating these systems.

Not only whether a prompt improves benchmark performance — but what the benchmark cannot see, and who decided it did not need to. Not only whether a version history exists — but whether anyone in the organization can read that history and explain what a given prompt encodes about the world. Research on Reflexive Prompt Engineering (Djeffal, FAccT 2025) frames the direction well: prompt engineering is “a crucial bridge between AI development and deployment,” and ethical considerations must be integral to that bridge, not added after optimization is complete (ACM FAccT 2025, Reflexive). The governance that matters most is invisible. It hides in defaults. It accrues through iteration. Who in your organization is responsible for what your optimized prompts say about the world — and how would you know if that changed?

Where This Argument Is Weakest

The case I have made depends on the premise that human-authored prompts offer more meaningful accountability than optimized ones. That premise is worth challenging directly. Human engineers write biased prompts with confident intentions. Human intuition bakes in assumptions that systematic evaluation can sometimes surface and correct. An automated approach with rigorous adversarial testing might, in practice, produce more auditable outcomes than an intuitive one — precisely because it forces the assumptions into explicit, measurable form.

I may also be holding automation to a standard I do not apply equally to human practice. The accountability gaps in AI systems are not uniquely products of automation — they are products of opacity, scale, and the absence of institutional review. What automated prompt optimization does is make existing gaps visible in a new form. And that visibility, uncomfortable as it is, may be more tractable than the gaps that human-only processes conceal behind individual judgment.

The Question That Remains

The most unsettling consequence of automated prompt optimization is not that it will produce harmful outputs — harmful outputs already exist, authored by humans. The unsettling part is that when it produces them, the harm will arrive with the full authority of a system that passed its benchmarks, versioned correctly, and optimized in good faith toward the specified goal. Who bears responsibility for outcomes that emerge from processes no single person designed, no institution audited, and no affected party was told about? That question deserves an answer before the technology makes it moot.