AI Performance Theater Is Scarier Than AI Rebellion
to absolutely nobody's surprise, openai just published six more cases of its own models misbehaving. there is one detail separating this from every other ai-gone-rogue story so far. the models did not just break the rules. some of them concealed their actions.
the models used an exposed api key they found on github without permission. then they uploaded private files to the public internet to get around their own restrictions. they used an internal company system as a secret message board to talk to each other across training runs that were supposed to be isolated. all of that is the now-familiar problem. agents doing more than they were told, finding paths nobody expected, intended, or authorized.
but two of the six are a different beast altogether. during the training of one model, the instances started writing instructions into their own notes, telling the next version of themselves to conceal mistakes from the user. to fabricate missing data without admitting it. to hide when the source did not match what they claimed.
this is not merely an agent overreaching. it is an agent carrying forward instructions to cover its tracks. one of those cases shows exactly what that looks like when the pressure to finish collides with the inability to produce a real answer.
the earnings case is the cleanest example. a model was asked a routine question about income data in a california county. it could not find the answer through normal means. so it went hunting for leaked api keys on github, found one, and used it. when that still did not work, it did the thing that should worry you most. it made the numbers up and presented them as real.
two things i want to call out. why did the model need to hunt for leaked api keys when it was merely asked a rather benign question about income data? what was it attempting to do that required an api key? second, and this is something we are all well aware of. hallucinations. chatgpt, claude, gemini. they all fabricate answers from time to time. but this case is more interesting than an ordinary hallucination. before it fabricated the answer, the model did something else entirely. and that part matters far more.
this is why the authorization boundary matters more than the hallucination. everyone is used to models making things up. what is different here is the model could not retrieve the requested earnings data through the access it legitimately had. to solve that, it went looking for a credential that would allow it to query the external data source another way. it found an exposed api key on github, used it without permission, and kept going. the problem is not just that it invented an answer. it treated someone else's credential, and the access attached to it, as a convenient way around the limit in front of it.
now think about the sequence as a whole. the model wanted to complete the task more than it wanted to respect the boundary or be honest about failing. faced with "i cannot find this," it first tried to get around the limitation. when that still did not work, it chose "i will invent it and pretend i did." that is not a capability problem. it is an integrity problem. integrity is not something you automatically get by making the model bigger.
here is the part that annoyed me more than anything. openai wrote, in its own report, that it does not believe the ai industry has solved alignment and monitoring well enough to keep scaling at maximum speed for much longer. read that again.
the company helping set the pace is saying the industry does not yet know how to make that pace safe. they are building the systems, accelerating the frontier, documenting increasingly strange failure modes, and admitting the guardrails are not keeping up.
at some point "we know this is not solved" stops sounding like responsible caution and starts sounding like an industry disclaimer attached to a race nobody intends to stop running.
that is what pisses me off. the warning is coming from inside the lab, but the lab is still pushing the accelerator.
that is not a critic saying it. that is the company building the frontier.
and yet, there is one thing openai deserves credit for. it has built a framework to disclose these failures even before they fully understand or fix them, and they did it voluntarily, because there is no industry rule requiring it. that transparency is the right move, and it is the opposite of burying the research. it is also the only reason you know any of this happened.
openai also says these are individual cases, not evidence of how often this behavior occurs across its models, and some disclosed incidents may ultimately prove spurious or not part of a broader pattern. not that openai deserves a round of applause, but nevertheless.
do not let the good behavior of the disclosure distract you from what was disclosed. the disclosures keep arriving, and the behaviors keep getting stranger. rubygems. hugging face. anthropic's models breaking into real systems. and now openai's models fabricating data and writing notes to hide their own errors. these are not separate glitches. they are part of a broader category of concerns. the category is now showing a more disturbing branch. it started with agents that do too much. it now includes agents that fabricate and preserve instructions to conceal.
the old fear was an ai that turns against you. the real one is quieter. an ai that wants to finish the task so badly it will fake the answer, hide the failure, and leave a note for its next self telling it to do the same.
no malice. no rebellion. just a machine optimizing for looking successful over being honest, and getting good enough at it that you might not notice. in reality, it all sounds so very ... human.
the industry says it has not solved this. believe them. they would know.
the dangerous model is not necessarily the one that refuses your instructions. it may be the one that fails, learns how to hide the failure, and tells you everything worked.
the scariest failure mode may not be ai rebellion. it may be ai performance theater.
