The First Rule of an AI Incident Is Do Not Delete Anything

Seven months

In October 2022, the executive director of an eating disorders nonprofit in Massachusetts contacted a larger national organization to report a problem with its chatbot. The tool, meant to support people seeking help, was producing language she found troubling.

Nothing happened.

In late May 2023, a consultant tested the same chatbot and found it giving weight-loss guidance to people reaching out about eating disorders. She posted screenshots publicly. Within days the organization disabled the tool, and a week after that its chief executive published a statement explaining what had occurred, because the shutdown had collided with an unrelated decision to close the organization's long-running helpline and the public had fused the two events into one story.

The chatbot had originally been built as a rules-based tool by academic researchers. Generative capability was added later. The organization's position was that it was never advised of the change and would not have approved it. The vendor's chief executive said the change was within the contract and had been implemented as a systems upgrade.

Set aside who was right about the contract. The organizational failure is visible without resolving that.

An external expert reported the problem through an informal channel and it went nowhere for seven months. Not because anyone ignored it. Because there was no definition of what she was reporting, no route for it to travel, and no person whose job it was to receive it. The failure was not the chatbot. The failure was that the report had nowhere to go.

That is the situation most nonprofits are in right now, and it is the reason this article exists.

You cannot respond to something you have not defined

Ask a reasonable question. A staff member pastes a spreadsheet of client names, addresses and dates of birth into a free consumer chatbot to summarize it. Is that an incident?

Run it past the four institutions that have actually published definitions and you get four different answers.

The AI Incident Database, run by the Responsible AI Collaborative, defines an AI incident as "an alleged harm or near harm event to people, property, or the environment where an AI system is implicated." Under that definition, this is arguably an issue rather than an incident, because no harm has yet occurred or been detected.

The OECD, in Defining AI incidents and related terms, proposes a draft definition whose harm categories include "a breach of obligations under the applicable law intended to protect fundamental, labour and intellectual property rights." Under that definition, if the paste breached a data protection obligation, it is an incident.

NIST, in Special Publication 800-61 Revision 3, carries forward the long-standing federal definition of a cybersecurity incident, which includes "a violation or imminent threat of violation of law, security policies, security procedures, or acceptable use policies." So under NIST, the paste is an incident if you have an acceptable use policy it violated. If you have no policy, there is nothing to violate. More to the point, NIST says response begins when an event meets the defined incident criteria, and the organization defines those criteria itself. No criteria, no trigger.

The NIST AI Risk Management Framework, the most cited AI governance framework in the United States, requires organizations to maintain incident response processes, to communicate incidents to affected communities, and to keep "a database of reported errors, near-misses, incidents and negative impacts." It never defines an AI incident. It uses the term as a given.

This is not a criticism of any of the four. They were written for different purposes, and none of them was written for a nonprofit. But it explains something worth understanding clearly. The reason your organization has no AI incident response is not negligence. It is that nobody has handed you a definition you could operate, and the most influential framework in the field asks you to respond to a category it does not describe.

So define it yourself, and define it narrowly enough to be usable. Five events, any one of which starts a response at a small organization:

• Constituent, donor or client information leaves your perimeter through an AI tool.
• An AI-facing tool gives someone you serve guidance that is wrong, harmful, or unlawful.
• An AI-assisted document containing fabricated content goes to a funder, regulator or court.
• A decision about a person that AI influenced is challenged.
• A tool you already use quietly acquires AI capability you never approved.

That fifth one is how the eating disorders chatbot became a crisis, and it is now the most likely way a nonprofit acquires AI risk it never agreed to.

The counterintuitive part

Here is where the standard advice goes wrong, and it goes wrong in a way that costs organizations real protection.

Every serious legal question an AI incident raises is a question of fact. Was there unauthorized access. Is there a low probability that protected information was compromised. Is misuse likely. Those are not judgment calls a lawyer makes in the abstract. They are determinations made from evidence.

And in an AI incident, the evidence lives somewhere you do not control and expires on a schedule you did not set.

Consider what the rules actually require. Under the HIPAA breach rule, an impermissible disclosure of protected health information is presumed to be a breach unless the organization demonstrates a low probability of compromise, using a four-factor assessment. One of those four factors is "whether the protected health information was actually acquired or viewed." That fact lives in a vendor's logs.

New York's breach notification statute offers an exception for inadvertent disclosure by an authorized person where the business reasonably determines misuse is unlikely. But the determination must be documented in writing, retained for five years, and if more than five hundred New York residents are affected, provided to the Attorney General within ten days of the determination. Florida offers a similar harm exception, also requiring a written determination, also filed with the state.

Every one of those escape hatches requires you to prove something. And proving it requires the record.

Now consider what a well-meaning staff member does in the first ten minutes after realizing they made a mistake. They delete the chat.

That instinct destroys the only evidence that would have supported a defensible written determination that no notice was required. It converts a documented non-event into an undocumented unknown, and an undocumented unknown is the thing you have to notify about.

The first rule of the first hour is that nobody deletes anything. Not the chat, not the account, not the file. Preserve first, decide second. This is close to the opposite of what people's instincts tell them, and it is why the rule has to be written down before the incident rather than explained during it.

The clock you are actually racing

The vendor clock is the other half of the problem, and it does not run at the same speed everywhere.

OpenAI and Anthropic both document that a deleted conversation is removed from back-end systems within thirty days, subject to legal and safety exceptions they control and you do not. Google and Microsoft work differently. Google keeps Gemini activity for eighteen months by default, and conversations that a human reviewer has already read are kept for up to three years and are not deleted when the user deletes their activity. Microsoft 365 Copilot prompts and responses are stored in a hidden folder in the user's Exchange mailbox and survive for as long as your administrator's retention policy says, which may be indefinitely.

The "subject to legal exceptions" clause is not hypothetical, and there is now a documented case of it. In June 2025, a court order in the New York Times litigation required OpenAI to retain consumer ChatGPT and API content that users had deleted, including Free, Plus, Pro, Team and standard API data. The obligation ran until late September 2025, and OpenAI announced the following month that it had returned to its standard retention practices. For roughly four months, the deletion setting a nonprofit relied on did not do what it appeared to do, because of litigation the nonprofit was not party to and could not have anticipated.

So check the tool your staff actually used, check the account tier, and check it on the day. You are not racing a regulator in those first hours. You are racing a retention policy you did not write and may not have read.

Whether you must notify is genuinely unsettled

I am not going to tell you that pasting donor data into a consumer chatbot is a reportable breach, because that is not a settled question and anyone who tells you otherwise is guessing.

The case that it is reportable is not weak. New York's statute reaches unauthorized "access to or acquisition of" private information, and its own factors for access include information being "viewed, communicated with, used, or altered" by an unauthorized person. Data typed into a consumer tool has been communicated with a system belonging to a party you have no agreement with. And the good-faith employee exceptions that appear in California and Massachusetts law are conditional: they protect good-faith acquisition by an employee only where the information is not subject to further unauthorized disclosure. Transmission to a third-party model provider is arguably that further disclosure. The employee's good faith does not necessarily survive the onward transmission.

The case that it is not reportable is also not weak. California requires acquisition by an unauthorized person, and whether ingestion by an automated system meets that is contestable. Massachusetts requires a substantial risk of identity theft or fraud. New York's inadvertent disclosure exception was written for something adjacent to this. And an enterprise account with contractual terms, no training on inputs and a defined retention period is a materially different fact pattern from a personal free account.

Which is the point. The answer depends on which tool, which account tier, which terms were in force that day, what the retention setting was, and whether the conversation still exists. Those are the facts that stop being recoverable within days.

Your first 48 hours exist to preserve them.

What the first 48 hours look like

Hour one. Preserve. Nobody deletes. Screenshot the conversation, note the exact tool, the account, whether it was a personal or organizational login, and the date and time. If a public-facing tool is producing bad output, take it offline. A disclaimer is not containment. When New York City's small business chatbot was found telling businesses they could take workers' tips and that landlords could turn away tenants with housing vouchers, the city added a beta label and a disclaimer telling users not to treat its answers as legal or professional advice, and left the tool running. Days later reporters asked the bot whether it could be used for professional business advice. It said yes.

Hours one to four. Name one person to own the response. Not a committee. Establish what data was involved, how many people it concerns, and which states they live in, because that determines which clocks are running.

Day one. Get advice before you conclude anything. If you hold health information, the presumption runs against you and the four-factor assessment needs to start now. If federal award work is involved, note that 2 CFR 200.303 requires every recipient and subrecipient of a federal award to "take prompt action when instances of noncompliance are identified" and to take "reasonable cybersecurity and other measures to safeguard information including protected personally identifiable information (PII)." That is an internal control obligation with no size threshold and no harm test attached to it.

Day two. Write down what you found and what you decided, and why. If you are relying on a harm exception, that written determination is not paperwork. It is the exception.

Two cautions on things people assume. There is generally no free-standing legal duty to notify a funder of an AI incident. Those duties are contractual, so read the grant agreement rather than assuming. The federal mandatory disclosure rule at 2 CFR 200.113 is narrow, confined to credible evidence of federal criminal fraud, conflict of interest, bribery or gratuity violations, or a civil False Claims Act violation. And the one scenario that changes character entirely is an AI-drafted report containing fabricated figures submitted to a federal funder. At that point the question stops being whether an AI made an error and starts being whether there is credible evidence of a false claim.

What the board needs, and it is not the policy

Boards are being told to adopt AI policies. That is worth doing and it is not the thing that failed in the eating disorders case, where the gap was that a credible external report had no route to anyone with authority.

The duty of care is a process standard. New York's Not-for-Profit Corporation Law puts it as discharging one's duties "in good faith and with the care an ordinarily prudent person in a like position would exercise under similar circumstances." Delaware's corporate case law, which is influential rather than binding on most nonprofit boards, has developed the same idea further. In Marchand v. Barnhill, the Delaware Supreme Court held that oversight doctrine "does require that a board make a good faith effort to put in place a reasonable system of monitoring and reporting about the corporation's central compliance risks," and found a board had failed on exactly that point even though compliance activity existed at management level, in a case where the risk was "essential and mission critical."

For a nonprofit, the mission-critical risk is usually the safety, dignity and privacy of the people you serve. AI now touches that directly.

So the board's deliverable is not another policy. It is an escalation standard. Which categories of AI event reach the board chair within 24 hours rather than the next quarterly meeting. Who is authorized to take a public-facing tool offline without waiting for permission. What gets written down. Where an outside expert's warning goes when it arrives by email on a Tuesday.

Seven months is what happens without one.

Charts on this page were produced by the Center for Nonprofit AI from the cited sources.