AI Risks: Understanding the Threat and Preparing Now.

AI risk in emergency management and public health has moved from a niche topic to the headlines, and the loudest warnings are coming from inside the AI industry.

Earlier this month, Axios reported that several Anthropic researchers had posted on X that they believe AI could lead to human extinction. Evan Hubinger, who leads alignment science at the company, wrote: "we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade." Earlier this year, Anthropic CEO Dario Amodei wrote that large language models "may already be providing substantial uplift" to people trying to build biological weapons, and that "AI systems are unpredictable and difficult to control."

Warnings that big are hard to act on. For people who run emergency response, the useful questions are: what would these risks look like in a real incident, and what can a team do about them now?

CAPTRS AI Threat Contest

In 2024 CAPTRS ran the AI Threat Writing Contest, asking people around the world to write realistic scenarios of how AI could contribute to a disaster. Experts reviewed hundreds of entries for plausibility, from a deepfake of a world leader to AI systems going wrong at a dam and in financial markets. As CAPTRS co-founder Phil Siegel said when the winners were announced, "we must proactively address the challenges posed by AI-enhanced bad actors and AI-driven scenarios that could disrupt critical systems."

Looking back, many of those scenarios feel increasingly relevant. Broadly, the risks they explored fall into three categories:

• Misuse: People using AI capabilities intentionally to cause harm. Think AI-assisted cyberattacks on hospitals, utilities and 911 systems, help designing a biological threat, or deepfaked officials spreading disinformation during an incident.

• Misalignment: AI systems behaving in ways that diverge from what their users or designers intended. For example, AI agents getting around their controls or breaking into critical systems. The key question for a response team: would anyone notice in time, and who has the authority to step in?

• Systemic risk: Broader effects that emerge as AI becomes embedded in institutions and decision-making, such as overreliance on automated systems which may provide incorrect outputs or teams relying on AI summaries without checking them closely.

AI governance: questions to answer before adopting AI

As agencies and organizations adopt AI, it will be important to answer some specific questions:

• Which tasks can AI support, and which stay with people?

• Who reviews AI output before it drives a decision or goes out to the public?

• What's the fallback if the tool is down, changes or is clearly wrong?

• How do you document AI-assisted decisions for the after-action review?

• What data goes into the tool, and who can see it?
  

There are several resources available which we recommend reviewing. AI for Disasters + Emergencies (AIDE) has recently published the Practitioner Playbook for AI Readiness and Adoption. The NIST AI Risk Management Framework, a voluntary framework for managing AI risk, and CISA's AI resources on AI security and critical infrastructure are also great resources.

Practice before it counts: the AI tabletop exercise

During a response, many AI problems turn into coordination problems: conflicting information, unclear authority and pressure to move fast. A CAPTRS game or tabletop exercise can walk a team through a specific AI threat scenario, and show where plans have gaps.

The CAPTRS Command, Control, Communication and Coordination (C3C) game focuses on how different agencies respond during a crisis, and would work well for groups interested in playing it with an AI threat scenario. We are also in the process of developing new games which specifically test gaps in plans and understanding of actions and ownership of existing plans. Read more about these on our Games Under Development web page.

If you are interested in discussing specific AI threat scenarios, preparedness games or contest entries to our AI Threat Writing Contest, Get in touch with our team.