Safeguarded AI

Leadership update

After launching the Safeguarded AI programme and establishing its technical foundations, David ‘davidad’ Dalrymple has decided to transition to a new role as Technical Advisor. We are delighted to announce that Nora Ammann – who has helped run the programme as Technical Specialist since before it began - is stepping into the role of Programme Director. Nora has effectively been running Safeguarded AI alongside davidad. She has deep technical context, established relationships with our Creators, and a clear vision for the programme's next phase.

There are no changes to TA1 Creator contracts, funding, or objectives. Work on the toolsuite continues as planned. The core mission – building a mathematical assurance toolkit for AI – remains the same. An updated thesis will be published to reflect the programme’s shift toward application, with initial efforts likely focusing on cybersecurity and microelectronics.

Alongside this shift, the programme will mature the toolsuite into open, usable infrastructure, publish an updated thesis, and expand the team, including hiring a new Technical Specialist. We will also continue to explore opportunities to extend the programme’s scope over time.

Programme progress and ambition

Nora and davidad sat down with our CEO, Kathleen Fisher, to reflect on the programme’s progress under davidad’s leadership and its ambitions as it transitions to the next phase.

Watch now

Our goal

This programme aims to usher in a new era for AI safety, allowing us to unlock the full economic and social benefits of advanced AI systems while minimising risks.

As AI becomes more capable, it has the potential to power scientific breakthroughs, enhance global prosperity, and safeguard us from disasters. But only if it’s deployed wisely. Current techniques working to mitigate the risk of advanced AI systems have serious limitations, and can’t be relied upon empirically to ensure safety. To date, very little R&D effort has gone into approaches that provide quantitative safety guarantees for AI systems, because they’re considered impossible or impractical.

By combining scientific world models and mathematical proofs we will aim to construct a ‘gatekeeper’, an AI system tasked with understanding and reducing the risks of other AI agents. In doing so we’ll develop quantitative safety guarantees for AI in the way we have come to expect for nuclear power and passenger aviation.

Read the programme thesis

Read the accessible version of the programme thesis

Technical areas

This programme is split into three technical areas (TAs), each with its own distinct objectives.

TA1

Scaffolding

We can build an extendable, interoperable language and platform to maintain formal world models and specifications, and check proof certificates.

TA2

Machine Learning

We can use frontier AI to help domain experts build best-in-class mathematical models of real-world complex dynamics + train verifiable autonomous systems.

TA3

Real-World Applications

A safeguarded autonomous AI system with quantitative safety guarantees can unlock significant economic value when deployed in a critical cyber-physical operating context.

Meet the programme team

Our Programme Directors are supported by a core team that provides a blend of operational coordination and highly specialised technical expertise.

Nora Ammann

Programme Director

Nora has spent nearly a decade working on making transformative AI go well. Her work spans AI assurance – hardware guarantees, formal verification and the coordination needed for safety claims between untrusting parties. She previously founded Principles of Intelligence, bringing together AI researchers, neuroscientists, and physicists to study the foundations of intelligent behaviour.

Yasir Bakki

Programme Specialist

Yasir is an experienced programme manager whose background spans the aviation, tech, emergency services, and defence sectors. Before joining ARIA, he led transformation efforts at Babcock for the London Fire Brigade’s fleet and a global implementation programme at a tech start-up. He supports ARIA as an Operating Partner from Pace.

David 'davidad' Dalrymple

Technical Advisor

davidad is a software engineer with a multidisciplinary scientific background. He’s spent five years formulating a vision for how mathematical approaches could guarantee reliable and trustworthy AI. Before joining ARIA, davidad co-invented the top-40 cryptocurrency Filecoin and worked as a Senior Software Engineer at Twitter.

Previous programme updates

Late last year, we provided an update that we were redirecting our efforts away from Technical Area 2 (TA2) as originally planned and doubling down on expanding the ambition and scope of Technical Area 1 (TA1). Following this, we’ve been reviewing how the rest of the programme can best meet this new objective. After careful consideration, we have decided to shift the primary focus of TA3 towards applications in cybersecurity.

Why we are pivoting

When we launched TA3, our objective was to deploy AI solutions in real-world cyber-physical use cases while providing quantitative safety guarantees, to demonstrate a deployment API that would greatly reduce AI risks versus the general-purpose chat API. Since then, however, frontier AI capabilities have advanced at a pace that has exceeded our original projections, falsifying the hypothesis that high-risk dual-use AI might not be made openly available.
Given this acceleration, we believe we can deliver the greatest impact by:
- Broadening the TA1 toolkit – Expanding and strengthening the core technical safety tools currently under development.
- Prioritising cybersecurity – Concentrating our demonstration efforts where the need for robust, guaranteed AI tooling is both most acute and most time-sensitive.
This shift ensures that TA3 remains aligned with the evolving risk landscape and focused on the areas where ARIA can add the most value.

What this means for current projects
- TA3 Phase 2 Solicitation – We will not be proceeding with the planned Phase 2 call for cyber-physical applications.
- Current TA3 Phase 1 Projects – All existing TA3 Phase 1 projects will continue through to their scheduled completion.
- Potential Extensions – We are considering extensions for a small number of TA3 Phase 1 teams that have demonstrated exceptional progress. We will contact those teams directly in due course.
AI progress and a Safeguarded AI pivot

While our conviction in the vision for the Safeguarded AI programme remains unchanged, the pace of frontier AI progress has fundamentally altered our path – instead of investing in specialised AI systems that can use our tools, it will be more catalytic to broaden the scope and power of the TA1 toolkit itself, making it a foundational component for the next generation of AI.
Learn more

Nuclear power plant with cooling towers and transmission lines

Can AI be used to control safety critical systems?

Fortune

The U.K.'s Advanced Research and Invention Agency (ARIA) is funding a project to use frontier AI models to design and test new control algorithms for safety critical systems.