Safeguarded AI

Backed by £59m, this programme aims to develop the safety standards we need for transformational AI

Why this programme

As AI becomes more capable, it has the potential to power scientific breakthroughs, enhance global prosperity, and safeguard us from disasters. But only if it’s deployed wisely.

Current techniques working to mitigate the risk of advanced AI systems have serious limitations, and can’t be relied upon empirically to ensure safety. To date, very little R&D effort has gone into approaches that provide quantitative safety guarantees for AI systems, because they’re considered impossible or impractical.

What we’re shooting for

By combining scientific world models and mathematical proofs we will aim to construct a ‘gatekeeper’, an AI system tasked with understanding and reducing the risks of other AI agents.

In doing so we’ll develop quantitative safety guarantees for AI in the way we have come to expect for nuclear power and passenger aviation.

Our goal: to usher in a new era for AI safety, allowing us to unlock the full economic and social benefits of advanced AI systems while minimising risks.

This programme is split into three technical areas (TAs), each with its own distinct solicitations.

Apply for TA3

Deadline: xyz

The first solicitation for this programme focuses on TA1.1 Theory. We are looking for R&D Creators, individuals and teams that ARIA will fund and support, to research and construct computationally practicable mathematical representations and formal semantics to support world-models, specifications about state-trajectories, neural systems, proofs that neural outputs validate specifications, and “version control” (incremental updates or “patches”) thereof.

Applicants that are shortlisted following full proposal review, will be invited to meet with the Programme Director to discuss any critical questions/concerns prior to final selection.

Successful/unsuccessful applicants for TA1.1 will be notified on 10 July 2024.

Previous funding calls within Safeguarded AI
TA1.1 Theory

The first solicitation for this programme focused on TA1.1 Theory, where we sought R&D Creators – individuals and teams that ARIA will fund and support – to research and construct computationally practicable mathematical representations and formal semantics to support world-models, specifications about state-trajectories, neural systems, proofs that neural outputs validate specifications, and “version control” (incremental updates or “patches”) thereof.

Download the funding call for TA1.1 [PDF]

Watch the solicitation presentation

Meet davidad

Safeguarded AI has been designed and overseen by Programme Director David ‘davidad’ Dalrymple with feedback from the R&D community, as part of the opportunity space Mathematics for Safe AI.

davidad is a software engineer with a multidisciplinary scientific background. He’s spent five years formulating a vision for how mathematical approaches could guarantee reliable and trustworthy AI. Before joining ARIA, davidad co-invented the top-40 cryptocurrency Filecoin and worked as a Senior Software Engineer at Twitter.

background image

Coming soon: TA1.4

Stay tuned for our upcoming funding call for TA1.4 Sociotechnical integration.  

Applications open soon. For now, learn more about TA1.4 in the programme thesis. 

Previous funding calls within Safeguarded AI
TA1.1 Theory

The first solicitation for this programme focused on TA1.1 Theory, where we sought R&D Creators – individuals and teams that ARIA will fund and support – to research and construct computationally practicable mathematical representations and formal semantics to support world-models, specifications about state-trajectories, neural systems, proofs that neural outputs validate specifications, and “version control” (incremental updates or “patches”) thereof.

Download the funding call for TA1.1 [PDF]

Watch the solicitation presentation

TA3: Applications

The second funding call sought potential individuals or organisations interested in using our gatekeeper AI to build safeguarded products for domain-specific applications, such as optimising energy networks, clinical trials, or telecommunications networks. Safeguarded AI’s success will depend on showing that our gatekeeper AI actually works in a safety-critical domain. The research teams selected for TA3 will work with other programme teams, global AI experts, academics, and entrepreneurs, in setting the groundwork to deploy Safeguarded AI in one or more areas.

In this first phase of TA3 funding, we intend to allocate an initial £5.4M aimed at eliciting requirements, sourcing datasets, and establishing evaluation benchmarks for relevant cyber-physical domains.

Applications are now closed. Successful/unsuccessful applicants will be notified on 18 November 2024. 

Download the funding call for TA3 [PDF]

Watch the solicitation presentation