Building Resilient AI-powered Software, And Planning For When Things Go Wrong
AIThis post was created with the assistance of artificial intelligence (AI).

TL;DR

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

The Pragmatic Engineer published excerpts from Sam Newman’s 2026 book, Building Resilient Distributed Systems, covering disaster recovery and resilience for AI-powered software. The material outlines recovery targets and patterns, and discusses provider dependence, model changes, infrastructure constraints and observability; it does not report new outage data or a specific incident.

The Pragmatic Engineer has published excerpts from Sam Newman’s new book, Building Resilient Distributed Systems, setting out approaches to disaster recovery and reliability for software that uses AI. The report focuses on planning for major infrastructure failures and on risks such as changing models, provider dependence and limited computing capacity; it does not document a particular outage or offer new incident statistics.

Newman’s excerpts describe disaster recovery (DR) as the procedures used when a major failure affects a large part of a system. Fires, floods, power problems and failures in cloud infrastructure can take services offline despite safeguards against failures in individual software instances. The book frames DR as a set of trade-offs: reducing acceptable downtime or data loss typically raises cost and operational complexity.

Two targets help teams describe what they need. The recovery point objective (RPO) sets the maximum acceptable period of potential data loss, based on how recently data was backed up or synchronized. If backups run hourly, for example, as much as an hour of data could be lost. The recovery time objective (RTO) is the acceptable period of service unavailability after an incident, including the time to restore data, provide computing resources and redirect traffic.

The report also identifies several recovery patterns: backup and restore, pilot-light failover, warm failover and multisite active-active operation. These approaches differ in the resources they keep ready and the speed of recovery they can support. The excerpts say infrastructure planning must account for constraints such as ongoing GPU and CPU shortages, rather than assuming replacement capacity will always be available.

At a glance
reportWhen: Published last week, according to The P…
The developmentThe Pragmatic Engineer has published excerpts from Sam Newman’s new book on disaster recovery and building resilient distributed and AI-powered systems.

Recovery Plans Must Fit AI Systems

The report connects conventional disaster recovery with newer dependencies in AI-powered software. A service that relies on an external model provider may face availability or capacity problems beyond its direct control. A model can also change, making it harder to know whether application behavior remains consistent. Newman’s material points to observability—the ability to monitor system behavior—as part of managing those shifts.

For engineering teams, the practical issue is not simply whether a system has a backup. They need to decide what level of downtime and data loss the service can tolerate, test whether recovery arrangements meet those targets, and account for the availability of the infrastructure and AI services those arrangements depend on. The report describes options such as multiple models or vendors, transparent cost tracking and regular vendor reviews as ways to manage uncertainty, not as guarantees that outages will be prevented.

Amazon

disaster recovery software for distributed systems

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

The Book’s Disaster Recovery Framework

The Pragmatic Engineer says Newman spent three years writing the book, one year longer than he had originally planned, and revisited its chapters to address AI’s relevance to system design. The report presents material from two chapters, including Chapter 10 on disaster recovery, and says the book has 16 chapters.

Newman uses the term “site” as a general label for the location of computing infrastructure, whether that is a public-cloud region, a private data center or another on-premises facility. That terminology lets the recovery patterns apply across different hosting arrangements. The report says the book also considers issues involved in using more than one cloud vendor, though the supplied excerpt does not detail those issues.

The excerpts are from Building Resilient Distributed Systems, published by O’Reilly Media in 2026 and used with permission, according to The Pragmatic Engineer. The newsletter’s author says the book recommendation was unpaid and that links were not affiliate links.

“The recovery point objective (RPO) defines the maximum amount of time that can have passed since the data was last backed up or synchronized.”

— Sam Newman, in an excerpt from Building Resilient Distributed Systems

Amazon

AI system observability tools

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

The Excerpts Leave Key Details Open

The published material is an excerpt, not a full account of the book or an assessment of a named company’s recovery plan. It does not provide specific RPO or RTO recommendations for particular workloads, nor does it quantify how often AI providers or infrastructure become unavailable. The report’s reference to more outages and more production bugs is presented as its framing; the supplied material includes no data establishing the scale or cause of that trend.

It is also unclear from the excerpt how teams should choose among the recovery patterns in particular cases, how often they should test failover, or what costs different arrangements would involve. Newman’s discussion of model and vendor uncertainty is identified in the report, but the provided text does not set out detailed implementation guidance or show that using multiple providers will avert service interruptions.

Amazon

cloud infrastructure backup solutions

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Teams Can Set and Test Targets

The immediate next step for organizations applying the material is to define acceptable data loss and downtime, then select a recovery approach that matches those targets and their available resources. That work includes checking whether backup, computing capacity and traffic-routing plans can function under a site-wide failure, rather than assuming infrastructure will be readily available during a crisis.

For AI-powered services, teams can also monitor model behavior and provider availability, track costs and review vendor dependencies as conditions change. The report does not announce a follow-up study, deadline or planned industry action. Further specifics on these recommendations would require consulting the full book or additional reporting.

Amazon

failover and redundancy hardware

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Key Questions

What is the news?

The Pragmatic Engineer published excerpts from Sam Newman’s 2026 book, Building Resilient Distributed Systems, on disaster recovery and resilience for AI-powered software.

What do RPO and RTO mean?

RPO is the maximum acceptable period of potential data loss. RTO is the maximum acceptable time a service remains unavailable after an incident.

Which disaster recovery patterns are mentioned?

The report lists backup and restore, pilot-light failover, warm failover and multisite active-active operation. The excerpt does not prescribe one approach for every system.

It points to dependence on model providers, models changing over time, infrastructure capacity limits and the need for observability. It also mentions multiple models or vendors, cost transparency and recurring vendor reviews as ways to manage uncertainty.

Does the report prove that AI is causing more outages?

No. The report frames the book as timely amid concerns about outages, increased code deployment and production bugs, but the supplied material provides no statistics or evidence establishing the scale or causes of an increase.

Source: rss

COLUMBUS DAY / I

Columbus Day / Indigenous Peoples' Day Picks

As an affiliate, we earn on qualifying purchases.

You May Also Like

The 27% Problem: Why Google Wrote a $750M Check to Catch Anthropic

Google commits $750 million to expand enterprise AI, aiming to surpass Anthropic’s 40% market share amid shifting industry dynamics.

A Guide To Reproducing AI Benchmark Results With UK AISI And EvalEval

AISI has published selected benchmark and cyber evaluation results with verification, context and configuration details through EvalEval’s Evaluation Cards.

Curating AI Art Exhibitions: Challenges and Opportunities

Keeping pace with AI art curation reveals complex challenges and exciting opportunities that could redefine how we experience creativity—discover what lies ahead.

AI Video’s New Power Trio: Seedance 2.5, Minimax H3 & Wan 3 – Technology Org

A ByteDance Seed-linked report groups Seedance 2.5, Minimax H3, and Wan 3 as a new competitive set in AI video generation, with details still emerging.