Aurora DSQL: Scalable, Multi-Region OLTP

-

The Aurora DSQL paper finally dropped. Reading it yesterday was an interesting experience, because I spent two years (2022-23) working with the AWS team that designed and built Aurora DSQL. Since I have been very familiar with the architecture, the paper's overview description of the system didn't excite me much. And if I am being honest, reading the paper also felt a bit dry, most likely because I am not doing the usual extra thinking to explore/understand the ideas in the paper. But taking a step back today, and leaving my subjective experience aside, I will try to elaborate on how the DSQL architecture is actually built on a set of aggressive and opinionated engineering bets. 

If I had to explain this architecture in a single sentence (just as I used to do for other teams at AWS) it would be this: We took a traditional monolithic database and blew out every single component into an independent, horizontally scalable service.


Exploding the Monolith

DSQL divides the database into the following specialized services:

  • Query Processors (QP): Stateless virtual machines that run a custom PostgreSQL engine to parse queries and buffer writes locally.
  • Storage Nodes: Sharded nodes that hold the data and use multiversion concurrency control (MVCC) to serve historical data to QP instantly.
  • Adjudicators: The conflict-resolution layer that checks if a transaction handed of to them by QP is safe to commit.
  • Journals: A highly available replication log that durably saves transactions across zones or regions. (Good for short term durability before this reaches the storage nodes. See my MemoryDB review.) 
  • Crossbars: The routing layer that reads the updates from the Journals and sends changes to the right Storage Nodes.


The Big Architectural Bets in DSQL Design

Well, in addition to fully committing to the disaggregation principle, here are the other big bets in DSQL design.

The Synchronized Clock Bet: To get reads without coordination, DSQL relies entirely on highly synchronized physical clocks, in this case AWS TimeSync. A QP just checks its local clock and asks storage for data from that exact microsecond.

No Pessimistic Locking: Optimistic Concurrency Control (OCC) may lead to high abort rates for databases, but DSQL makes it work by pairing it with MVCC under Snapshot Isolation. Because readers look at a snapshot of the past, read-write conflicts are impossible. Awesome, but what about write-skew? When needed, customers should just use FOR UPDATE, and also design their schemas to force write-write conflicts for business logic violations.

Eventual Consistency is Dead: DSQL provides strong consistency (linearizability), arguing that developers simply cannot write correct business logic on eventually consistent systems. It’s a subtle point, but linearizability (a guarantee about single-object real-time operations) and snapshot isolation (a guarantee about multi-object transaction visibility) control different things, as Jepsen's consistency models outline. DSQL offers a consistent snapshot for your snapshot-isolated transactions, and that individual key operations strictly respect real-time ordering.

Forcing Guardrails: DSQL hard-caps transactions at 3,000 rows and 10MiB. The paper cites Little's Law to justify this, essentially forcing users to accept smaller transactions in exchange for highly predictable stable tail latency.

Linearized 2PC: For transactions that span multiple Adjudicators, traditional Two-Phase Commit (2PC) is too slow over wide area networks as it requires 2RTTs. DSQL uses a "Warp-inspired" trick where Adjudicators vote, but only the leader writes the final commit to its single Journal. This avoids coordinating multiple logs.


The Payoff

Independent Scalability: Compute, commit logic, and storage are completely separated. If you need more read capacity, you add storage nodes. If you have a spike in connections, the system spins up more QPs. You can also tune/optimize them separately, for example, potentially reconfigure adjudicator-range placement based on access patterns.

0-RTT Consistent Reads: Because the QP assigns a local timestamp and storage handles the rest, reading data requires zero coordination with a leader. It is almost as fast as your network latency to the storage node. This is a big win because in OLTP SQL, reads are significantly more common than writes. Even most writes (like UPDATES or INSERTS with unique indexes) are actually reads first.

1-RTT (or 1.5 RTT) Commits: Whether you write one row or a hundred, coordination only happens once at commit time, costing just 1 RTT (or slightly more for a multi-shard commit).

No "Slow Lock Holder" Problem: Because there are no pessimistic locks, a developer going to lunch with an open transaction terminal (exact quote from the paper) cannot bring down the database. Readers never block writers, and writers never block readers.


While it is hard to critique my old team, this won't be a true Murat Buffalo (now Bay Area?) review without pointing out the tradeoffs and shortcomings in the paper. Because of the long read-modify-commit duration during a transaction, DSQL may be prone to write-write conflicts on hot keys. This limits how many back-to-back operations you can squeeze into a single row (which is bad application design anyway). Under heavy contention, transactions will abort, whereas a traditional database may have queued them up. If two regions write to the same key, OCC will abort one of them, and unfortunately this conflict is only detected at commit time, meaning you pay the WAN network latency penalty before finding out you have to retry.

While the paper couldn't provide extensive evaluation, quantitative data on user adoption or business payoff, it does provide a candid Lessons Learned section which talks about some friction with respect to Foreign Key Constraints and high-locality sequences.


Building at Scale

I have one final takeaway that is somewhat counterintuitive: building a completely novel global production database didn't actually feel like as much effort as it should have.

Yes, it was a lot of work, but the execution felt surprisingly smooth after the team got going. I attribute this to two things: a great upfront design, and a deliberate choice to avoid reinventing the wheel. Rather than building everything from scratch, the team used the PostgreSQL engine for SQL parsing, execution, and the client protocol, while discarding its local storage and transaction processing layers. We didn't build a new replication log; we used AWS's existing internal Journal service. We were also equipped with hard-earned lessons from preceding AWS database projects, like JournalDB and the unfortunately named QLDB (Quantum Ledger Database).  Beyond the tooling, the team dynamics were excellent. Marc Brooker is technically brilliant, and he did a masterful job leading a talented team of principal engineers. Our weekly whiteboard thinking sessions were a lot of fun. Because the foundational architecture was so well-designed, the actual development felt significantly easier than at least what I would expect for a product of this magnitude. (But then again, I wasn't with the team for the last year of the project!)

Comments

Post a Comment

Popular posts from this blog

Hints for Distributed Systems Design

-
This is with apologies to Butler Lampson, who published the " Hints for computer system design " paper 40 years ago in SOSP'83. I don't claim to match that work of course. I just thought I could draft this post to organize my thinking about designing distributed systems and get feedback from others. I start with the same  disclaimer Lampson gave. These hints are not novel, not foolproof recipes, not laws of design, not precisely formulated, and not always appropriate. They are just hints.  They are context dependent, and some of them may be controversial. That being said, I have seen these hints successfully applied in distributed systems design throughout my 25 years in the field, starting from the theory of distributed systems (98-01), immersing into the practice of wireless sensor networks (01-11), and working on cloud computing systems both in the academia and industry ever since. These heuristic principles have been applied knowingly or unknowingly and has proven...
Read more >>

Chess invariants

-
Image
Chess is a lot trickier than it looks. It has so many rules: castling, en passant, pawn promotion, pinning, the discovered check, and the deadlock case of stalemate. It is a concurrent system, but with a very specific kind of concurrency: interleaved execution. More specifically, taking turns: white, then black, then white. You know what we do with concurrent systems here? Here we model them, and we distill their invariants. Here is some setup definitions first. In a CS or math paper, if you write "Section 2: Model and Problem" well enough, the rest of the paper writes itself. With this setup you can sort of see what the actions will be. In fact, forget about the actions. Let's look at some invariants. Invariants When deriving invariants we ask: what must always be true? I find it useful to split the safety invariants into two camps: state invariants (which are predicates over a single state) and transition invariants (which are predicates over a step). The transition inv...
Read more >>

The Agentic Self: Parallels Between AI and Self-Improvement

-
2025 was the year of the agent. The goalposts for AGI shifted; we stopped asking AI to merely "talk" and demanded that it "act". As an outsider looking at the architecture of these new agents and agentic system, I noticed something strange. The engineering tricks used to make AI smarter felt oddly familiar. They read less like computer science and more like … self-help advice . The secret to agentic intelligence seems to lie in three very human habits: writing things down, talking to yourself, and pretending to be someone else. They are almost too simple. The Unreasonable Effectiveness of Writing One of the most profound pieces of advice I ever read as a PhD student came from Prof. Manuel Blum, a Turing Award winner. In his essay "Advice to a Beginning Graduate Student", he wrote: "Without writing, you are reduced to a finite automaton. With writing you have the extraordinary power of a Turing machine." If you try to hold a complex argument enti...
Read more >>

Learning about distributed systems: where to start?

-
This is definitely not a "learn distributed systems in 21 days" post. I recommend a principled, from the foundations-up, studying of distributed systems, which will take a good three months in the first pass, and many more months to build competence after that. If you are practical and coding oriented you may not like my advice much. You may object saying, "Shouldn't I learn distributed systems with coding and hands on? Why can I not get started by deploying a Hadoop cluster, or studying the Raft code." I think that is the wrong way to go about learning distributed systems, because seeing similar code and programming language constructs will make you think this is familiar territory, and will give you a false sense of security. But, nothing can be further from the truth. Distributed systems need radically different software than centralized systems do.  --A. Tannenbaum This quotation is literally the first sentence in my distributed systems syllabus. Inst...
Read more >>

5 Lessons at 50

-
Image
Looking at my peak male physique, and my Keanu Reeves baby face, you would never suspect it, but I recently turned 50. As is the tradition, I thought about writing a post titled "50 Lessons at 50". Unfortunately, I don't have that kind of wisdom. The thing is, I still feel like I'm 18, same age as my son. Turns out this is the secret old guys have been hiding from us all along. You get older only on the outside, but inside you still see yourself as the same young lad. Still, fifty years should count for something. So what did I actually learn? How am I mentally different than my 18-year-old self. Here is my attempt to tally it up.  1. Caution is warranted I finally understand my parents. As you age, you accumulate battle scars, and the scars turn into habits. Anything that can go wrong will go wrong. You forget the cooktop on once, and suddenly you check it three times before leaving the house. True story. You stop diving head-first into a pile of leaves, because ther...
Read more >>

Foundational distributed systems papers

-
I talked about the importance of reading foundational papers last week. To followup, here is my compilation of foundational papers in the distributed systems area. (I focused on the core distributed systems area, and did not cover networking, security, distributed ledgers, verification work etc. I even left out distributed transactions, I hope to cover them at a later date.)  I classified the papers by subject, and listed them in chronological order. I also listed expository papers and blog posts at the end of each section. Time and State in Distributed Systems Time, Clocks, and the Ordering of Events in a Distributed System. Leslie Lamport, Commn. of the ACM,  1978. Distributed Snapshots: Determining Global States of a Distributed System. K. Mani Chandy Leslie Lamport, ACM Transactions on Computer Systems, 1985. Virtual Time and Global States of Distributed Systems.  Mattern, F. 1988. Practical uses of synchronized clocks in distributed systems. B. Liskov, 1991. Exp...
Read more >>

Building a Database on S3

-
Image
Hold your horses, though. I'm not unveiling a new S3-native database. This paper is from 2008. Many of its protocols feel clunky today. Yet it nails the core idea that defines modern cloud-native databases: separate storage from compute. The authors propose a shared-disk design over Amazon S3, with stateless clients executing transactions. The paper provides a blueprint for serverless before the term existed. SQS as WAL and S3 as Pagestore The 2008 S3 was painfully slow, and 100 ms reads weren't unusual. To hide that latency, the database separates "commit" from "apply". Clients write small, idempotent redo logs to Amazon Simple Queue Service (SQS) instead of touching S3 directly. An asynchronous checkpoint by a client applies those logs to B-tree pages on S3 later. This design shows strong parallels to modern disaggregated architectures . SQS becomes the write-ahead log (WAL) and logstore. S3 becomes the pagestore. Modern Aurora follows a similar logic : t...
Read more >>

Cloudspecs: Cloud Hardware Evolution Through the Looking Glass

-
Image
This paper (CIDR'26) presents a comprehensive analysis of cloud hardware trends from 2015 to 2025, focusing on AWS and comparing it with other clouds and on-premise hardware. TL;DR: While network bandwidth per dollar improved by one order of magnitude (10x), CPU and DRAM gains (again in performance per dollar terms) have been much more modest. Most surprisingly, NVMe storage performance in the cloud has stagnated since 2016. Check out the NVMe SSD discussion below for data on this anomaly. CPU Trends Multi-core parallelism has skyrocketed in the cloud. Maximum core counts have increased by an order of magnitude over the last decade. The largest AWS instance u7in now boasts 448 cores. However, simply adding cores hasn't translated linearly into value. To measure real evolution, the authors normalized benchmarks (SPECint, TPC-H, TPC-C) by instance cost. SPECint benchmarking shows that cost-performance improved roughly 3x over ten years. A huge chunk of that gain comes from AWS G...
Read more >>

TLA+ modeling tips

-
Model minimalistically Start from a tiny core, and always keep a working model as you extend. Your default should be omission. Add a component only when you can explain why leaving it out would not work. Most models are about a slice of behavior, not the whole system in full glory: E.g., Leader election, repair, reconfiguration. Cut entire layers and components if they do not affect that slice. Abstraction is the art of knowing what to cut . Deleting should spark joy.  Model specification, not implementation Write declaratively. State what must hold, not how it is achieved. If your spec mirrors control flow, loops, or helper functions, you are simulating code. Cut it out. Every variable must earn its keep. Extra variables multiply the state space (model checking time) and hide bugs. Ask yourself repeatedly: can I derive this instead of storing it? For example, you do not need to maintain a WholeSet variable if you can define it as a state function of existing variables: WholeSet =...
Read more >>

Supporting our AI overlords: Redesigning data systems to be Agent-first

-
Image
This Berkeley systems group paper opens with the thesis that LLM agents will soon dominate data system workloads. These agents, acting on behalf of users, do not query like human analysts or even like the applications written by them. Instead, the LLM agents bombard databases with a storm of exploratory requests: schema inspections, partial aggregates, speculative joins, rollback-heavy what-if updates. The authors calls this behavior agentic speculation . Agentic speculation is positioned as both the problem and the opportunity. The problem is that traditional DBMSs are built for exact intermittent workloads and cannot handle the high-throughput redundant and inefficient querying of LLM agents. The opportunity also lies here. Agentic speculation has recognizable properties and features that invite new designs. Databases should adapt by offering approximate answers, sharing computation across repeated subplans, caching grounding information in an agentic memory store, and even steering...
Read more >>