Posts

Brainrot

Image
I drive my daughter to school as part of a car pool. Along the way, I am learning a new language, Brainrot. So what is brainrot ? It is what you get when you marinate your brain with silly TikTok, YouTube Shorts, and Reddit memes. It is slang for "my attention span is fried and I like it". Brainrot is a self-deprecating language. Teens are basically saying: I know this is dumb, but I am choosing to speak it anyway. What makes brainrot different from old-school slang is its speed and scale. When we were teenagers, slang spread by word of mouth. It mostly stayed local in our school hallways or neighborhood. Now memes go global in hours. A meme is born in Seoul at breakfast and widespread in Ohio by six seven pm. The language mutates at escape velocity and gets weird fast.  Someone even built a brainrot programming language . The joke runs deep , and is getting some infrastructure. Here are a few basic brainrot terms you will hear right away. He is cooked : It means he is finis...

Best of metadata in 2025

Image
It is that time of year again to look back on a year of posts. I average about sixty posts annually. I don't explicitly plan for the number, and I sometimes skip weeks for travel or work, yet I somehow hit the number by December. Looking back, I always feel a bit proud. The posts make past Murat look sharp and sensible, and I will not argue with that. Here are some of the more interesting pieces from the roughly sixty posts of 2025. Advice Looks like I wrote several advice posts this year. I must be getting old. The Invisible Curriculum of Research Academic chat: On PhD What I'd do as a College Freshman in 2025 My Time at MIT What makes entrepreneurs entrepreneurial? Publish and Perish: Why Ponder Stibbons Left the Ivory Tower Databases Concurrency Control book reading was fun. Also the series on use of time in distributed databases. And it seems like I got hyperfocused on transaction isolation this year.  Concurrency Control and Recovery in Database Systems Book reading series...

Optimize for momentum

Progress comes from motion.  Momentum is the invisible engine of any significant work. A project feels daunting when you face it as a blank page. It feels easier when you built some momentum with some next steps. So, momentum makes the difference between blocked and flowing. Think of a stalled truck on a desert road. You can't lift it with superhuman strength. But by rocking it with small periodic forces at the right rhythm (matching its natural frequency) you can get it rolling. Each tiny push adds to the previous one because the timing aligns with the system's response. The truck starts to move, and then the engine catches. Projects behave the same way. A big project has its own rhythm. If you revisit it daily, even briefly, your pushes line up. Your brain stays warm. Context stays loaded . Ideas from yesterday are still alive today. Each session amplifies the last because you are operating in phase with your own momentum. When you produce something every day, you never feel...

Mitigating Application Resource Overload with Targeted Task Cancellation

Image
The Atropos paper (SOSP'25) argues that overload-control systems are built on a flawed assumption. They monitor global signals (like queue length or tail latency) to adjust admission control (throttling new arrivals or dropping random requests). This works when the bottleneck is CPU or network, but it fails when the real problem is inside the application. This considers only the symptoms but not the source. As a result, it drops the victims rather than the culprits. Real systems often run into overload because one or two unlucky timed requests monopolize an internal logical resource (like buffer pools, locks, and thread-pool queues). These few rogue whales have nonlinear effects. A single ill-timed dump query can thrash the buffer pool and cut throughput in half. A single backup thread combined with a heavy table scan can stall writes in MySQL as seen in Figure 3. The CPU metrics will not show this. Atropos proposes a simple fix to this problem. Rather than throttling or dropping ...

Disaggregated Database Management Systems

Image
This paper is based on a panel discussion from the TPC Technology Conference 2022. It surveys how cloud hardware and software trends are reshaping database system architecture around the idea of disaggregation. For me, the core action is in Section 4: Disaggregated Database Management Systems. Here the paper discusses three case studies (Google AlloyDB, Rockset, and Nova-LSM) to give a taste of the software side of the movement. Of course there are many more. You can find Aurora , Socrates , and Taurus , and TaurusMM reviews in my blog. In addition, Amazon DSQL (which I worked on) is worth discussing soon. I’ll also revisit the PolarDB series of papers , which trace a fascinating arc from active log-replay storage toward simpler, compute-driven designs. Alibaba has been prolific in this space, but the direction they are ultimately advocating remains muddled across publications, which reflect conflicting goals/priorities. AlloyDB AlloyDB extends PostgreSQL with compute–storage disagg...

Taurus MM: A Cloud-Native Shared-Storage Multi-Master Database

Image
This VLDB'23 paper presents Taurus MM, Huawei's cloud-native, multi-master OLTP database built to scale write throughput in clusters between 2 to 16 masters. It extends the single-master TaurusDB design (which we reviewed yesterday) into a multi-master design while following its shared-storage architecture with separate compute and storage layers. Each master maintains its own write-ahead log (WAL) and executes transactions independently; there are no distributed transactions. All masters share the same Log Stores and Page Stores, and data is coordinated through new algorithms that reduce network traffic and preserve strong consistency. The system uses pessimistic concurrency control to avoid frequent aborts on contended workloads. Consistency is maintained through two complementary mechanisms: a new clock design that makes causal ordering efficient, and a new hybrid locking protocol that cuts coordination cost. Vector-Scalar (VS) Clocks A core contribution is the Vector-Scal...

Taurus Database: How to be Fast, Available, and Frugal in the Cloud

Image
This SIGMOD’20 paper presents TaurusDB, Huawei's disaggregated MySQL-based cloud database. TaurusDB refines the disaggregated architecture pioneered by Aurora and Socrates, and provides a simpler and cleaner separation of compute and storage.  In my writeup on Aurora , I discussed how "log is the database" approach reduces network load, since the compute primary only sends logs and the storage nodes apply them to reconstruct pages. But Aurora did conflate durability and availability somewhat and used quorum-based replication of six replicas for both logs and pages. In my review of Socrates , I explained how Socrates (Azure SQL Cloud) separates durability and availability by splitting the system into four layers: compute, log, page, and storage. Durability (logs) ensures data is not lost after a crash. Availability (pages/storage) ensures data can still be served while some replicas or nodes fail. Socrates stores pages separately from logs to improve performance but the e...

TLA+ Modeling of AWS outage DNS race condition

Image
On Oct 19–20, 2025, AWS’s N. Virginia region suffered a major DynamoDB outage triggered by a DNS automation defect that broke endpoint resolution. The issue cascaded into a region-wide failure lasting nearly a full day and disrupted many companies’ services. As with most large-scale outages, the “DNS automation defect” was only the trigger; deeper systemic fragilities ( see my post on the Metastable Failures in the Wild paper ) amplified the impact. This post focuses narrowly on the race condition at the core of the bug, which is best understood through TLA+ modeling. My TLA+ model builds on Waqas Younas’s Promela/Spin version . To get started quickly, I asked ChatGPT to translate his Promela model into TLA+, which turned out to be a helpful way to understand the system’s behavior, much more effective than reading the postmortem or prose descriptions of the race. The translation wasn’t perfect, but fixing it wasn’t hard. The translated model treated the enactor’s logic as a single atom...

Barbarians at the Gate: How AI is Upending Systems Research

Image
This recent paper from the Berkeley Sky Computing Lab has been making waves in systems community. Of course, Aleksey and I did our live blind read of it, which you can watch below. My annotated copy of the paper is also available here. This is a fascinating and timely paper. It raises deep questions about how LLMs will shape the research process, and how that could look like. Below, I start with a short technical review, then move to the broader discussion topics. Technical review The paper introduces AI-Driven Research for Systems (ADRS) framework. By leveraging the OpenEvolve framework ,  ADRS integrates LLMs directly into the systems research workflow to automate much of the solution-tweaking and evaluation process. As shown in Figure 3, ADRS operates as a closed feedback loop in which the LLM ensemble iteratively proposes, tests, and refines solutions to a given systems problem. This automation targets the two most labor-intensive stages of the research cycle, solution tweaking...

Academic chat: On PhD

Image
This week, Aleksey and I met not to dissect a research paper, but to chat about "the process of PhD". I had recently wrote a post titled "The Invisible Curriculum of Research" , where I framed research as an iceberg, with the small visible parts (papers, conferences) resting on the hidden 5 Cs: Curiosity/Taste: what problems are worth solving. Clarity: how to ask precise and abstracting questions. Craft: writing, experimentation, presentation. Community: collaboration and contribution. Courage: resilience through setbacks. Above is the video of our chat, with a lot of personal anecdotes and a few rants. But if you want to cut to the chase, the highlight reel is below. What a PhD Really Produces The real product of a PhD is not the thesis, but you, the researcher! The thesis is just the residue of this long internal transformation. Like martial arts, the training breaks you and rebuilds you into someone who sees and thinks differently. This transformation cannot be ...

Popular posts from this blog

Hints for Distributed Systems Design

My Time at MIT

Scalable OLTP in the Cloud: What’s the BIG DEAL?

Foundational distributed systems papers

Advice to the young

Learning about distributed systems: where to start?

Distributed Transactions at Scale in Amazon DynamoDB

Optimize for momentum

Making database systems usable

Use of Time in Distributed Databases (part 1)