Metadata

General methods that scale with computation will inevitably displace hand-engineered domain knowledge! Sutton’s Bitter Lesson hangs like a Sword of Damocles over all of us. This paper, which just dropped, applies that lesson to data agents, and says that as LLMs improve, they will quickly absorb the agent scaffolding researchers have spent the last few years painstakingly building. It argues that researchers should instead work on building curated contextual information about the data environment (aka. persistent semantic context) to help data agents be efficient across many queries. The Bitter Lesson for Data Agents The evaluation section aims to capture the Bitter Lesson in action. The authors compared general coding agents (the Codex harness with no task-specific engineering) against state-of-the-art human-designed data agents on two benchmarks, TAG-Bench and DAB. They use the same models on both sides, so the only variable is the scaffolding. They find that: Although the human-d...

I am a distributed systems researcher. I mostly read and write about distributed systems and lightweight formal methods. But as my blog definition says, the blog is about "distributed systems broadly defined and other curiosities".  The last two years, LLMs were unavoidably the biggest part of those curiosities. However, I was still surprised how much I had written about them, when I went looking for a line I remembered coining. Something like, "LLMs are good at mediocrity, but really fast". I couldn't find the line, but instead I found a pile of my takes on LLMs scattered across the blog, and I figured it was worth collecting them into an index. So here it is, enjoy! I also threw in my overall take on LLMs that has stayed constant through their four years of reign so far.   Hot takes on LLMs Our Collective Bike Shed Moment (June 26) Are We Becoming Architects or Butlers to LLMs? (Feb 26) How LLMs may affect academic writing (Feb 26)  Agentic AI and The Mythi...

Specula is an agentic system that automates the process of software bug finding through authoring and model-checking a spec for the code. It derives TLA+ specifications automatically from the code, checks code-spec conformance through trace validation, model checks the spec to find concurrency bugs, and reproduces the bug at the code layer by writing integration tests with precise timing. I remember reading the Daikon paper "Quickly detecting relevant program invariants" in 2000 and getting impressed by it, and here we are after 26 years, solving the end-to-end problem much better than I ever thought would be possible in a push-button manner in the year of our lord 2026. But somehow, I am still somewhat unsatisfied with the paper. This may be me being hypercritical and trying to get more out of the paper by arguing with it . So bear with me until I resolve (or learn to accept) these problems over time. I know many of the authors of the Specula work, and respect them, and I...

Shortly after I joined MongoDB research, we ran a TLA+ workshop. It was a two-day ordeal. We had a 1.5 days of instruction on TLA+ and syntax, after which we tried to help people get started with modeling. People liked learning about TLA+ on the first day, but except for a person or two, we didn't get anyone onboarded with TLA+ modeling. It was too much to offload on people and ask them to level up in a short time frame. Well, two years after that first workshop, on May 11th, 2026, we ran a second edition of this workshop with one very big difference. What is that big difference, you ask? AI! AI makes formal methods not only necessary, but also more feasible and easier! Jesse, Will, and I planned this workshop to be aggressively short. We provide under two hours of instruction, then everyone starts modeling hands-on. We act as TAs and help people as they go. The AI takes care of the syntax problem for TLA+, and also helps with modeling. We just need to teach people enough to read a...

The transformative power of LLMs in coding has been irrefutable, and it feels like we are living through a magical computing renaissance. On the socials, we hear impressive numbers of lines of code generated, features delivered, and bugs fixed. But, the macroeconomic indicators seem to be still lagging. Heck, if you talk with an engineering manager, you find that their product shipping dates haven't miraculously compressed by a factor of five, either. This paper just landed 10 days ago. It is from MIT and Wharton by Mert Demirer, Leon Musolff, and Liyuan Yang. Their study attempts to provide a structured economic model for evaluating actual productivity obtained from AI coding tools. By pairing confidential Microsoft telemetry with the public footprints of over 100,000 GitHub developers (tracking everything from open-source utilities to web app repositories), the authors show significant systemic friction downstream of AI code generation. Of course, I do my usual skeptical critic ...

As I mentioned in my previous post , I traveled to San Jose at the end of May for the ACM CAIS conference. On Day 0, I gave a very short talk at the Supporting our AI Overlords (SAO) workshop. This post is the promised summary of our paper, "A Case for Simulation-Driven Resilience in Agentic Data Systems" , joint work with Aleksey Charapko (University of New Hampshire) and Akshat Vig (MongoDB). Metastability is critical for building the next generation of distributed systems Our story starts with metastability. Metastability is the failure mode where the mechanisms built to protect the system (retries, queues, timeouts, load shedding) turn into amplifiers. Even after the trigger that caused the overload goes away, the system stays behind, churning through busy work, perpetually trying to catch up with the remnants of failed and behind-schedule tasks. It's a bit like missing some foundational math in high school. You spend so long backfilling the old gaps that you never ke...

Last week, I traveled to San Jose to attend the ACM CAIS conference . On Day 0, I gave a short talk at the Supporting our AI Overlords (SAO) workshop . And yes, I promise to write a summary of our paper, " A Case for Simulation-Driven Resilience in Agent-First Data Systems " soon!  To start with an overall impression of the conference: much of the work presented felt exploratory and anecdotal. Since the compound AI space is still so new, many work seemed to share on-the-ground best practices that worked for them rather than principled results. Some talks really leaned into the "agent, act like a senior engineer and don't make mistakes" vibe. This was especially apparent in the "Agent Skills Workshop". I am not saying this is a bad thing, I learned some valuable lessons from that workshop, which I'll share below. CAIS defines the conference's scope broadly as "research on compound AI architectures, optimization, and deployment". Unfort...

Ok, finally getting sometime to put my butt down to write about day 2 of BugBash. Why do so few buildings fall down? Brian Potter, Senior Infrastructure Fellow @ Institute for Progress, Author of Construction Physics  newsletter. Buildings rarely collapse. The rate of major structural failing is  between 1/100K to 1/ 1 million. (This is how I know this is a serious statistic: it is an interval.)  Why don't more buildings fall down? There are some technical reasons to it: buildings are simple stuctures with no (or little) moving parts. Buildings exhibit a limited number of behavior when you load their structure: stress, deflection, vibration, creep, etc. And these behaviors are commensurate to the  proportion of the force you put in. Finally, buildings are designed for 2X-3X of expected load. Let's go deeper into structural elements. We have good theories for how structural elements behave, and individual components are tested extensively and are standardized. A build...