Thoughts on LLMs

-

I am a distributed systems researcher. I mostly read and write about distributed systems and lightweight formal methods. But as my blog definition says, the blog is about "distributed systems broadly defined and other curiosities". 

The last two years, LLMs were unavoidably the biggest part of those curiosities. However, I was still surprised how much I had written about them, when I went looking for a line I remembered coining. Something like, "LLMs are good at mediocrity, but really fast".

I couldn't find the line, but instead I found a pile of my takes on LLMs scattered across the blog, and I figured it was worth collecting them into an index. So here it is, enjoy! I also threw in my overall take on LLMs that has stayed constant through their four years of reign so far.

 

Hot takes on LLMs

Our Collective Bike Shed Moment (June 26)

Are We Becoming Architects or Butlers to LLMs? (Feb 26)

How LLMs may affect academic writing (Feb 26) 

Agentic AI and The Mythical Agent-Month (Jan 26)

The Agentic Self: Parallels Between AI and Self-Improvement (Jan 26) 

Too Close to Our Own Image? (Jan 26)

Rethinking the University in the Age of AI (Jan 26)

Welcome to Town Al-Gasr (Jan 26)

The Invisible Curriculum of Research (Oct 25) — mostly about AI, trust me

Academic chat: On PhD (Oct 25) — follow-up to the above

What I'd do as a College Freshman in 2025 (Apr 25)


Hot damn, 11 posts deep... Apparently I am an AI thought leader now by accident.

Looking at the dates, I was clearly most obsessed with LLMs in January 2026. There is no surprise there, since agentic tools and models turned a real corner that Thanksgiving of 2025. 

What has stayed constant over the years, however, even as the models themselves got much better, is my verdict on them. The phrase I had been hunting for turned out to be: "LLMs excel at high-throughput mediocrity". 

LLM output looks excellent when you are not the expert in the room. But on a topic you actually know, you would evaluate the output only as a notch above mediocre. This is the Gell-Mann amnesia effect in action. However, the difference is that LLMs can produce this output fast, and they never get tired.  

That is actually very handy when you don't want to work on a part of a project that requires mediocre effort. And the best thing is, you don't have to get bogged down doing it, and keep your momentum going as the LLMs handle those parts. They are a gift from the Gods if you have ADHD. The mundane half of a project that used to stall me out completely is no longer an obstacle.

Which is all to say,  LLMs are a tool. Use the right tool for the right job and you'll love it. Master your tools, and don't focus on their shortcomings for everything and all things.

Even with all these tools running, my highest-ROI tool is still just Emacs (going strong against LLMs at 50 years old, like me), where the actual thinking, writing, and planning happens. Use LLMs for the uninteresting stuff, where mediocrity is sufficient, so you have your time and energy for the work that actually matters.


AI x Systems Research

Well, LLMs didn't just show up as a curiosity in my work, they also collided more directly with my research. Here as they show up in my paper reviews, workshop write-ups, and conference notes, where formal methods and AI actually crossed paths.


Specula: Scaling formal specifications for autonomous model checking of system code (August 26)

Our MongoDB TLA+ Workshop (June 26)

Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools (June 26)

ACM CAIS: Conference on AI and Agentic Systems (June 26)

BugBash'26 Keynote: We won, what now? (April 26) 

Measuring Agents in Production (March 26)

Measuring AI Ability to Complete Long Software Tasks (March 26) 

SysMoBench: Evaluating AI on Formally Modeling Complex Real-World Systems (March 26) 

Beat Paxos (March 26)

Barbarians at the Gate: How AI is Upending Systems Research (Oct 25)

Supporting our AI overlords: Redesigning data systems to be Agent-first (Sep 25)

Neurosymbolic AI: Why, What, and How (Aug 25)

Comments

Post a Comment

Popular posts from this blog

Hints for Distributed Systems Design

-
This is with apologies to Butler Lampson, who published the " Hints for computer system design " paper 40 years ago in SOSP'83. I don't claim to match that work of course. I just thought I could draft this post to organize my thinking about designing distributed systems and get feedback from others. I start with the same  disclaimer Lampson gave. These hints are not novel, not foolproof recipes, not laws of design, not precisely formulated, and not always appropriate. They are just hints.  They are context dependent, and some of them may be controversial. That being said, I have seen these hints successfully applied in distributed systems design throughout my 25 years in the field, starting from the theory of distributed systems (98-01), immersing into the practice of wireless sensor networks (01-11), and working on cloud computing systems both in the academia and industry ever since. These heuristic principles have been applied knowingly or unknowingly and has proven...
Read more >>

The Agentic Self: Parallels Between AI and Self-Improvement

-
2025 was the year of the agent. The goalposts for AGI shifted; we stopped asking AI to merely "talk" and demanded that it "act". As an outsider looking at the architecture of these new agents and agentic system, I noticed something strange. The engineering tricks used to make AI smarter felt oddly familiar. They read less like computer science and more like … self-help advice . The secret to agentic intelligence seems to lie in three very human habits: writing things down, talking to yourself, and pretending to be someone else. They are almost too simple. The Unreasonable Effectiveness of Writing One of the most profound pieces of advice I ever read as a PhD student came from Prof. Manuel Blum, a Turing Award winner. In his essay "Advice to a Beginning Graduate Student", he wrote: "Without writing, you are reduced to a finite automaton. With writing you have the extraordinary power of a Turing machine." If you try to hold a complex argument enti...
Read more >>

Learning about distributed systems: where to start?

-
This is definitely not a "learn distributed systems in 21 days" post. I recommend a principled, from the foundations-up, studying of distributed systems, which will take a good three months in the first pass, and many more months to build competence after that. If you are practical and coding oriented you may not like my advice much. You may object saying, "Shouldn't I learn distributed systems with coding and hands on? Why can I not get started by deploying a Hadoop cluster, or studying the Raft code." I think that is the wrong way to go about learning distributed systems, because seeing similar code and programming language constructs will make you think this is familiar territory, and will give you a false sense of security. But, nothing can be further from the truth. Distributed systems need radically different software than centralized systems do.  --A. Tannenbaum This quotation is literally the first sentence in my distributed systems syllabus. Inst...
Read more >>

5 Lessons at 50

-
Image
Looking at my peak male physique, and my Keanu Reeves baby face, you would never suspect it, but I recently turned 50. As is the tradition, I thought about writing a post titled "50 Lessons at 50". Unfortunately, I don't have that kind of wisdom. The thing is, I still feel like I'm 18, same age as my son. Turns out this is the secret old guys have been hiding from us all along. You get older only on the outside, but inside you still see yourself as the same young lad. Still, fifty years should count for something. So what did I actually learn? How am I mentally different than my 18-year-old self. Here is my attempt to tally it up.  1. Caution is warranted I finally understand my parents. As you age, you accumulate battle scars, and the scars turn into habits. Anything that can go wrong will go wrong. You forget the cooktop on once, and suddenly you check it three times before leaving the house. True story. You stop diving head-first into a pile of leaves, because ther...
Read more >>

Foundational distributed systems papers

-
I talked about the importance of reading foundational papers last week. To followup, here is my compilation of foundational papers in the distributed systems area. (I focused on the core distributed systems area, and did not cover networking, security, distributed ledgers, verification work etc. I even left out distributed transactions, I hope to cover them at a later date.)  I classified the papers by subject, and listed them in chronological order. I also listed expository papers and blog posts at the end of each section. Time and State in Distributed Systems Time, Clocks, and the Ordering of Events in a Distributed System. Leslie Lamport, Commn. of the ACM,  1978. Distributed Snapshots: Determining Global States of a Distributed System. K. Mani Chandy Leslie Lamport, ACM Transactions on Computer Systems, 1985. Virtual Time and Global States of Distributed Systems.  Mattern, F. 1988. Practical uses of synchronized clocks in distributed systems. B. Liskov, 1991. Exp...
Read more >>

Building a Database on S3

-
Image
Hold your horses, though. I'm not unveiling a new S3-native database. This paper is from 2008. Many of its protocols feel clunky today. Yet it nails the core idea that defines modern cloud-native databases: separate storage from compute. The authors propose a shared-disk design over Amazon S3, with stateless clients executing transactions. The paper provides a blueprint for serverless before the term existed. SQS as WAL and S3 as Pagestore The 2008 S3 was painfully slow, and 100 ms reads weren't unusual. To hide that latency, the database separates "commit" from "apply". Clients write small, idempotent redo logs to Amazon Simple Queue Service (SQS) instead of touching S3 directly. An asynchronous checkpoint by a client applies those logs to B-tree pages on S3 later. This design shows strong parallels to modern disaggregated architectures . SQS becomes the write-ahead log (WAL) and logstore. S3 becomes the pagestore. Modern Aurora follows a similar logic : t...
Read more >>

Cloudspecs: Cloud Hardware Evolution Through the Looking Glass

-
Image
This paper (CIDR'26) presents a comprehensive analysis of cloud hardware trends from 2015 to 2025, focusing on AWS and comparing it with other clouds and on-premise hardware. TL;DR: While network bandwidth per dollar improved by one order of magnitude (10x), CPU and DRAM gains (again in performance per dollar terms) have been much more modest. Most surprisingly, NVMe storage performance in the cloud has stagnated since 2016. Check out the NVMe SSD discussion below for data on this anomaly. CPU Trends Multi-core parallelism has skyrocketed in the cloud. Maximum core counts have increased by an order of magnitude over the last decade. The largest AWS instance u7in now boasts 448 cores. However, simply adding cores hasn't translated linearly into value. To measure real evolution, the authors normalized benchmarks (SPECint, TPC-H, TPC-C) by instance cost. SPECint benchmarking shows that cost-performance improved roughly 3x over ten years. A huge chunk of that gain comes from AWS G...
Read more >>

Specula: Scaling formal specifications for autonomous model checking of system code

-
Image
Specula is an agentic system that automates the process of software bug finding through authoring and model-checking a spec for the code. It derives TLA+ specifications automatically from the code, checks code-spec conformance through trace validation, model checks the spec to find concurrency bugs, and reproduces the bug at the code layer by writing integration tests with precise timing. I remember reading the Daikon paper "Quickly detecting relevant program invariants" in 2000 and getting impressed by it, and here we are after 26 years, solving the end-to-end problem much better than I ever thought would be possible in a push-button manner in the year of our lord 2026. But somehow, I am still somewhat unsatisfied with the paper. This may be me being hypercritical and trying to get more out of the paper by arguing with it . So bear with me until I resolve (or learn to accept) these problems over time. I know many of the authors of the Specula work, and respect them, and I...
Read more >>

Aurora DSQL: Scalable, Multi-Region OLTP

-
Image
The Aurora DSQL paper finally dropped . Reading it yesterday was an interesting experience, because I spent two years (2022-23) working with the AWS team that designed and built Aurora DSQL. Since I have been very familiar with the architecture, the paper's overview description of the system didn't excite me much. And if I am being honest, reading the paper also felt a bit dry, most likely because I am not doing the usual extra thinking to explore/understand the ideas in the paper. But taking a step back today, and leaving my subjective experience aside, I will try to elaborate on how the DSQL architecture is actually built on a set of aggressive and opinionated engineering bets.  If I had to explain this architecture in a single sentence (just as I used to do for other teams at AWS) it would be this: We took a traditional monolithic database and blew out every single component into an independent, horizontally scalable service. Exploding the Monolith DSQL divides the database ...
Read more >>

What I'd do as a College Freshman in 2025

-
Do Computer Science Absolutely. Still would. Many are spooked by LLMs. Some, like Jensen Huang, argue that "nobody has to learn how to program."   I argue the opposite . And I double down.  Being supported by AI tools is not a substitute for mastery . You can’t borrow skills. You have to earn them.   Computer science builds vital skills : hacking, debugging, abstract thinking, and quick adaptation. These don’t go out of style.   Do STEM. It’s LLM-resistant. LLMs can retrieve and remix information, but do you know what to do with them? Like the dog chasing the car, what now? STEM teaches you that. It teaches you to think, to reason, to act. It gets you from information to wisdom. But only after you've mastered the foundations. We're heading into the age of π-shaped people: depth in two areas, and generalist across . Building depth first, and then ranging is good strategy .   So yes, I would learn the foundations of both CS and AI. And then do AI + X, where X is s...
Read more >>