Posts

Showing posts matching the search for HPTS

HPTS trip report (day 2)

-
This is a belated wrap up of my HPTS trip report. Read the first part of the HPTS trip report here, if you haven't before. The first day of HPTS was a very busy, productive, and tiring day. Since my body clock was still on East Coast time, I was wide awake at 6am again. I went for a longer run this time. I ran for 5 miles along the beach on the aptly named Sunset drive. Monterey Bay, Asilomar is an eerily beautiful piece of earth. After breakfast, HPTS started with the data analytics session. That night it was announced that Chris Re (from Stanford) won a  MacArthur genius award . Chris was scheduled to talk in this first session of HPTS day 2. It was a very engaging talk. Chris talked about the DeepDive macroscopic data system they have been working on.  DeepDive is a system that reads texts and research papers and collect macroscopic knowledge about a topic with a quality that exceeds paid human annotators and volunteers. This was surprising news to me; another victor...
Post a Comment
Read more >>

HPTS'17 day 0

-
Image
A couple weeks ago, I attended HPTS'17. As I wrote in my HTPS'15 posts ( day1 and day2 ), HPTS is an unconventional workshop. "Every two years, HPTS brings together a lively and opinionated group of participants to discuss and debate the pressing topics that affect today's systems and their design and implementation, especially where scalability is concerned. The workshop includes position paper presentations, panels, moderated discussions, and significant time for casual interaction. The only publications are slide decks by presenters who choose to post them." HPTS is by invitation only and keeps it under 100 participants. The workshop brings together experts from both industry and academia so that they can mix and interact. Although some people prefer to keep what happens at HPTS to stay at HPTS, I find it hard to not talk about HPTS. I learn a lot at HPTS and I want to share at least some of those. And this year I don't think there was any confidential t...
1 comment
Read more >>

HPTS trip report (days 0 and 1)

-
Image
Last week, from Sunday to Tuesday night, I attended the  16th International Workshop on High Performance Transaction Systems (HPTS). HPTS is an unconventional workshop.  "Every two years, HPTS brings together a lively and opinionated group of participants to discuss and debate the pressing topics that affect today's systems and their design and implementation, especially where scalability is concerned. The workshop includes position paper presentations, panels, moderated discussions, and significant time for casual interaction. The only publications are slide decks by presenters who choose to post them."  HPTS is by invitation only and keeps it under 100 participants. The workshop brings together experts from both industry and academia so they mix and interact. Looking at the program committee, you can see names of entrepreneurs venture capitalists ( David Cheriton , Sequoia Capital), large web companies (Google, Facebook, Salesforce, Cloudera), and academics. HPTS is le...
Post a Comment
Read more >>

HPTS'24 Day 1, part 1

-
Image
Wow, what a week that was! The two days of HPTS (Monday and Tuesday this week) felt like a week to me. I learned a lot, and had a lot of good conversations, and even was able to squeeze in some beach walks in there.  HPTS has been operating since 1985, convenening mostly every two years. It has been described as Davos for database systems. Pat Helland has been running HPTS since 1989, and as usual he kicked it off with some short opening remarks 8:30am Monday morning.  Pat reminisced on the early HPTSs which discussed cutting edge work on scalable computing, punching past 100 txn on a mainframe! Then he quickly set the context of the workshop. This event is about meeting people. There are 130 people, with 20+ of them students. He said "your job is to meet everyone, HPTS is about the connections you make". The HPTS website also emphasizes this. "HPTS is about the breaks. The presentations punctuate the breaks! HPTS exists to promote community and relationships. This comm...
Post a Comment
Read more >>

HPTS'17 day 2

-
On HPTS day 2, there were 4 presentation sessions (I mention 2 below) and an evening session on remembering Jim Gray and Ed Lassettre. (Follow these links for HPTS'17 day 0 and day 1 .) Verification of Systems sesion The first talk was Jepsen VIII by Kyle Kingsbury, who breaks databases for a living. He gave a very fast paced talk. A good rule of thumb for presentations is to go with 2 minutes per slide. Kyle flips this rule upside down and then goes even further to present 5 slides per minute. He presented 150+ slides in less than 30 minutes , and somehow he made this work. I can't believe how quickly/smoothly he was able to transition from a slide to the next, and how he managed to memorize all those transitions. Kyle's Jepsen toolkit tests databases as blackboxes using a client to submit overlapping operations, where the start/end of operations define the operation intervals. To prepare these tests, Kyle first carefully reads through the documentation to see wh...
Post a Comment
Read more >>

HPTS day 2, part 2

-
Image
Continuing with our HPTS series . This is now the afternoon of second day. The first session was on HTAP and streaming, and the second one on caching.  Session 7: HTAP and Streaming  Who cares about HTAP? - Tianyu Li (MIT) Tianyu argued that while Hybrid Transactional/Analytical Processing (HTAP) showed great promise in 2014, it has failed to make a significant impact in the decade since. Instead, he proposed that the real disruption in the database world is coming from the migration of workloads to the cloud, with companies like Snowflake and Databricks leading the charge. The trend is moving towards virtualization and cloud-native architectures. The trend has been for composing specialized data engines connected by pipelines and streams, rather than relying on monolithic HTAP systems. He highlighted the MITDBG projects on developing modern abstractions and architectures for cloud-native data processing. Amazon Zero-ETL - Gopal Paliwal and Gokul Soundararajan (AWS) This tal...
Post a Comment
Read more >>

Notes from USENIX NSDI Days 2 & 3

-
Before talking about Days 2 & 3, here are my notes from NSDI day 1 if you are interested. Again all of the papers mentioned (and omitted) below are accessible publicly at the NSDI web page . Enjoy! NSDI day 2  Day 2 had the following sessions: web and video performance isolation and scaling congestion control cloud Here are the papers I took detailed notes on. I ordered them by how interesting I found them. Towards Battery-Free HD Video Streaming This paper is by Saman Naderiparizi, Mehrdad Hessar, Vamsi Talla, Shyamnath Gollakota, and Joshua R Smith, University of Washington. This work was mind blowing for me. This is a strong group that has developed the Wisp motes before, but even then the presented battery-free video streaming sensors was very impressive and the talk included a live demo of them. So here is the deal. The goal of this work is to design sticker form factor, battery-free cameras. But that is crazy right? Video streaming is power hungry...
Post a Comment
Read more >>

HPTS'17 day 1

-
This post is a continuation from yesterday's post. I cover only some of the talks, you may check the other talks from the HPTS agenda. Mind your state for your state of mind (Pat Helland -Salesforce) HPTS day 1 started with a keynote from Pat Helland. Pat Helland is a database veteran. I had covered some of his papers on this blog before. He writes insightful position papers with a unique style. The keynote provided an overview of trends in storage and computing and hit the high notes from his earlier position papers as well, and mentioned this table:                           | fast-reads | fast-writes | read-your-write linearizable       | no             | no              | yes            nonlinearizable | yes            | yes          ...
Post a Comment
Read more >>

HPTS day 2, part 1

-
Image
Continuing with our series . This is day 2, Tuesday morning. It had two session on Hardware. I wasn't exaggerating when I said hardware/software codesign was all the buzz at HPTS this year. It looks like future databases will be more tightly integrated with hardware capabilities and more responsive to user needs. You may have gotten a bit tired of all these technical paper summaries, so let's start with a preamble on the venue. HPTS has always been held at Asilomar, Pacific Grove. It is very close to Monterey Bay. The conference grounds is old, the Chapel being more than a century old. The rooms are built of wood, and walls are paper thin, with no sound insulation. What do you say, the place has a lot of character! Joking aside, it grows on you. It is clean and the food is good. It is on the beach, which has surreal vegetation. Plants and occasionally trees grow literally on the white sands. The trees have all bent down tracing the blow of the strong winds from the sea to the i...
Post a Comment
Read more >>

Popular posts from this blog

Hints for Distributed Systems Design

-
This is with apologies to Butler Lampson, who published the " Hints for computer system design " paper 40 years ago in SOSP'83. I don't claim to match that work of course. I just thought I could draft this post to organize my thinking about designing distributed systems and get feedback from others. I start with the same  disclaimer Lampson gave. These hints are not novel, not foolproof recipes, not laws of design, not precisely formulated, and not always appropriate. They are just hints.  They are context dependent, and some of them may be controversial. That being said, I have seen these hints successfully applied in distributed systems design throughout my 25 years in the field, starting from the theory of distributed systems (98-01), immersing into the practice of wireless sensor networks (01-11), and working on cloud computing systems both in the academia and industry ever since. These heuristic principles have been applied knowingly or unknowingly and has proven...
Read more >>

The Agentic Self: Parallels Between AI and Self-Improvement

-
2025 was the year of the agent. The goalposts for AGI shifted; we stopped asking AI to merely "talk" and demanded that it "act". As an outsider looking at the architecture of these new agents and agentic system, I noticed something strange. The engineering tricks used to make AI smarter felt oddly familiar. They read less like computer science and more like … self-help advice . The secret to agentic intelligence seems to lie in three very human habits: writing things down, talking to yourself, and pretending to be someone else. They are almost too simple. The Unreasonable Effectiveness of Writing One of the most profound pieces of advice I ever read as a PhD student came from Prof. Manuel Blum, a Turing Award winner. In his essay "Advice to a Beginning Graduate Student", he wrote: "Without writing, you are reduced to a finite automaton. With writing you have the extraordinary power of a Turing machine." If you try to hold a complex argument enti...
Read more >>

Learning about distributed systems: where to start?

-
This is definitely not a "learn distributed systems in 21 days" post. I recommend a principled, from the foundations-up, studying of distributed systems, which will take a good three months in the first pass, and many more months to build competence after that. If you are practical and coding oriented you may not like my advice much. You may object saying, "Shouldn't I learn distributed systems with coding and hands on? Why can I not get started by deploying a Hadoop cluster, or studying the Raft code." I think that is the wrong way to go about learning distributed systems, because seeing similar code and programming language constructs will make you think this is familiar territory, and will give you a false sense of security. But, nothing can be further from the truth. Distributed systems need radically different software than centralized systems do.  --A. Tannenbaum This quotation is literally the first sentence in my distributed systems syllabus. Inst...
Read more >>

5 Lessons at 50

-
Image
Looking at my peak male physique, and my Keanu Reeves baby face, you would never suspect it, but I recently turned 50. As is the tradition, I thought about writing a post titled "50 Lessons at 50". Unfortunately, I don't have that kind of wisdom. The thing is, I still feel like I'm 18, same age as my son. Turns out this is the secret old guys have been hiding from us all along. You get older only on the outside, but inside you still see yourself as the same young lad. Still, fifty years should count for something. So what did I actually learn? How am I mentally different than my 18-year-old self. Here is my attempt to tally it up.  1. Caution is warranted I finally understand my parents. As you age, you accumulate battle scars, and the scars turn into habits. Anything that can go wrong will go wrong. You forget the cooktop on once, and suddenly you check it three times before leaving the house. True story. You stop diving head-first into a pile of leaves, because ther...
Read more >>

Foundational distributed systems papers

-
I talked about the importance of reading foundational papers last week. To followup, here is my compilation of foundational papers in the distributed systems area. (I focused on the core distributed systems area, and did not cover networking, security, distributed ledgers, verification work etc. I even left out distributed transactions, I hope to cover them at a later date.)  I classified the papers by subject, and listed them in chronological order. I also listed expository papers and blog posts at the end of each section. Time and State in Distributed Systems Time, Clocks, and the Ordering of Events in a Distributed System. Leslie Lamport, Commn. of the ACM,  1978. Distributed Snapshots: Determining Global States of a Distributed System. K. Mani Chandy Leslie Lamport, ACM Transactions on Computer Systems, 1985. Virtual Time and Global States of Distributed Systems.  Mattern, F. 1988. Practical uses of synchronized clocks in distributed systems. B. Liskov, 1991. Exp...
Read more >>

Building a Database on S3

-
Image
Hold your horses, though. I'm not unveiling a new S3-native database. This paper is from 2008. Many of its protocols feel clunky today. Yet it nails the core idea that defines modern cloud-native databases: separate storage from compute. The authors propose a shared-disk design over Amazon S3, with stateless clients executing transactions. The paper provides a blueprint for serverless before the term existed. SQS as WAL and S3 as Pagestore The 2008 S3 was painfully slow, and 100 ms reads weren't unusual. To hide that latency, the database separates "commit" from "apply". Clients write small, idempotent redo logs to Amazon Simple Queue Service (SQS) instead of touching S3 directly. An asynchronous checkpoint by a client applies those logs to B-tree pages on S3 later. This design shows strong parallels to modern disaggregated architectures . SQS becomes the write-ahead log (WAL) and logstore. S3 becomes the pagestore. Modern Aurora follows a similar logic : t...
Read more >>

Cloudspecs: Cloud Hardware Evolution Through the Looking Glass

-
Image
This paper (CIDR'26) presents a comprehensive analysis of cloud hardware trends from 2015 to 2025, focusing on AWS and comparing it with other clouds and on-premise hardware. TL;DR: While network bandwidth per dollar improved by one order of magnitude (10x), CPU and DRAM gains (again in performance per dollar terms) have been much more modest. Most surprisingly, NVMe storage performance in the cloud has stagnated since 2016. Check out the NVMe SSD discussion below for data on this anomaly. CPU Trends Multi-core parallelism has skyrocketed in the cloud. Maximum core counts have increased by an order of magnitude over the last decade. The largest AWS instance u7in now boasts 448 cores. However, simply adding cores hasn't translated linearly into value. To measure real evolution, the authors normalized benchmarks (SPECint, TPC-H, TPC-C) by instance cost. SPECint benchmarking shows that cost-performance improved roughly 3x over ten years. A huge chunk of that gain comes from AWS G...
Read more >>

Specula: Scaling formal specifications for autonomous model checking of system code

-
Image
Specula is an agentic system that automates the process of software bug finding through authoring and model-checking a spec for the code. It derives TLA+ specifications automatically from the code, checks code-spec conformance through trace validation, model checks the spec to find concurrency bugs, and reproduces the bug at the code layer by writing integration tests with precise timing. I remember reading the Daikon paper "Quickly detecting relevant program invariants" in 2000 and getting impressed by it, and here we are after 26 years, solving the end-to-end problem much better than I ever thought would be possible in a push-button manner in the year of our lord 2026. But somehow, I am still somewhat unsatisfied with the paper. This may be me being hypercritical and trying to get more out of the paper by arguing with it . So bear with me until I resolve (or learn to accept) these problems over time. I know many of the authors of the Specula work, and respect them, and I...
Read more >>

Aurora DSQL: Scalable, Multi-Region OLTP

-
Image
The Aurora DSQL paper finally dropped . Reading it yesterday was an interesting experience, because I spent two years (2022-23) working with the AWS team that designed and built Aurora DSQL. Since I have been very familiar with the architecture, the paper's overview description of the system didn't excite me much. And if I am being honest, reading the paper also felt a bit dry, most likely because I am not doing the usual extra thinking to explore/understand the ideas in the paper. But taking a step back today, and leaving my subjective experience aside, I will try to elaborate on how the DSQL architecture is actually built on a set of aggressive and opinionated engineering bets.  If I had to explain this architecture in a single sentence (just as I used to do for other teams at AWS) it would be this: We took a traditional monolithic database and blew out every single component into an independent, horizontally scalable service. Exploding the Monolith DSQL divides the database ...
Read more >>

What I'd do as a College Freshman in 2025

-
Do Computer Science Absolutely. Still would. Many are spooked by LLMs. Some, like Jensen Huang, argue that "nobody has to learn how to program."   I argue the opposite . And I double down.  Being supported by AI tools is not a substitute for mastery . You can’t borrow skills. You have to earn them.   Computer science builds vital skills : hacking, debugging, abstract thinking, and quick adaptation. These don’t go out of style.   Do STEM. It’s LLM-resistant. LLMs can retrieve and remix information, but do you know what to do with them? Like the dog chasing the car, what now? STEM teaches you that. It teaches you to think, to reason, to act. It gets you from information to wisdom. But only after you've mastered the foundations. We're heading into the age of π-shaped people: depth in two areas, and generalist across . Building depth first, and then ranging is good strategy .   So yes, I would learn the foundations of both CS and AI. And then do AI + X, where X is s...
Read more >>