Metadata

Chapter 6 of the Designing Data Intensive Applications (DDIA) book discusses partitioning, a key technique for scaling large datasets and high query throughput in distributed databases. By breaking data into smaller partitions, it can be distributed across multiple nodes in a shared-nothing cluster. This allows the storage and processing load to be spread across many disks and processors. Fun fact: Partitioned databases were pioneered in the 1980s by products such as Teradata and Tandem NonStop SQL, and in 2000s rediscovered by NoSQL databases and Hadoop-based data warehouses. Partitioning is often combined with replication , where each partition is stored on multiple nodes for fault tolerance. In a typical setup, each partition has a leader node that handles writes, with follower nodes replicating the data. Partitioning of Key-Value Data There are two main approaches to partitioning key-value data. Partitioning by key range: Assign a continuous range of keys to each partition, like v

Continuing with our HPTS series . This is now the afternoon of second day. The first session was on HTAP and streaming, and the second one on caching.  Session 7: HTAP and Streaming  Who cares about HTAP? - Tianyu Li (MIT) Tianyu argued that while Hybrid Transactional/Analytical Processing (HTAP) showed great promise in 2014, it has failed to make a significant impact in the decade since. Instead, he proposed that the real disruption in the database world is coming from the migration of workloads to the cloud, with companies like Snowflake and Databricks leading the charge. The trend is moving towards virtualization and cloud-native architectures. The trend has been for composing specialized data engines connected by pipelines and streams, rather than relying on monolithic HTAP systems. He highlighted the MITDBG projects on developing modern abstractions and architectures for cloud-native data processing. Amazon Zero-ETL - Gopal Paliwal and Gokul Soundararajan (AWS) This talk focused

Continuing with our series . This is day 2, Tuesday morning. It had two session on Hardware. I wasn't exaggerating when I said hardware/software codesign was all the buzz at HPTS this year. It looks like future databases will be more tightly integrated with hardware capabilities and more responsive to user needs. You may have gotten a bit tired of all these technical paper summaries, so let's start with a preamble on the venue. HPTS has always been held at Asilomar, Pacific Grove. It is very close to Monterey Bay. The conference grounds is old, the Chapel being more than a century old. The rooms are built of wood, and walls are paper thin, with no sound insulation. What do you say, the place has a lot of character! Joking aside, it grows on you. It is clean and the food is good. It is on the beach, which has surreal vegetation. Plants and occasionally trees grow literally on the white sands. The trees have all bent down tracing the blow of the strong winds from the sea to the i

This paper from SOSP 2011 describes a distributed storage system called Walter. Walter targets web applications that operate across multiple geographic sites, and it aims to balance consistency and latency in such systems by providing strong consistency within a site, and weaker consistency across sites. Parallel Snapshot Isolation (PSI) The paper introduces a new isolation property called Parallel Snapshot Isolation (PSI). PSI ensures consistent snapshots and transaction ordering within a site, causal ordering across sites, and prevention of write-write conflicts. PSI extends snapshot isolation (see Fig 3 for illustration of snapshot isolation, and here for an example of snapshot isolated database modeling ) by allowing different sites to have different commit orderings. For example, suppose site A executes transactions T1, T2 and site B executes transactions T3, T4. PSI allows site A to first incorporate just T1, T2 and later T3, T4, while site B first incorporates T3, T4 and later

This is part 2 of day 1 of HPTS'24. (You can tell I did some lisp programming back in the day, huh?) Here is the first part of day 1 , you should check that out as well.  There were 2 sessions each with 4 talks in the afternoon of day 2. After dinner, we had a gong show presentation on miscellaneous topics as well.  Session 3: DBOS Virtual Memory: a Huge Step Backwards- Daniel Bittman (Elephance) Daniel argued that virtual memory, despite its widespread use, is fundamentally flawed. He contends that virtual memory presents an abstraction of memory as  flat, uniform, contiguous, infinite, isolated, and fast - all of which are untrue.  He pointed to significant performance issues with virtual memory, noting that virtual memory adds substantial overhead to every memory access. Every memory access costs 4 extra memory accesses and 100s of nanoseconds of wasted CPU time, which is an eternity in CPU world. He criticized the Transaction Look-Aside Buffer (TLB) for its complexity and lack

Wow, what a week that was! The two days of HPTS (Monday and Tuesday this week) felt like a week to me. I learned a lot, and had a lot of good conversations, and even was able to squeeze in some beach walks in there.  HPTS has been operating since 1985, convenening mostly every two years. It has been described as Davos for database systems. Pat Helland has been running HPTS since 1989, and as usual he kicked it off with some short opening remarks 8:30am Monday morning.  Pat reminisced on the early HPTSs which discussed cutting edge work on scalable computing, punching past 100 txn on a mainframe! Then he quickly set the context of the workshop. This event is about meeting people. There are 130 people, with 20+ of them students. He said "your job is to meet everyone, HPTS is about the connections you make". The HPTS website also emphasizes this. "HPTS is about the breaks. The presentations punctuate the breaks! HPTS exists to promote community and relationships. This comm