Metadata

As we enter the age of AI, it becomes more important for knowledge workers to excel in their strengths, and aim big for some strikes instead of settling for a comfortable average all around. How can colleges reform to endow graduates better? Here are my opinions, for what they are worth. Human skills for the AI era In the age of AI, doing rather than knowing becomes more important. Shallow information is worthless, but mastery of principles, critical thinking, and synthesis is priceless. Colleges should teach collaboration, entrepreneurship/innovation, communication/writing, and critical thinking and problem solving skills. How can colleges reform to cultivate these skills? First, they should transition from zero-sum mentality to the win-win mentality. This is not easy, because the system has been built on making students compete against each other and stack-ranking them. I don't know what kind of structural changes and scaffolding can help for this. I have some practical advice

This paper got the best paper award at SOCC 2021 . The paper conducts a comprehensive study of large scale microservices deployed in Alibaba clusters.  They analyze the behavior of more than 20,000 microservices in a 7-day period and profile their characteristics based on the 10 billion call traces collected. They find that: microservice graphs are dynamic in runtime most graphs are scattered to grow like a tree size of call graphs follows a heavy-tail distribution Based on their findings, they offer some practical tips about improving microservice runtime performance. They also develop a stochastic model to simulate microservice call graph dependencies and show that it approximates the dataset they collected (which is available at https://github.com/alibaba/clusterdata ). What are microservices? Microservices is a software development approach that divides an application into independently deployable services, owned by small teams organized around business capabilities. Each service c

This paper is from VLDB2020. Aria is an OLTP database that does epoch-based commits similar to the Silo paper we discussed last week. Unlike Silo, which was a single-node database, Aria is a distributed and deterministic database. Aria's biggest contribution is that it improves on Calvin by being able to run transactions without prior knowledge of read and write sets. Another nice thing in Aria is its deterministic re-ordering mechanism to commit transactions in an order that reduces the number of conflicts. Evaluation results on YCSB and TPC-C show that Aria outperforms other protocols by a large margin on a single node and up to a factor of two on a cluster of eight nodes. Aria versus Calvin Recall that Calvin uses locks ( here is a summary of the Calvin paper ). The key idea in Calvin is that read/write locks for a transaction are acquired according to the ordering of input transactions and the transaction is assigned to a worker thread for execution once all needed locks are gr

This paper is from SOSP'13. The Polyjuice paper, which we studied last week, built on the Silo codebase and commit protocol, which led me to read the Silo paper. Silo is a single-node multi-core in-memory database. It avoids all centralized contention points, including centralized transaction ID (TID) assignment. Silo's key contribution is a commit protocol based on optimistic concurrency control (OCC) that provides serializability while avoiding all shared-memory writes for records that were only read. Logging and recovery is provided by linking periodically-updated epochs with the commit protocol. Silo achieves almost 700,000 transactions per second on a standard TPC-C workload mix on a 32-core machine, as well as near-linear scalability. That is pretty impressive over 2013 hardware. The Silo paper got 445 citations in 10 years. That is also impressive. So let's dive in. Architecture Silo's organization is typical of databases. Tables are implemented as collections

This paper appeared in OSDI 2021 . I really like this paper. It is informative, it taught me new things about concurrency control techniques. It is novel, it shows a practical application of simple machine learning to an important systems problem, concurrency control. It shows significant benefits for a limited but reasonable setup. It has good evaluation coverage and explanation. It is a good followup paper to the benchmarking papers we have been looking at recently. So let's go on learning about how Polyjuice was brewed. Problem and motivation There is no supreme concurrency control (CC) algorithm for all conditions. Different CC algorithms return the best outcome under different conditions.  Consider two extreme CC algorithms. Two phase locking (2PL) waits-for every dependent transactions to finish. Optimistic concurrency control (OCC) don't track or wait for any dependent transaction but validate at the end. We find that OCC is better with less contention, because it avoids

I realize I haven't done a book review post recently. The last one was from September 2021 . But I have been reading (and listening to) books. Well, recently mostly unsuccessfully. Here are some of them in reverse chronological order. (I think there must be some other non-fiction books I read, but I can't remember now.) It (Stephen King, 1986) The horror. Oh my God, it doesn't end... The book doesn't end. I won't spoil the book for you, because I couldn't get to the ending. I borrowed the book in audiobook, and extended twice. It was good listening at the car, while driving for chores. But I only have so much stamina. The book keeps dragging on. I gave up. I will just watch the movie with my son this weekend. Build (Tony Fadell, 2022) This is another book that won't end. The book felt massive. I extended the renewal for 2-3 times, but I couldn't finish the book. This book is actually multiple books bundled together. You can learn about product managemen

This paper is from EDBT 2016 . Database veterans would know of the tradeoffs mentioned here, but it is very useful to have this "systematization of knowledge" paper, because it gives a good mental framework for new-comers to the field, and seeing practical/implicit tradeoffs explicitly stated can create an epiphany for even the veterans. The problem Algorithms and data structures for organizing and accessing data are called access methods. Database research/development community has been playing catchup redesigning and tuning access methods to accommodate changes to hardware and workload. As data generation and workload diversification grow exponentially, and hardware advances introduce increased complexity, the effort for the redesigning and tuning have been accumulating as well. The paper suggests it is important to solve this problem once and for all by identifying tradeoffs access methods face, and designing access methods that can adapt and autotune the structures and te

This paper appeared in VLDB2022 and is a good followup to our TPC-E vs TPC-C post . The paper investigates the following question: Many articles compare to prior works under certain settings, but how much of their conclusions hold under other settings? Their methodology is as follows. They used TPC-C and YCSB benchmarks as they are most widely used. They reproduced and evaluated 11 work (see Table 1). They then tuned benchmark parameters and system features to study effects on the performance of these work. They find that the evaluations of these work (and conclusions drawn from them) are sensitive to experiment settings. They make some recommendations as to how to proceed for evaluation of future systems work. The paper is well written and is an important contribution to systems work. In my summary below, I use many sentences from the paper verbatim. Analysis of TPC-C results TPC-C simulates the database of a wholesale company. It includes a number of warehouses, each maintaining sto