Metadata

HotNets'18 was held at Microsoft Research, Building 99. This is walking distance to my office at Cosmos DB, where I am working at my sabbatical. So I got tempted and crushed this workshop for a couple sessions. And oh my God, am I happy I did it. The space session was particularly very interesting, and definitely worth writing about. My God, it is full of satellites! According to a 2018 estimate, there are 4,900 satellites in orbit , of which about 1,900 operational, while the rest have lived out their useful lives and become space debris. Approximately 500 operational satellites are in l ow-Earth orbit , 50 are in medium-Earth orbit (at 20,000 km), and the rest are in geostationary orbit (at 36,000 km) . The low earth orbit LEO satellites are not stationary and fast moving around the earth at 1.5 hour per rotation. We are talking about the lowest ring in this picture , where International Space Station (ISS) resides. Since LEO satellites are close to Earth, this make...

This is a follow up to my "Master your tools" post . As an example of one of my tools, I talk about my Emacs use. I have been using Emacs for the last 20 years. At this point, I don't even know Emacs, my fingers do. If you ask me the shortcut for something, I will need to let my fingers do it and try to observe what they are doing. And sometimes ---as in the story of the caterpillar who forgets how to walk when asked to demonstrate it--- I forget about how to do something when I try to attempt it consciously. Four stages of competence: 4. Unconscious Competence (Right Intuition) 3. Conscious Competence (Right Analysis) 2. Conscious Incompetence (Wrong Analysis) 1. Unconscious Incompetence (Wrong Intuition) https://t.co/peAEUtkMZV pic.twitter.com/OxZ3WqreVa — Colby Serpa💡 (@ColbySerpa) November 6, 2018   From text-editing to text-wrangling I have been learning Emacs at a glacial pace, but I think that worked for me better. I figured I can internalize so...

For the past 7-8 years, several research papers have used this example to motivate causal consistency. You must have read this example, right? Alice removes her boss from her friend list, and posts that on her feed that she is looking for a new job.  Tom removes his mom from the friend list, and posts his Spring Break photos. Well, being the empirical researchers we are, Aleksey and I wanted to put Twitter in to test for this scenario. On September 25, we performed this test. (I also have a video recording of this. But since I can't stand to hear myself talk in recordings, I am not posting it. I sound really weird, man.) I first blocked Aleksey on my Twitter account, and then tweeted that Aleksey drinks a lot of tea (it's true). When we checked Aleksey's timeline, we saw that his timeline indeed did not display my tweet. @alekseycharapko drinks way too much tea. — Murat Demirbas (@muratdemirbas) September 25, 2018 So, this was kind of an anticlimax. Tw...

There are a lot of interesting meetups at Seattle, and I try to attend one every couple weeks. Ruben Lozano Aguilera was the speaker for this meetup on Oct 17. Ruben is a product manager at Google Cloud, and before that he was a product manager at Amazon. What is ML? Programming transforms data + rules into answers. Machine learning turns data + answers into rules. When should you use ML? Use ML if the problem: handles complex logic scales up really fast requires specialized personalization adapts in real-time For example ML is a good fit for the "search" problem.  Search requires complex logic, for which it is not easy to develop rules.  It scales up really fast in terms of new keywords, combinations and content. It requires personalization depending on the context, and has some real-time adaptation component as well. Another important point is that the problem should have existing examples of actual answers. When you bootstrap from a good enough dataset...

In the last decade, the Paxos protocol family grew with the addition of new categories. Rotating leader: Mencius Leaderless: EPaxos, Fast Paxos Paxos federations: Spanner , vertical Paxos  Dynamic key-leader: WPaxos   This paper, which appeared in SOCC 18, proposes SDPaxos which prescribes separating the control plane (single leader) from the replication plane (multiple leaders). SD in SDPaxos stands for "semi-decentralized". The motivation for this stems from the following observation. Single leader Paxos approach has a centralized leader and runs into performance bottleneck problems. On the other hand, the leaderless (or opportunistic multileader) approach is fully decentralized but suffers from the conflicting command problems. Taking a hybrid approach to capture the best of both worlds, SDPaxos makes the command-leaders to be decentralized (the closest replica can lead the command), but the ordering-leader (i.e., the sequencer) is still centralized/unique in t...

Sometime ago, I was visiting an apartment block. This was a relatively new apartment block, I think less than 5 years old. I entered the elevator, and I heard "Hello, hello... Are you there?" booming from the speakers. This never happened to me before of course; elevators don't talk to me often. I thought maybe someone had pressed the call for help button, and left the elevator, and I can help sort this out. I said, "Hi, yes!" The voice continued saying: -"The reason I am calling you is because of your last energy bill". After a second of cognitive dissonance, I started laughing: -"Umm, I don't know what happened, but I am in an elevator now, and your voice is literally streaming from the elevator." The voice stopped for a brief couple of seconds, then continued explaining something about the bill, following the script he was given. -"Dude, this is my floor, and I got to go now. Ok, bye." It turns out most eleva...

Let's say you want to crowdsource the answer to the question: (Q1) What is the capital of Brazil? The surprisingly-popular option  strategy for crowdsourcing suggests piggybacking a control  question to the real question Q1: (Q2) What do you think the majority of other people will respond to this question? Many people (non-Brazilian and non-geography-nerds) will answer with Rio for both Q1 and Q2. But there will be some people that will answer with Brasilia for Q1 and yet Rio for Q2. The first set of people that replied with Rio to both questions did not know much about Brazil, and went with what they know as the most prominent city in Brazil. The second set of people not only knew of the correct answer, Brasilia, but they also anticipated that the majority of participants will go wrong by answering with Rio. Rio is the popular option for Q1, but Brasilia is the surprisingly popular option  because the respondents for Brasilia had anticipated that Rio would be t...

Last Wednesday, I attended one of the monthly meetings of the "Everything is Broken" meet up at Seattle. It turns out I selected a great meeting to attend, because both speakers, Charity Majors and Tammy Butow , were excellent. Here are some select quotes without context. Observability-driven development - Charity Majors Chaos engineering is testing code in production. "What if I told you: you could test both in and before production." Deploying code is not a binary switch; deploying code is a process of increasing your confidence in your code. "Microservices are hard!" as a caption for a figure comparing the LAMP stack 2005 versus the complexity of the Parse stack 2015. We are all distributed systems engineers and unknowns outnumber the knowns! Distributed systems have an infinite number of almost-impossible failures! Without observability you don't have chaos engineering, you have a chaos. Monitoring systems have not changed signi...

This post talks about why you should model your systems and exhaustively test these models/designs with the TLA+ framework. In the first part, I will discuss why modeling your designs is important and beneficial, and in the second part I will explain why TLA+ is a very suitable framework for modeling, especially for distributed and concurrent systems. Modeling is important If you have worked on a large software system, you know that they are prone to corner cases , failed assumptions , race conditions , and cascading faults . There are many corner cases because there are many parameters, and these do interfere in unanticipated ways with each other. The corner cases violate your seemingly reasonable implicit assumptions about the system components and environment, e.g.,"1-hop is faster than 2-hops", "0-hop is faster than 1-hop", and "processes work with the same rate". There are abundant race conditions because today (with the rise of SOA, cloud, and ...

Blockchains are supposed to solve the trust problem. But blockchains attack only the easy part of the trust problem, and avoid the hard part. The easy part is to store the transactions in a tamper-resistant database. The hard part is to attest to physical world actions and state. The blockchain is a database technology and it does not attempt to attest to physical world actions/state. It solves the problem of tamper-proofing the state after it is added to the database. It doesn't attempt to validate/test/certify if the state is correct as it is added to the database. If humans create the state, there is inherently a trust problem: Were the lettuce bad before it was loaded to the trucks, or are the truck conditions to blame? Did the farmer or the trucker lie? If sensors create the state, this is still a very hard problem, but not because the sensors may have been tampered with ---that is a relatively easy problem to solve in hardware. The problem is hard because of the corner-ca...

I picked this up from my local library, because the title was interesting. I wrote about this earlier. Once you get a B.S., you think "you know everything". Once you get an M.S., you realize "you know nothing". Once you get a Ph.D., you realize that "yes, you know nothing, but that is not a problem, because nobody knows anything!" This turned out to be a nice read. The author, Stuart Firestein, has a very interesting background. He was working at a theater, and started a biology undergraduate at 30, and got his PhD at 40. Here are some tidbits from the book. Leibniz. page 38 The 17th-century German philosopher and mathematician Gottfried Leibniz, one of the inventors of calculus, had a lifelong project to construct a "basic alphabet of human thoughts" that would allow one to take combinations of simple thoughts and form any complex idea, just as a limited number of words can be combined endlessly to form any sentence -- including sentence...

In my previous post , I had presented a TLA+ modeling of distributed data store that provides the consistent prefix property. In this post, I extend this model slightly to build bounded and strong consistency. In fact the strong consistency specification is achieved when we take the Delta on the bounded consistency as 1. The TLA+ (well, PlusCal to be more accurate) code for this model is available at https://www.dropbox.com/s/qvmhhgjf9iycaca/boundedstrongP.tla?dl=0 The system model As in the previous post, we assume there is a write region (with ID=1) and read regions (with IDs 2 through NumRegions). The write region performs the write and copies it to the read regions. There are FIFO communication channels between the write and read regions. WriteRegion == 1 ReadRegions == 2..NumRegions chan = [n \in 1..NumRegions |-> <<>>];  We use D to denote the Delta on the bounded staleness consistency. Bounded staleness ensures that read results are not too stale. ...