Humans of Computer Systems: Obdurodon

Programming

How did you learn to program?

Some college, some self taught, mostly on the job

Tell us about the most interesting/significant piece of code you wrote.

Cluster manager (leader election + fault detection/response) for HACMP/6000 in 1983. Primitive by today's standards, but cutting edge for its time. Used in production at a significant percentage of banks, retail stores, etc. for about a decade.

Who did you learn most from about computer systems?

Various senior/principal engineers at Encore when I was still junior in ~1990. It was one of the early UNIX SMP systems. Learned much of what I know about kernels, TCP/IP, multi-processor synchronization, memory hierarchies, etc.

Who is the greatest programmer you met, and what is impressive about them?

TBH some of the best programmers I've met worked primarily in Verilog, designing chips and tools around them - e.g. Wilson Snyder and others at SiCortex/Veripool. Huge volume, but also rigorous testing leading to *very* low defect rates even for that domain.

What do you believe are the most important skills to be successful in your field?

Communication/collaboration skills, with the caveat that one must be willing to use them for good. The skills themselves can also be used for personal glory or to form/maintain cliques, which is not good. Drawing together people with complementary skills/interests (especially those who are under-recognized or under-utilized where they are) and keeping them aligned toward a clear goal can produce a whole *much* greater than the sum of its parts.

What quality or ability do you value most in a computer systems person?

Understanding of what it takes to ensure that a particular type of system (e.g. non-DB storage for me) is correct and maintainable/operable in the long term. Understanding how to design a system (especially a distributed one) for these properties, where/how to apply various forms of validation or testing, etc. can make the difference between a system that serves both developers and users vs. one that burns them out.

Personal

Which of your work/code/accomplishments are you most proud of?

Teaching some hundreds of others the value/importance of making code correct before it ships, and techniques to that end. The idea that I might have helped improve systems that I'll never work on myself pleases me.

What comes to you easy that others find hard? What are your superpowers?

I seem to be better than most at "visualizing" the state of a system with many concurrently moving parts, and anticipating what states it can get into. This extends all the way from debugging (finding the obscure sequence of events leading to an observed failure state) to developing new algorithms for specific purposes.

What was a blessing in disguise for you? What seemed like a failure at the time but led to something better later for you?

Being laid off from my first programming job in Michigan. It was kind of traumatic, but it led directly to me moving to where there were *far* more kinds of programming jobs available. As a direct result of the physical move, I also moved from application-level programming to system-level programming - particularly networking and distributed systems.

What do you feel most grateful for?

All the people who gave me a chance, from Karl Zinn at UMich's CRLT when I was still a scruffy student to the people who hired or promoted me, who invited me to speak at conferences, who persuaded me to lead entire projects. In many cases, there was little reason to believe I'd be good at The New Thing.

What made you most happy in the last year?

Retiring. :)

Work

What was your biggest mess up? What was the aftermath?

It was kind of a slow-motion mess up, but being product architect at Revivio (continuous data protection) was kind of a disaster. It was my job to create alignment between three other teams that were pulling in different directions, so that we could ship a viable product faster. I failed, in large part because I focused too much on the hands-on details and not enough on the big picture. There were reasons, there were certainly others who contributed more to the outcome, but ultimately the company ceased to exist.

What was your most interesting/surprising or disappointing interaction at work?

On at least two occasions, I had to work with engineers who literally just refused to do the things they were supposed to do, preferring to work on their own alternatives instead. One eventually got fired for it, which was very educational for me. The other, even more surprisingly, never did.

What do you like most about your job/profession?

The intellectual/technical challenges, of solving big sets of very hard problems, making everything fit together, and (eventually) producing something of value to users.

What do you dislike most about your job/profession?

The cargo-culting and factionalism, often driven by the dominance of Big Money trying to create/sustain industry behemoths instead of healthy ecosystems.

What would be the single change that would improve your work environment most?

Adoption of remote-friendly communication methods and workflows. "All in one room at the same time" is just not going to be the primary model going forward, and using video etc. to replicate that doesn't work. It's the shared-memory SMP model of work, and it doesn't scale under even moderate latency. Truly scalable teams have to embrace asynchrony and explicit communication, but few do.

Technical

What do you think are the hardest questions in your field?

How to balance needs for unity/efficiency vs. isolation/diversity. At the technical level, this appears e.g. as microservices vs. monoliths, and I've been around long enough to see both overdone. At the organizational level it's more about team structures and workflows.

What are you most disappointed about the state-of-the-art in your field?

As I said on Twitter, the focus on features/performance/profit over correctness/sustainability, *especially* among the organizations that are best placed to shift that direction.

What are the topics that you wish received more attention? What do you think is a promising future direction in your field?

Somewhere there's an intersection between correctness (formal methods, testing etc.) and composability that needs to be explored to create larger but still integrated (and efficient) systems. Kind of like the old "software components" idea, kind of serial specialization like the FPGA folks keep claiming to enable at the single-machine level but at a whole-system level instead.

What is your favorite computer systems paper? Why?

Probably the Amazon Dynamo paper. There's a lot that it left unaddressed, and of course it describes a proprietary system, but it drew together a lot of previously disparate threads and set off an explosion of activity interesting to me. For a narrower focus, Karger et al on consistent hashing is just very satisfying intellectually and has found more uses than anyone could have predicted.

What are the most interesting blogs/twitter accounts you follow?

Yours, of course. ;) Also Vijay Chidambaram, Eric Brewer, Brendan Gregg, Jonathan Ellis.

Story

Tell us your story.

https://obdurodon.silvrback.com/tags/memoirs

If you enjoy reading this series, consider taking 10 minutes and submitting a response. All questions are optional. You can skip most, and tell a lot more on other questions you choose.