How I read a research paper

I can't tell you how you should read a research paper. Probably a different and customized style would work best for you.  But I can tell you how I read a research paper. And if you think some of the things below make sense, you can try them for yourself and see if they work for you as well.

So, how do I read a paper? The first answer that pops to my mind is "I fight with it". I strive to completely understand and "Grok" the paper by arguing with it. I resist to believe the claims in the paper at face value, I try to poke holes at it. I believe that the paper has to earn my trust, and I only accept the parts that I am obliged to accept.

My algorithm for reading a paper

0. Print the paper
1. Ask questions and argue with the paper
2. Write a review
3. Fight with the paper
4. Sleep on it and come back to it

Print the paper

I like to physically touch the paper and handwrite and doodle on the paper. I have highlighter and different color pens ready with me, when I am reading a paper.

Ask questions

I am a slow and deliberate reader. I ask a lot of questions while reading the paper. Why did the paper define it this way? Was this the only way to define it? Was this the best way to define it? Do I believe it? Why should I believe it? I continuously have some questions in my mind going through the paper. If nothing else, I try to guess where the next paragraph will be (or should be) leading the paper. This is called critical reading.

I get very detail-oriented when reading the paper. I highlight, I underline, I cross over sentences. I mark important paragraphs with stars. I use the margins to take notes about my arguments and about some connections that occur to me. Obviously, there will be several places I won't understand in my first reading. I mark these with WDYM (what do you mean). I hypothesize (make up guesses) about things I don't understand, and revise my hypothesis as I read further in to the paper.

I am a bit ashamed of this, but I often write provocative things on the margin, like "This claim is stupid", "This is just bullshit". It is silly, but I do this to keep myself engaged with the paper and be emotionally involved as well. This way, I never fall asleep reading a paper.

Write a review

"Writing is nature's way of telling you how sloppy your thinking is."

I write a short review of the paper to test how much I understand of it. Since I already mark enough text from the paper and write enough notes in the margins while reading the paper critically, it becomes easy to start writing my personalized review of the paper using these.

I am often surprised by how little I understood the paper in my first reading. When writing the review, since I try to explain things myself in my own way, I often realize that I didn't really understand the paper.

To elaborate this point, I will borrow from my advice to undergraduate students:
There are different levels of knowing/understanding. The first level is "knowing by word". With passive learning (like passively reading), you get the most limited form of understanding/knowing. It is ephemeral, and you are not able to reconstruct that on your own. The second level is "knowing by first hand experience". At this level, you are able to understand/witness and if necessary rediscover the knowledge on your own. Finally, the third level is "knowing by heart". At this level, you internalized the knowledge so that it becomes part of your nature. You grok it.

Fight with the paper

Now that I somewhat understand the paper, it is time for me to think about it more in depth to understand it better. How would I do it? Could I find a better/simpler solution to this problem? What would be the major disadvantages of the proposed solution? Or, even, considering the problem the paper addressed, was this the right question to ask in the first place?

I find that this exercise also has benefits for my own writing/research. Trying to poke holes in others' papers helps me become a better writer and write stronger/better-defended papers. (If you are a graduate student, this will help you write a well-defended strong thesis, and you can get by fighting a small snake in the snake fight portion of your thesis defense.)

It is now time for me to read the paper a second time. This time I try to test/verify my major objections about the paper and also try to understand the places in the paper I had marked as WDYM (or sometimes as WDYFM).

Sleep on it and come back to it

At this point, I might have been overly negative and hard on the paper, and now is the time to empathize with the paper and put it back in context. It is important not to confuse critical reading with being hypercritical about a paper and dismissing the contributions made by the paper.

So I sleep on it, I take time off and go on with other things (life?) for a day or so. But during this time, my mind comes back unconsciously to the paper and reconsiders my judgments about it. And often, as a result I come to respect the paper and appreciate it more. Even though I fought with the paper and argued with it fiercely, now I try to respect and truly learn from the paper.

Maybe this quote from "Ender's game" explains this best. Ender says "In the moment when I truly understand my enemy, understand him well enough to defeat him, then in that very moment I also love him."

Questions-Answers

How do other researcher read papers?
Unfortunately, I don't know too much about how other researchers read. It would be nice to learn tricks and approaches used by others. But I guess these things are not discussed much explicitly.

I think my approach to paper reading was shaped strongly by my PhD advisor Anish Arora. He would also write on the paper, underline carefully, and argue in the margins. He was also extremely meticulous about paper reviewing for conference committees. He would insist that we understand every aspect of the paper, and sometimes understand its contributions better than the authors that wrote the paper :-)

I also had a chance to observe my postdoc advisor Nancy Lynch reading papers. She is a very detail-oriented person as well. She was able to find even the tiniest smallest mistakes in the papers with ease. She once told me that her mind worked like a debugger when reading a paper, and these bugs jumped at her. The way she worked with students was that she would dedicate herself solely on a student/paper for the duration of an entire week. That week, she would avoid thinking or listening other works/students. This is because, she wanted to immerse and keep every parameter about the paper she is working on in her mind, and grok it.

After reading this post, Ted Herman shared some of his useful tricks. While reading, he would think about how he would rewrite the paper in a way that would have cleared up the misunderstandings. He would also ask these questions of the paper: "Why is this interesting? What are the surprises? Should we even be surprised?"

There is also this short CCR paper about how to read a paper.

How much time do I spend reading a paper this way?
Many many hours. Of course if the subject is easy or I know the domain well, I can read a paper in a more relaxed manner. But I don't remember getting a lot of benefit from such readings. I get the best return on investment on the papers I had to struggle with and truly grok.

Isn't this wasteful for bad papers?
Yes, it is. I do this for reading papers that are important for my research and/or reading good quality papers that appeared at good venues. Or for papers I am stuck with reviewing for a conference or journal.

Related earlier posts:
My advice to graduate students
My advice to undergraduate students
Tell me about your thought-process, not just your results