Moneyball, but for academia

"Moneyball: The Art of Winning an Unfair Game" is a book by Michael Lewis, published in 2003, about the Oakland Athletics baseball team and its general manager Billy Beane. Its focus is the team's analytical, evidence-based, sabermetric approach to assembling a competitive baseball team despite Oakland's small budget. 

I had zero knowledge and interest about baseball, but the book was very engaging, and I could stop reading. Michael Lewis is one of my favorite writers. I had read Flash Boys, Big Short, and Next by him, and all of them were very good.

Fundamentally, Moneyball is about making radical but rational choices to the face of flawed ways of the tradition. Where there is a tradition-ridden unoptimized market, there is potential for disruption: if you are brave enough to do things differently, you can benefit a lot from doing so.  Initially only a few people are daring enough to see this and ignore the tradition and status-quo to start doing things differently. The gravitational pull of tradition and status-quo is very strong especially at the institutions-level. It seems that at the individual level, it is easier to escape from this pull, but at the institutions level this pull is almost inescapable because it is safer to do things "the way it is always done", and it is very hard to shift the entire practices of an institution as institutions are very risk-averse.

Since the faculty recruitment season is in motion, I started thinking about how these lessons would apply to faculty search.

Disclaimer. These are my views, not necessarily that of my institution/department. It should be obvious that I speak only for myself, and I speak subjectively and with generalizations which do not always hold.

Status quo

Academia is a nonlinear game. The quality of the research produced by faculty has nonlinear rewards/returns. Having 100 research faculty does not make a department automatically 4 times better than a department with 25 faculty. It is the quality and not the quantity of research/publications that matter.

In other words, horizontal scalability does not work, and recruiting high-performing research faculty is important. If you have unlimited resources/appeal, you would want to get the stars. In the absence of that, you need to plan carefully about how you can  discover stars before they become stars and/or when they are undervalued?

Hiring assistant professors fresh out of graduate school is one way of finding undervalued talent. These freshly minted PhDs did not have an opportunity to develop their research agendas fully and prove themselves. They will also be incentivized to do their best work to secure tenure at your department. I suppose this is why hiring fresh PhDs as assistant professors is the most common way of faculty recruiting in the academia.

Among the senior hires, a candidate with great publications and funding but coming from a lower ranked department is, by definition, undervalued. Most of the senior hires is of that nature. Ah... The R word... rankings! This brings us to the biggest inefficiency/flaw in the current faculty recruitment traditions.

Fallacies of current hiring practices

Ranking fallacy. 
There is a big premium put on whether the candidates come from a top ranked school. The rule of thumb many departments use is that they only seriously consider candidates that come from schools at least 20 places higher than their own rankings.

But this is a crazy obsession. Humans have similar brains (we are all dumb). It is unwise to think that candidates coming from top schools are much smarter. It is also not necessarily the case that they receive a much better education. This is the age of Internet, information/education is accessible to those seeking for it.

However, the candidates coming from top schools have more institutional support. This is a nontrivial effect. It takes a village to raise a child. I ran an informal poll on Twitter with 50 votes.

Aside from the branding, what is the biggest advantage of getting a PhD from a higher ranked CSE department?

If you have other answer, please reply here.
(4th option: access to better internship & networking opportunities)

— Murat Demirbas (@muratdemirbas) December 6, 2019

The students at high rank departments can easily find more faculty and students to collaborate with. I also think the role of regression to the mean is a big factor. Students in top ranked departments push harder for achieving more.

In any case, the objective criteria is simple. When the search committee evaluates the faculty applications, they should consider the quality and quantity of the publications and the coherency of the research statement. If a candidate from a lower ranked department has comparable publication record to another one from a top ranked department, isn't it better to hire the former as that is a more impressive accomplishment? We should check for skills and potential rather than lineage and schools.

Hype fallacy.
Another common flaw with the current hiring practices is that it is very much coupled to the hype cycles. There is a big premium put on whether the applicant work on a currently hot domain and not enough emphasis given to the toolkit of the applicant and the potential application areas of this toolkit. The domains are ephemeral, but the toolkits are long lived. If a candidate has a good toolset, it is easy for her to switch domains, and work on many domains.

For example, instead of trying to hire machine learning people who are in high demand by all departments, it would be a better strategy to hire people who are undervalued. Maybe hire theory people, programming languages people, or people who have formal methods experience, but with practical twists. After the big players take their pick on machine learning people, and when there is an excess of high quality machine learning candidates on the market, hire from machine learning.

Charisma fallacy.
I think the departments care too much about how well the candidate's talk went. That is not a very relevant skill for academic success. Organization of the presentation, yes. But the delivery of the talk, and looking good doing it, does not have much to do with the quality of the researcher. Many high caliber researchers are not very good on thinking on their feet, and nor are they required to be. Yet, unfortunately some candidates get penalized as they do not come across as high energy or assertive enough, and their presentation is not lively. It is better not to make this mistake, and recruit these undervalued researchers.

This point is also relevant to increasing diversity. Increasing the diversity of the departments are very important and genuinely beneficial to the departments. I am happy to see that this is now strongly encouraged with new systems put in place in many universities. The faculty should keep in check the hidden biases they may have during the candidate visit and evaluation.

What are optimal strategies for recruitment?

It is somewhat easy to account for the above fallacies. But devising optimal strategies for recruitment is very hard, and is beyond my pay grade. From what I can see, there aren't much work on this out there either. Are there any data-oriented study on faculty recruiting with controls? Are there any departments that play this moneyball game better?

I don't have answers, I only have questions.

Is it better to recruit a candidate that is from a weak area for the department, or a candidate that aligns with a strong area for the department? Both has some advantages. It is not very clear, which would be better when.

Being a successful faculty require different skillset from being a successful PhD student, such as being a good mentor, team builder, novel thinker, proposal writer, even organizational skills. What are ways to evaluate these? For some of these we rely on the recommendation letters, but this is not objective?

What are best ways to objectively measure the creativity and vision of the candidate?

How do we measure how good a supervisor and collaborator the candidate could be?