:::: MENU ::::

On reference letters (for undergraduates)

This advice post is geared at undergrads seeking reference lettes for grad school, particularly in economics. Graduate students have different considerations, and I think the advice would likely be less applicable to them or already known.

The main thrust of my advice is: consider the other side of the market.

When you apply to a program and you list a reference, that reference will typically be asked to rate you relative to your peers on a variety of dimensions. This is true even when applications say they will solicit reference “letters” – yes, they ask for letters, but they also generally ask some additional questions that are used to weed out candidates.

For example, a typical program might ask recommenders to rate you compared to your peers on “work ethic” or “integrity” or “teamwork”. The options might be something like “top 1%”, “top 10%”, “top 20%”, “top 50%”, “below average” (the precise options vary from program to program). Almost all programs require answering these kinds of questions.

Now suppose you are a student who ended up in the bottom half of the class. You skipped most lectures, so as a prof I would be hard-pressed to know who you are or have anything distinctive to say about you. If all I know about you is how you did on your assignments, and you did poorly, I’ll be forced to tick the “below average” box for most of the questions. That doesn’t help you. Since it doesn’t help you, my policy is to not write reference letters for students who did below the average in my class unless there are extremely unusual mitigating circumstances. Providing reference letters is a repeated game: I don’t want to say things that are false, because then my future references would carry less weight. If reference letter readers can clearly see from your transcript that you did poorly in my class, but I am to rate you highly, I had better have a very good justification to provide for why I am doing so.

Now suppose you are a student who did well overall on all exams and assignments but didn’t actively participate in class. Because you did well, you are compared favourably to your peers, and your application does not get immediately screened out. Depending on the program you are aiming for, that might be enough. But your letter will be pretty uninformative and boring compared to what it could be. All I would be able to say is something like “Student X was in my class. Here is a description of the class. Student X’s highest marks were on the exams/final paper/etc.” It is almost a “mad libs” style, fill-in-the-blank reference. Admissions committee memebers could get most of the information in it from reading your transcript – the letter does not add much value. It’s good enough for many programs (depending on what kinds of programs you are targeting) but maybe not enough for the very best.

A great letter gives readers information they don’t otherwise know about you. If you want such a letter, I need to know more about you. How can I know more about you? Maybe you actively participate in class or office hours. Then I can reference your curiosity or your engagement and say that you always had good questions. Maybe you write a really great term paper. Then I can describe it and why it was great. Maybe you are not a student in my class but someone who provided research assistance. Then I have far more opportunity to observe and comment on your abilities.

I hope this gives you some idea. It’s fun to write letters for great students, but sometimes I need you to help me help you.

Finally, a good reference takes time, and we are all very busy. Please contact your references no later than 3 weeks before a deadline. Ideally, after they have agreed, send them invitations to submit references for your programs all at once so they can do all your references in one batch.

When requesting a reference, it helps to also say something about which types of programs you are planning to apply to. Ideally, attach a list of programs and their deadlines. If you just ask me for a reference without saying where you are planning to apply, in most cases my first question will be to ask you to provide the list of programs and I won’t give you a definite yes or no response until then. (It makes sense: you recommend people for particular things, not to ANY thing in the universe. I don’t want to commit to recommending you for something unrealistic.) So you may as well provide the list or some sense of where you are thinking of applying upfront, and this will also help ensure it gets in by the deadline.

Summing up, my advice is:

  • Knowing what you now know about how letters are evaluated, ask for references from professors whose classes you did well in. My personal policy is to not provide reference letters for undergrads who do below-average in my class unless there are truly extenuating circumstances such that I can still give them a high ranking, but other people may set different bars.
  • Great letters need something more. You don’t need all your letters to go on and on at length, but the more competitive the program, the more you want your letters to contain information beyond what the admissions committee can tell from the rest of your application materials.
  • If you want to make things easy for your profs, request reference letters at least 3 weeks in advance. Specify which programs you are interested in applying to and what their deadlines are. Doing this can also improve your applications: you might get good advice about other places or programs you might consider applying to!


Pre-doc hiring

I am looking to hire 1-2 pre-doctoral fellows at the University of Toronto in applied microeconomics. The deadline is March 31, 2025.

Here are descriptions of the different types of work available:

1. Work on the Social Science Prediction Platform

Prof. Stefano DellaVigna (UC Berkeley) and I are seeking a pre-doctoral research fellow to assist with the Social Science Prediction Platform (SSPP) and be an integral part of the team. The Social Science Prediction Platform is an online platform that enables researchers to collect ex ante forecasts of what their studies will find. These forecasts can be useful in a number of applications, some of which are summarized in this Science Policy Forum piece.

Since the SSPP started, we have seen an enormous growth of projects posted on the platform, and we are beginning to analyze data from the first 100 projects.

2. Work on guaranteed income experiments

I have been evaluating several different guaranteed income experiments with Alex Bartik (UIUC), David Broockman (UC Berkeley), Patrick Krause (OpenResearch), Sarah Miller (Michigan), and Elizabeth Rhodes (OpenResearch). Some results have been written up, but there is still more to do, including analysis of children’s outcomes, crime, intrahousehold outcomes, and potential analysis of heterogeneity across sites. Some of this work may include building and estimating models.

This role would also include the chance to work closely with OpenResearch, a nonprofit research organization with a startup mentality, and would provide the opportunity to learn about cutting-edge best practices in research.

3. More open-ended work

There is the opportunity for more open-ended work in evidence-based decision-making and topics in AI. Past pre-docs have worked on existing projects while developing their own projects through active discussion and mentorship.

Pre-docs may work on one or more of the three topics above, depending on fit, availability, and interests.

Eligibility

Applicants must, at minimum, have:

  • A bachelor’s degree (or be graduating this year);
  • Experience in R or Stata;
  • Work authorization in Canada, whether by citizenship or an open work permit. This is strictly required. In practice, citizens of several countries can often obtain an open work permit. In the pre-interview application screener, you will see some questions designed to help you figure out if you might be eligible (though you may need to do further investigation on your own).

The ideal candidate would have:

  • A strong quantitative background and potentially a master’s degree;
  • Proficiency with more than one programming language;
  • Familiarity with ML;
  • Previous research experience, such as through past research assistantships or an independent research project;
  • An interest in pursuing a PhD in Economics or a related field.

To apply:

To apply, please fill out a pre-interview screener here, including uploading a transcript and CV. Only shortlisted applicants will be contacted for an interview.


OpenResearch Unconditional income Study

At long last, the first working papers have been released for a project I’ve been working on since 2016: an evaluation of a program that provided $1000/month unconditionally for 3 years to 1000 low-income individuals, while another 2000 received $50/month as the control group.

The papers:

In time, I’ll say a lot more about what we found, but for now I just want to recognize what an immense team effort it has been. It’s been an incredible experience working with Alex Bartik, David Broockman, Patrick Krause, Sarah Miller, and Elizabeth Rhodes. But also, so, so many people have contributed to this project in one way or another.

It also wouldn’t be an exaggeration to say that OpenResearch really enabled a different kind of project than is typical in economics. Massive amounts of thought went into this, from a lengthy piloting phase to Elizabeth Rhodes tracking down participants and designing a font that looked like her handwriting so she could add personalized notes to postcards sent to keep people engaged. All the details were thought of, all the bases covered, all the grants were applied to, and all the outcomes collected. Partners helped to pass a law in Illinois to ensure participants would not lose important benefits. A custom mobile app was developed. We were even able to collect biomarkers at endline to investigate potential health effects, a massive logistical effort.

It’s been a real pleasure working with this team. Some first papers are linked above, but expect more coming soon.


Pages:1234567...16