I am heading to Ottawa next week for NSERC Competition Week to judge NSERC Discovery grants in Mathematics & Statistics. Colleagues have asked me about how this works. I am reblogging an article I wrote from last year, which summarizes how the week unfolds. I’ve updated a few points, but the structure and function of the Evaluation Group in Mathematics and Statistics (1508) remains the same as last year (although a few members have changed).

Enjoy!

***

As a follow up to my earlier blog about the Competition Week at NSERC, in the present post I share insights into the judging process for Discovery Grants. NSERC received hundreds of applications evaluated by the Mathematics and Statistics Evaluation Group (EG) 1508 this year. The role of the 31 mathematicians and statisticians on the EG was to judge these and assign them scores as described in the merit indicator grid: Exceptional, Outstanding,  down through to Moderate and Insufficient.

Fifteen minutes

As Andy Warhol famously said, everyone will be famous for fifteen minutes. That is how long each application actually gets with the EG during Competition Week (longer if there is a revisit). The time is short, and that reflects the volume of applications the EG has to judge in a  window of a few days. NSERC says that the volunteer work of all the STEM academics serving on the EGs is equal to about $30 million dollars per year. Kudos to the members of the EGs, who work hard and often get little or no compensation from their institutions.

Each application is assigned a 1st reviewer, 2nd reviewer, and also 1st, 2nd, and 3rd readers. The 1st reviewer goes into the most detail, and the 2nd reviewer can add to the discussion if something was missed or needs greater emphasis. The readers also may add new points if needed. The 1st and 2nd reader are often in the research area of the application, but this can vary a bit depending on the EG’s constitution. For example, I often am 1st reviewer for applications focusing on discrete mathematics, but I don’t obviously have expertise in every subfield of the topic. We do our best and rely on the external expert referee comments.

What happens in those fifteen minutes

I felt a kind of nervous excitement during the judging of my first application. You want to be fair, cover all the major points in your notes, and be open to wisdom from your colleagues. We did mock reviews (with actual past applications) before and at the beginning of Competition week which I found helpful.

Here is how the fifteen minute judging is structured.

1. The NSERC program officer reads the name of the applicant and their institution.
2. The chair calls for the merit scores in each of the three categories. Some EGs other than Mathematics & Statistics skip this step, and the members just state their scores during their assessments.
3. The 1st reviewer gets three to four minutes to discuss the application. They usually summarize comments for each of the categories: 1) Excellence of the Applicant, 2) Merit of the Proposal, and 3) Training of Highly Qualified Personnel. When I was 1st reviewer, I would finish by highlighting the external referee comments.
4. The 2nd reviewer then can add anything relevant. Often but not always this reinforces what the 1st reviewer said. This get two minutes.
5. The three remaining readers then chime in, each getting one minute.
6. The chair then summarizes the scores, noting any category that has a spread of scores. The chair doesn’t try to reach consensus but helps guide the discussion. Say Merit of the Proposal has four votes of Outstanding and one Very Good, then the chair may ask the member who voted Very Good to describe their rationale. Scores can change at this point, and there is often friendly but engaged discussion when there isn’t agreement on a given score.
7. The chair calls for everyone to enter their scores on secure PCs set up around the room. Scores are averaged out, and then the PO reads the three scores to the group.
8. The chair then calls for discussion of a possible Discovery Accelerator Supplement nomination.

In the Mathematics and Statistics EG, the cost of research is usually deemed Normal, so that part is not discussed (or at least it wasn’t in any application I judged).

Preparation, revisits, and messages to applicants

The above nine steps may seem like much to cover in a short 15 minutes. It is!

Deliberations are fast-paced, and you must come prepared. There is much pre-processing done by the EG members before those precious few 15 minutes. For each of my 30 or so assigned applications, I spent considerable time in the weeks and months before Competition Week reading the proposal, the CCV, research contributions, and referee reports.

Any of the five readers can call for a revisit on a file. This is triggered if anyone thinks something was missed or something needs longer discussion.

If any of the three scores are Moderate or Insufficient, then the 1st reviewer needs to write a Message to the Applicant, which is reviewed by the 2nd reviewer and the chair. For instance, Early Career Researchers (who are within two years of the start date of an NSERC eligible position, and who have no academic or non-academic independent research experience prior to the two-year window) usually get a default Moderate score in the HQP category. That is natural as they have no or limited exposure to training students at the beginning of their career.

The onus is on the applicant

We are only allowed to judge an application based on what is in the file; not on what we have heard anecdotally or from the applicant’s website. That is why submitting a solid and well-written application is so vitally important.

Everything must be crystal clear in the proposal. For example, if you have co-authored publications, then explain your role in them. Describe your choice of venue for publication, the role of HQP in your work, and clearly list your long and short-term goals and objectives. NSERC publishes guides on how to write your application, and I would highly recommend that every applicant reads these and carefully follows the instructions.

One very important thing: ask your colleagues to read your proposal! Make sure it is free of typos and readable. The more eyes and the more feedback, the better in my opinion.

Referees: are you out there?

We may have many referee reports for a given application, but sometimes we only receive a small number. Early in the process, I was asked to suggest seven referees per application (up to four taken from the list provided by the applicant), but apparently it is rare for all seven to respond. When I last enewed my grant, I received six referee reports which is unusually high. One EG member said this year, however, that they were asked to invite as many as twelve reviewers but ended up with only one report.

If you are asked to referee an NSERC DG application, then please agree to do it and complete it on time. It is vitally important to the EG to see these comments and can sway a score either way. The applicants are counting on you for your expert opinions.

Anthony Bonato