Fair and Expert Review

Satoshi Braque is a pseudonym. This story is fictitious as are all names, characters, and incidents portrayed in it. Identification with actual persons is not intended nor should it be inferred.  Any resemblances to living persons are coincidental.  The names of characters were chosen somewhat randomly from websites that have lists of names on them.  The author would have preferred names like Hermione, Hodor, and Haldir but wasn't sure if copyright issues would get in the way.

Fair and Expert Review
A Tale of Uncertainty and Peers

by Satoshi Braque

Celine woke up at four in the morning. She'd slept fitfully dreaming about the grant review meeting that she would manage in the morning.  The LED on her government-issued Blackberry was flashing red. Sighing, she typed in her passcode and saw a message from a reviewer saying that he'd tried to upload his critiques but his `password wasn't working'. Critiques were to have been submitted three days ago and she'd already been provided one unconvincing reason for why he would be late. She muttered darkly about the futility of working with unreliable academics. The same ones who complained about the poor quality of reviews when it came to their own grant applications but were always finding excuses to avoid reviewing and on the rare occasions that they could be coaxed into contributing to the system that they lived off, they turned in substandard critiques after the deadline for submission. "I should have listened when Bill said that he was unreliable but he's so well qualified!" she murmured.  She sent the reviewer a message asking him to email his critiques and scores to her so she could upload them for him. She wondered, for the thousandth time, why she did this for a living and why she wasn't looking for another job. 

Celine was a Scientific Review Officer, also known as an SRO. She worked at the Center for Scientific Review (CSR) at the National Institutes of Health (NIH). Each year, NIH distributed billions of dollars  in biomedical research grants. The granting system relied on peer review to identify the best applications and CSR was a large peer review organization  within NIH.Over 60,000 grant applications were submitted by scientists each year to CSR for evaluation by panels of expert reviewers. Each review panel was managed by an SRO who read the set of applications assigned to the panel, identified expertise needs, and tried to persuade the best qualified scientists to participate at a meeting. 

Celine was one of the better SROs at the CSR. She loved her job, her panels were well organized, and the reviewers in these panels were qualified and responsible. However, every meeting involved at least one flaky reviewer and she now knew who it was this time. She also had a sneaking suspicion that one of the other reviewers, a first timer, wouldn't turn out to be very solid. She wondered whether she needed a new panel Chair too. Joe was a distinguished mathematician, an algebraic topologist, much liked by the panel but he had been less than an effective Chair. He was just too nice and she secretly craved someone powerful. It was just too challenging to manage an inadequately incentivized  group of intelligent, opinionated volunteers by persuasion alone. She needed an enforcer of sorts, someone like that scary guy from the Midwest who could be stentorian in his displeasure. She also wondered if she wasn't just a little neurotic herself.

Celine gently eased herself out of bed to avoid waking Margaret who was snoring gently. Margaret, her partner, was a program officer at NIH. They'd met at a peer review meeting a year ago and bought a house in Rockville together a few months later. Discovering Margaret and growing into their stable and satisfying relationship had been fulfilling. She looked at Margaret sleeping peacefully, felt an awakening of physical interest and quickly extinguished it. Margaret was not an early morning person and Celine had to deal with a peer review meeting. So she switched her attention to making coffee, carefully dripping water at 193 Fahrenheit into freshly ground Aged Sumatra from Peet's. She brightened up a bit after coffee, Lipitor, and a shower. Then she dressed formally,  packed her laptop and meeting papers and got into her Volkswagen hatchback. It was early enough that she had an easy drive down the George Washington Parkway and into the Watergate Hotel. Mustafa, the friendly parking attendant recognized her immediately and said, "Hi Celine! I'll park your car over here so you can just drive off when you're done instead of waiting for the valet on duty to bring the car back from the garage." She thanked him gratefully and slipped him the $10 that ensured such privileges. Celine never tired of visiting the Watergate Hotel. It had an understated, swanky, politically decadent  Washington DC atmosphere that was quite irresistible. The reviewers loved Watergate meetings too. One of them even collected Watergate Hotel pens and gave them to his family members as gifts. 

She found the meeting room after going down a flight of stairs. It wasn't the one she wanted with a view of the Potomac but it was nice enough. Celine then drifted into her usual pre-meeting meditative trance as she waited for the grants technical assistant to show up and complete the arrangements. Abiola showed up twenty minutes later, brisk and organized. Meeting folders, name tents, and assorted peer review paraphernalia were in place in fifteen minutes. They waited in a companionable silence  for people to show up as they nibbled on sickly sweet pastries provided by the hotel. It was rumored that NIH would stop serving food at review meetings and she wondered how the reviewers would manage an eight hour meeting with only water. She shrugged indifferently. She wasn't an SRO with a parent complex about panel members and their nutritional needs. 

Joe showed up next beaming with pleasure and shook her hand warmly. He's stayed up late at night reading all the critiques and making notes of issues he wanted to raise. She felt quite guilty about about wanting to replace him, he really was a nice man but she wished he wouldn't avoid conflict during discussions. Two Program Officers showed up next. She liked both of them. They often provided helpful comments after meetings and they'd been very good about identifying potential reviewers from their own portfolios. She was grateful that Horace, another program officer, wasn't in evidence. He meant well too but he could be a bit of a handful at meetings.

Celine's cell phone rang. It was Margaret  calling to wish her luck. Celine made appropriate cooing sounds and terminated the call quickly. She was always tense, the morning of a meeting and much as she loved Margaret, she couldn't have a conversation with her right then. The room began to fill up some more and she wandered around shaking hands with new reviewers and chatting with the panel members she knew. So far so good. 

It was eight am so she clinked a fork against a glass to let everyone know that it was time. She started the meeting with her usual comments about federal peer review committees, rules, and regulations and ended up with, "Thus NIH and the scientific community rely on you to provide these applications a fair and expert review." She then handed over to Joe who bustled through his welcoming remarks and asked everyone to introduce themselves. Celine was mortified. She'd omitted the very first step in her protocol, the formal introductions. She wondered whether she'd every get over this and what people would think of her later. Breathing into a brown paper seemed like the right thing to do but she didn't have one handy and it would likely distract the panel. 

The Chief of her Review Group, James Green, tiptoed in a few minutes later and sat down unobtrusively in a corner. She shakily asked him to identify himself. He stood up briefly gave his name and sat down again. She felt an overwhelming rush of affection for him. She was always glad to have his stable presence at her meetings even if he was a bit smelly. He was an excellent supervisor, blunt when necessary but reserved his critical comments for afterwards, and was fiercely supportive of his staff. She grimaced at the thought of being supervised by some of the other Chiefs instead. Like most organizations, the Center for Scientific Review had its own complement of staff inclined to eccentricity and these excellent individuals were represented at all levels. The tension abated a bit and she giggled as she wondered what the SRO community must seem like to those outside it. "I really ought to look inwards before I start complaining about reviewers and program officers" she thought.

The meeting began as Joe announced the first application on this list. Six weeks ago, each application had been assigned by Celine to three reviewers who had prepared written critiques and preliminary scores. Today they would be invited to present oral assessments. The first reviewer had the additional responsibility of explaining the application to the panel in a few sentences. After these presentations, the Chair opened the application to general discussions for a few minutes, then requested scores from the assigned reviewers as the panel worked its way through administrative questions if any. The entire panel voted at this point, barring those who had the left the room because they had conflicts of interest with the application in question.  This procedure was repeated for every application that had been considered competitive enough to discuss. The applicant received an average scaled score a few days later based on scores voted and decisions about funding were subsequently made by NIH that that were driven by the panel score, available budget, and program priorities.

Review of the first application went well as did review of the next four. All the reviewers presented coherent arguments and the group discussion seemed to add value. Then disaster struck. The first reviewer for application number six  announced that he hadn't really read the application in great detail because he'd been busy with writing his own grant application but he'd been to graduate school with the investigator, was aware of his work and knew that he was a good scientist. Thus, his enthusiastic score was guided by this knowledge and the applicant's strong track record. Several reviewers looked disturbed. Celine attempted to appear impassive as she darted a beseeching glance at the second reviewer. Jacque stepped into the breach nobly and made some crisp comments about not letting her high esteem for the applicant's character influence her assessment of what was proposed in the application. The tension subsided a bit but the third reviewer, Dennis, dropped a mighty clanger shortly after by saying that he knew nothing about the field and had no idea why he was assigned to the application so he was just going to assign a score midway between first two reviewers. He followed up with some generic comments that suggested a cursory reading of the application at best. "Irresponsible fucker'' Celine thought to herself as she reminded the panel, while gazing demurely at Dennis, that all panel members had been instructed (and reminded) to share any expertise concerns within two weeks of receiving their assignments. This in order that the SRO could reassign applications if necessary or bring in additional expertise. She gently kicked Joe's foot under the table. He started and  commented apologetically that Celine had made a very relevant point. Somewhat overcome by his own leadership, he flushed scarlet and started shuffling papers loudly. Various panel members offered helpful assessments and Celine asked one of them to write an extra opinion that would become part of the official record. The panel moved uncertainly on to the next application which was reviewed competently and uneventfully. James Green got up, gave Celine a discreet thumbs up and left. 

Soon it was time for lunch. Celine declared an hour's break and everyone went off chattering. She hoped that they wouldn't conduct unsupervised peer review of the applications in the hotel restaurant. Celine grabbed Joe as he attempted to glide off and chatted briefly about not letting reviewers get away with personal testimonials to the exclusion of rigorous review of what was proposed in applications. He agreed that this was a problem. 

Horace showed up shortly after. Horace was an annoying Program Officer. He announced that he would like to speak to the panel about the importance of the Program Announcement that he was in charge of and why it was important to 'score boldly'. "No Horace, we've been through this before. No speeches but if you aren't disruptive, I'll invite you to join the Chair and me for a drink after the review.". He reluctantly acquiesced but smiled at Celine and said, "I know I'm difficult but I respect you and the job you do." Celine wasn't quite sure what to make of this but decided that it was probably a good thing. 

The afternoon session was usually the worst part of the day. Everyone was sleepy and the discussions usually dragged on. However, Joe seemed to have come into his own for the first time. He interrupted a lengthy self-referential monologue and deftly summarized a group discussion. Celine looked at him with approval and he nodded back with a boyish grin. Suddenly, there was a note of excitement in the air. Everyone seemed to wake up and things moved along. A particularly elegant discussion of an application for renewal of funding went by and they took a break for coffee. A reviewer came by and thanked Celine for inviting her to the meeting. "I've learnt so much at this meeting Celine and Joe is a fantastic chair!". Celine wondered how long her enthusiasm would last. 

The panel resumed its discussions and was soon bogged down in a discussion on potential versus track record in the case of a newly tenured Associate Professor at a lower-ranked university. Graciela, a rather earnest and shy panel member from Iowa tentatively raised her hand. Celine hissed in Joe's ear, "Give her a chance". Joe gestured in her direction and Graciela spoke eloquently about the problem being studied, explained that it had been formulated thirty years ago and that the applicant had not referenced any of the extensive prior art. In a polite and anxious tone, she also explained why the application was uncertain in its objectives and significantly lacking in novelty. The panel looked at her with new respect. When she was done, two reviewers thanked her for the excellent review and the entire panel clapped briefly. The atmosphere was almost electric. Every  panel member wanted to perform as well as Graciela after that. Celine made a note to herself to think about whether Graciela could be Chair one day.

The panel was similarly uncertain about the next application and curiously enough Graciela spoke up again. This time to explain why it was such a good application. However, she wasn't able to convince the naysayers led by Roger who were unable to forgive the lack of enough preliminary evidence to mitigate riskiness. The panel ended up with a divided opinion and Joe did a masterly job of summarizing the discussion. 

Two more hours and it was time to wrap up. While Abiola collected the score sheets, Celine made the usual thank you speeches and asked for a round of applause for Joe. Joe, in turn, gallantly pointed at Celine and Abiola then asked the panel to give itself a round of applause. The meeting was over. The panelists began to drift out in twos and threes as those who were staying overnight discussed where to have dinner. Celine had rather austere views about socializing with panel members. Despite the fact that she found some of the reviewers immensely likable, she found it easiest to play SRO well by maintaining an arm's length distance. 

Jacque came by and whispered to Celine, "Over the last year, you've herded us into a real review panel. Congratulations!"  Celine thanked her and turned to the Chair. "Joe, you were stellar today.", she said. He blushed and said, "You're a super SRO Celine. I can't buy you presents because you're a fed but I do remember you asking about Turing machines. So if you have a few minutes, I'd like to explain the Halting Theorem to you over a glass of wine. I hope you'll let me buy it" She agreed on the condition that she got to buy the second round. Lynn, another panel member overheard the conversation and asked if she could join. Horace had already found a table in the hotel bar and was sitting appreciatively in front of a tall glass. They joined Horace and after ordering drinks, Joe sketched out the proof and explained it with sincerity and passion. Then he inscribed the sheet of paper on which he'd scribbled. "To Celine- the best SRO at NIH! '' Lynn and Horace signed  as witnesses. Although receiving compliments made her uncomfortable, it was a magical moment for Celine even as she remarked that there were better SROs at CSR.

Recognizing her embarrassment, Lynn proposed a broader toast to NIH, peer review, program management and good science. Everyone raised their glasses and Celine felt a warm glow. Being an SRO was all right after all. Horace was a good sort too. Lynn was an excellent reviewer and Joe...he was a good Chair, she'd been far too hard on him. He didn't have the casual authority and silky elegance of speech that Ferenc the Swiss computer scientist had but he was better and better each time. Besides, the panel respected him and loved his courteous behavior. She was going to stick with him. She felt like getting up and hugging each one in turn but decided that the overpriced Mark West Pinot Noir was thinking for her. She also realized that she was tired, her back hurt and her ankles were swollen. So she rose, dropped four ten dollar bills on the table, shook hands with all present, and headed out to her car for the drive back to Silver Spring. 

On the way back she thought some more about why she liked her job. Above all, it was because the cause was a good one. The US taxpayer and science benefited from this peer review process, noisy and error-prone as it was. There were good reviewers on every panel who tried their best to do things well and there were always a few truly exceptional reviewers whose scholarly discussions elevated the performance of the entire panel. It was worth being a part of this public service operation.

She noticed that her Blackberry was flashing again. There were about sixty unread messages in her inbox. The first one was from James Green complimenting her on how she handled the meeting and suggesting that she re-educate Dennis or not invite him back. She agreed and also decided that she was going to have application number six reviewed again by a different panel to ensure that it received a decent evaluation. The next few were from various applicants wanting to know if she could let them know how they'd done. 

The SRO Celine
With reviewers would convene
At risk of being boring
She insisted on preliminary scoring

However, Celine
Could also be mean
In search of bureaucratic perfection
Her voice could take a sharp inflection