The Future of Research Symposium:
Metrics: The Folly of Hoping for X whilst Rewarding Y
Posted by Gary McDowell, on 29 September 2014
This is the third of four posts relating to the Future of Research symposium which was announced in a previous blog post. Each of these posts will discuss a topic that is the focus of a workshop at the Symposium. Even if you can’t attend, please tweet @FORsymp with suggestions, or follow us to respond to our questions about what YOU, trainee scientists, think is important. The hashtag for this post on the Workforce workshop is: #FORmetrics
Rewards and incentives
What is “good science”? What is it that we want science to accomplish?
Does our current system reward “good science”, our shared goal, or are there perverse incentives that do the opposite? What are the problems and how could we reward scientists and institutions to produce the behaviors (such as collaboration, openness and honesty) that we believe support “good science”?
We all have ideas about what science is supposed to do/achieve, but these outcomes happen at a broad, communal, systems level. Meanwhile, rewards/incentives operate on individual scientists. Often, there is a disconnect between those two.
Many of the current incentives work on individuals or small groups of people (grants, jobs, fame, etc.), while many of the things we want science to do (cure disease, figure out how the world works, etc.) need to happen at a broader systems level. This may contribute to the disconnect between incentives and desired outcomes.
Intrinsic motivators may include curiosity, desire for freedom, etc.; the reasons people got into science in the first place. The extrinsic motivators are things like grant money, lab space, prestige, etc. How can we reinforce or encourage people’s own intrinsic motivations (at least the ones that lead to good science)?
The objective of the Metrics workshop is to ask: How can we fix the current system of incentives so that we reward scientists and institutions for the behaviors that we believe support “good science”?
Key areas where science “breaks”
Reproducibility and negative data
Part of the scientific endeavour is to provide checks and balances and reproduce results, or highlight when reproducibility fails. However, it is difficult to publish the results of replicating experiments or negative data and there is a worrying trend in the lack of reproducibility in some forms of analysis highlighted recently with regards to the widely-used technique of fluorescence-activated cell sorting (FACS, Hines et al., 2014). Some journals have made a call specifically looking for negative data and there are indications that the NIH may be looking to drive more studies reproducing data (Collins and Tabak, 2014) but more work on this area is key.
Publish or perish
Success in grant applications and career progression relies on publications (van Dijk et al., 2014). However this can lead to hyper-competition for “high-imapct” publications and in some recent, sad, cases, a lack of truth in publishing (Sovacoll, 2008; Nosek et al., 2012). Clearly the need to publish needs to be balanced with rigorous and honest scientific communication.
Other examples of broken systems
What can we learn from other folks who have tried to fix broken systems? For instance, the aviation industry has gone from a culture of “blame the dead pilot” to rigorous investigations of aviation accidents, which lead to improvements in the system for everyone. People like Atul Gawande in his book “Better” and Malcom Gladwell in “Outliers” have studied “positive deviants”, people who do their job much better than average in order to try to understand how we could all do better, and how we can change the system so that doing better becomes routine. Could we learn from hospitals that have used real patient involvement to improve safety and health outcomes, or from the aviation industry?
Hospitals using patient engagement to bring about real change have instituted a policy that every time a decision that will impact patients is made, a patient representative will be at the table (including who to hire, what equipment to buy, pricing issues, etc.). In science/universities, who are the stakeholders? Who should be at the table when decisions are made?
Where are the dark corners? Where do we hide our embarrassing failures? Examples could include things like replications failures or professional failures like honesty slip-ups. In the cockpit, everyone is responsible for the safety of the aircraft, and anyone who sees a problem is supposed to speak up; formerly, the all-powerful pilot was not supposed to be questioned. In medicine, whistle-blowing has improved patient outcomes by changing the culture so that everyone in the room is responsible for the outcome and it’s expected for those lower down the power curve to speak up if they see something. In science, how could we learn to talk openly about our mistakes without losing professional standing? How could we reward people for saying “I think (or know) that our paper was wrong/misleading/incomplete”?
Questions – please give us feedback!
How would we like scientists to behave, and what do we think science is supposed to achieve?
What are the current rewards and incentives, and what kind of behaviors and outcomes do they promote?
What else could we do to promote or reward the behaviors/outcomes we’d like to see?
What are the best parts of your job as a scientist? How could the system be changed so that you spend more of your time doing those things?
What prevents you from doing your science and doing it well?
How can we change the culture? How can we all agree on what the goal is?
This post has been written from input provided by the moderators of the workshop on “Metrics and Incentives in Science”.
References
Collins F. S. and Tabak L. A. Policy: NIH plans to enhance reproducibility. Nature, 2014; 505: 612-13.
Hines W. C., Su Y., Kuhn I., Polyak K. and Bissell M. J. Sorting Out the FACS: A Devil in the Details. Cell Reports, 2014; 6: 779-81.
Nosek B. A., Spies J. and Motyl M. Scientific Utopia: II – Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science, 2012.
Sovacool B. K. Exploring Scientific Misconduct: Isolated Individuals, Impure Institutions, or an Inevitable Idiom of Modern Science? Journal of Bioethical Inquiry, 2008; 5 (4): 271-82.
van Dijk D., Manor O. and Carey L. B. Publication metrics and success on the academic job market. Current Biology, 2014; 24 (11) R516–R517. PI Predictor






In the early embryo, the first fate decision separates the trophectoderm from the inner cell mass (ICM). Subsequently, the ICM segregates into epiblast (Epi) and primitive endoderm (PrE), but how do cells decide which of these two fates to adopt? Claire Chazaud, Geneviève Dupont and co-workers address this question in the early mouse embryo (see p.
Intra-cytoplasmic sperm injection (ICSI) is a method of fertilisation that involves the injection of a single sperm into an oocyte. The technique has been applied successfully to generate viable offspring in human, mouse and other vertebrates, but never before has a live bird been produced using ICSI due to difficulties in mimicking avian polyspermic fertilisation. Now, on p.
Physical properties such as tissue stiffness have been shown to influence stem cell fate in vitro. It is possible that spatiotemporal changes in stiffness may influence tissue formation during development; however, it has been difficult to analyse this due to the technical limitations of measuring stiffness in embryonic tissues. In this issue (p.
Gastrulation occurs when a single-layered blastula transitions to a gastrula containing all three germ layers: ectoderm, endoderm and mesoderm. Many aspects of vertebrate gastrulation have been well-characterised using the model vertebrate Xenopus, but surprisingly little is known about how the dorsal mesoderm moves from the outer surface to the interior of the gastrula. Now, on p.
The mitotic-to-meiotic transition during spermatogenesis is essential for generating haploid spermatid cells from diploid spermatogonial cells. After differentiation and proliferation of spermatogonial cells, two rounds of meiosis are required to generate spermatid cells, but how the complex transition from mitosis to meiosis is coordinated remains unclear. In this issue (p.
In May this year, Stockholm hosted a Keystone Symposium on Adult Neurogenesis, attracting scientists from around the world despite the lack of customary snow. The symposium offered an extraordinary program, covering diverse topics that ranged from the neural stem cell lineage and regulation of neurogenesis to functional aspects of neurogenesis in homeostasis and disease, and even computational modeling. Here
Poised (bivalent) chromatin is defined by the simultaneous presence of histone modifications associated with both gene activation and repression. This epigenetic feature was first observed at promoters of lineage-specific regulatory genes in embryonic stem cells in culture. More recent work has shown that,
Biological clocks play key roles in organismal development, homeostasis and function. In recent years, much work has focused on circadian clocks, but emerging studies have highlighted the existence of ultradian oscillators – those with a much shorter periodicity than 24 h. Accumulating evidence, together with recently developed optogenetic approaches, suggests that such ultradian oscillators play important roles during cell fate decisions, and analyzing the functional links between ultradian oscillation and cell fate determination will contribute to a deeper understanding of the design principle of developing embryos. Here, Akihiro Isomura and

