The Future of Research Symposium:

Metrics: The Folly of Hoping for X whilst Rewarding Y

This is the third of four posts relating to the Future of Research symposium which was announced in a previous blog post. Each of these posts will discuss a topic that is the focus of a workshop at the Symposium. Even if you can’t attend, please tweet @FORsymp with suggestions, or follow us to respond to our questions about what YOU, trainee scientists, think is important. The hashtag for this post on the Workforce workshop is: #FORmetrics

Rewards and incentives

What is “good science”? What is it that we want science to accomplish?

Does our current system reward “good science”, our shared goal, or are there perverse incentives that do the opposite? What are the problems and how could we reward scientists and institutions to produce the behaviors (such as collaboration, openness and honesty) that we believe support “good science”?

We all have ideas about what science is supposed to do/achieve, but these outcomes happen at a broad, communal, systems level. Meanwhile, rewards/incentives operate on individual scientists. Often, there is a disconnect between those two.

Many of the current incentives work on individuals or small groups of people (grants, jobs, fame, etc.), while many of the things we want science to do (cure disease, figure out how the world works, etc.) need to happen at a broader systems level. This may contribute to the disconnect between incentives and desired outcomes.

Intrinsic motivators may include curiosity, desire for freedom, etc.; the reasons people got into science in the first place. The extrinsic motivators are things like grant money, lab space, prestige, etc. How can we reinforce or encourage people’s own intrinsic motivations (at least the ones that lead to good science)?

The objective of the Metrics workshop is to ask: How can we fix the current system of incentives so that we reward scientists and institutions for the behaviors that we believe support “good science”?

Key areas where science “breaks”

Reproducibility and negative data
Part of the scientific endeavour is to provide checks and balances and reproduce results, or highlight when reproducibility fails. However, it is difficult to publish the results of replicating experiments or negative data and there is a worrying trend in the lack of reproducibility in some forms of analysis highlighted recently with regards to the widely-used technique of fluorescence-activated cell sorting (FACS, Hines et al., 2014). Some journals have made a call specifically looking for negative data and there are indications that the NIH may be looking to drive more studies reproducing data (Collins and Tabak, 2014) but more work on this area is key.

Publish or perish
Success in grant applications and career progression relies on publications (van Dijk et al., 2014). However this can lead to hyper-competition for “high-imapct” publications and in some recent, sad, cases, a lack of truth in publishing (Sovacoll, 2008; Nosek et al., 2012). Clearly the need to publish needs to be balanced with rigorous and honest scientific communication.

Other examples of broken systems

What can we learn from other folks who have tried to fix broken systems? For instance, the aviation industry has gone from a culture of “blame the dead pilot” to rigorous investigations of aviation accidents, which lead to improvements in the system for everyone. People like Atul Gawande in his book “Better” and Malcom Gladwell in “Outliers” have studied “positive deviants”, people who do their job much better than average in order to try to understand how we could all do better, and how we can change the system so that doing better becomes routine. Could we learn from hospitals that have used real patient involvement to improve safety and health outcomes, or from the aviation industry?

Hospitals using patient engagement to bring about real change have instituted a policy that every time a decision that will impact patients is made, a patient representative will be at the table (including who to hire, what equipment to buy, pricing issues, etc.). In science/universities, who are the stakeholders? Who should be at the table when decisions are made?

Where are the dark corners? Where do we hide our embarrassing failures? Examples could include things like replications failures or professional failures like honesty slip-ups. In the cockpit, everyone is responsible for the safety of the aircraft, and anyone who sees a problem is supposed to speak up; formerly, the all-powerful pilot was not supposed to be questioned. In medicine, whistle-blowing has improved patient outcomes by changing the culture so that everyone in the room is responsible for the outcome and it’s expected for those lower down the power curve to speak up if they see something. In science, how could we learn to talk openly about our mistakes without losing professional standing? How could we reward people for saying “I think (or know) that our paper was wrong/misleading/incomplete”?

Questions – please give us feedback!

How would we like scientists to behave, and what do we think science is supposed to achieve?

What are the current rewards and incentives, and what kind of behaviors and outcomes do they promote?

What else could we do to promote or reward the behaviors/outcomes we’d like to see?

What are the best parts of your job as a scientist? How could the system be changed so that you spend more of your time doing those things?

What prevents you from doing your science and doing it well?

How can we change the culture? How can we all agree on what the goal is?

This post has been written from input provided by the moderators of the workshop on “Metrics and Incentives in Science”.

References

Collins F. S. and Tabak L. A. Policy: NIH plans to enhance reproducibility. Nature, 2014; 505: 612-13.

Hines W. C., Su Y., Kuhn I., Polyak K. and Bissell M. J. Sorting Out the FACS: A Devil in the Details. Cell Reports, 2014; 6: 779-81.

Nosek B. A., Spies J. and Motyl M. Scientific Utopia: II – Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science, 2012.

Sovacool B. K. Exploring Scientific Misconduct: Isolated Individuals, Impure Institutions, or an Inevitable Idiom of Modern Science? Journal of Bioethical Inquiry, 2008; 5 (4): 271-82.

van Dijk D., Manor O. and Carey L. B. Publication metrics and success on the academic job market. Current Biology, 2014; 24 (11) R516–R517. PI Predictor

(2 votes)

Loading...