The importance of research within modern society is undeniable. Ongoing scientific research enables the development of new technologies that can improve health and quality of life, provide new ways to harness and sustain the Earth’s natural resources, produce clean and renewable energy, bolster the economy, and yield a more complete understanding of the natural world. Materials research in particular has provided wide-reaching benefits to society and will continue to play a key role in enabling future advancements. Many technologies have reached a point where further advancements require the development of new materials, or materials systems, that can be manipulated to have the required properties, characteristics, and/or performance to suit specific applications.
Despite the important role that scientific research plays in society, concerns about integrity, transparency, and reproducibility have recently become a discussion point both within the scientific community and among policymakers. To address these concerns, the US National Academies of Sciences, Engineering, and Medicine (Academies) has undertaken three studies—one on integrity in research, one on open science, and one on reproducibility and replicability.
Scientific integrity has always been important to scientists, but the changing landscape, both in science and policy, has produced a need and desire to improve the approaches used to promote and support integrity in research. The first of the three Academies studies, Fostering Integrity in Research (reported in the April 2018 issue of MRS Bulletin), was completed in April 2017. The report provides a review of current integrity-related issues and best practices as well as recommendations to improve integrity within the scientific enterprise. The report charges the science community with better aligning the reality of the research environment with its ideals.
The second Academies study is looking at the challenges and benefits of open science, which is defined as free public access to the results of publicly funded scientific research. Begun in July 2017 and ex-pected to take 18 months to complete, Toward an Open Science Enterprise will focus on providing solutions that can be employed to shift publicly funded scientific research results from limited access into open science. The Academies website for the study calls for the products of public research to be “findable, accessible, interoperable, and reusable (FAIR), with limited exceptions for privacy, proprietary business claims, and national security.” Conducted by an ad hoc committee under the Board on Research Data and Information, the study will culminate in a consensus report that will detail the findings and recommendations of the committee.
The focus of this open science study is to identify a strategy to make open science the default for all publicly funded research. The seven tasks specified within the study range from determining to what extent open science is already practiced in science and engineering, to identifying both barriers to and facilitators of open science, to recommending specific solutions (both existing and new) that would facilitate the implementation of open science on a broader scale (with specific focus on solutions for federal science agencies).
Within the science community, the idea of open science produces mixed responses. “The benefits [of open science] are straightforward—taxpayers would have access to results they have paid for, and people not affiliated with institutions that provide access can also participate in scientific inquiry,” says Shefford P. Baker, associate professor of materials science and engineering at Cornell University, and chair of the Materials Research Society (MRS) Publications Committee. Alan J. Hurd, executive advisor at Los Alamos National Laboratory and former president of MRS, agrees that open science has good aspects like greater transparency, access for those lacking the resources to pay for scientific articles, and the opportunity for scientific studies to be more widely read.
But Hurd characterizes the downside of open science as “frightening” and points out that it might lead to the requirement for user facilities to open up their databases or provide permanent storage of, and access to, data. Indefinite data storage on this scale would come with a hefty price tag, and neither researchers nor Congress are likely to want to cover the additional costs. Hurd also posits that “a few very dominant journal publishers may become even more dominant” if open science becomes the norm. This is primarily because producing quality journals requires capital, much of which is currently generated by selling access to content. Baker also points out that moving toward open access has already produced an increase in the number of predatory journals, saying “if people can simply pay to have their work published, there will be those who will be motivated to publish it, regardless of its veracity.”
Baker’s observation highlights another area of concern—the quality and authenticity of published results. According to a 2016 Nature survey of 1500 scientists, 90% of those surveyed believe there is a “reproducibility crisis” in scientific research, with 52% saying it is a significant crisis. The third Academies study, begun in December 2017, seeks to shed light on this issue. Reproducibility and Replicability in Science is the result of a Congressional mandate that reflects some policymakers’ diminishing trust in science. The committee in charge of the 18-month study consists of 15 experts spanning a range of science disciplines, and will highlight those areas of science that already have good data on reproducibility and replicability, and those that do not. The study will also review methods to improve reproducibility and replicability, present examples of best practices, and explore issues that hinder scientists’ ability to reproduce or replicate experiments and results across the breadth of the science enterprise.
Based on the Nature survey, reproducibility appears to be a problem across scientific disciplines, and both Hurd and Baker agree that materials research is no exception. Baker says, “It is not surprising that so many published results in materials research are irreproducible. Most interesting materials problems are complex, with many variables. Under high pressure to publish and little risk of being held accountable, many researchers simply conduct their work using their own distributions of dependent and independent variables. The results may be correct for that set of experiments, but cannot be reproduced.” Hurd agrees with Baker’s assessment that the pressure to succeed, often measured by the ability to produce high impact and/or large numbers of published results, is a driving factor of reproducibility issues within materials research. He also points out that open science might produce greater transparency and serve as one of the possible solutions to the issue of reproducibility in research.
Baker also stresses the importance of taking a serious look at the issue of integrity in scientific research, and what role it plays in the “reproducibility crisis.” According to Baker, shining the light on these issues and bringing the discussion more out into the open would be an excellent first step toward a solution. “I suspect that people [within the materials community] are just not aware of the magnitude of the problem, so don’t take the snippets that they do hear too seriously,” Baker says. He also points out that sometimes even when it is discussed, the importance of the issue is overlooked, saying, “the well-known joke that ‘typical results’ can be translated as ‘the only results like this we ever saw, but they agree with my predictions’ is probably true in more than a few instances.”
Integrity, transparency, and reproducibility in scientific research are multifaceted issues that will require changes to be implemented across the scientific enterprise, including the materials research community as it develops the critical technologies of the future.
You have
Access
The importance of research within modern society is undeniable. Ongoing scientific research enables the development of new technologies that can improve health and quality of life, provide new ways to harness and sustain the Earth’s natural resources, produce clean and renewable energy, bolster the economy, and yield a more complete understanding of the natural world. Materials research in particular has provided wide-reaching benefits to society and will continue to play a key role in enabling future advancements. Many technologies have reached a point where further advancements require the development of new materials, or materials systems, that can be manipulated to have the required properties, characteristics, and/or performance to suit specific applications.
Despite the important role that scientific research plays in society, concerns about integrity, transparency, and reproducibility have recently become a discussion point both within the scientific community and among policymakers. To address these concerns, the US National Academies of Sciences, Engineering, and Medicine (Academies) has undertaken three studies—one on integrity in research, one on open science, and one on reproducibility and replicability.
Scientific integrity has always been important to scientists, but the changing landscape, both in science and policy, has produced a need and desire to improve the approaches used to promote and support integrity in research. The first of the three Academies studies, Fostering Integrity in Research (reported in the April 2018 issue of MRS Bulletin), was completed in April 2017. The report provides a review of current integrity-related issues and best practices as well as recommendations to improve integrity within the scientific enterprise. The report charges the science community with better aligning the reality of the research environment with its ideals.
The second Academies study is looking at the challenges and benefits of open science, which is defined as free public access to the results of publicly funded scientific research. Begun in July 2017 and ex-pected to take 18 months to complete, Toward an Open Science Enterprise will focus on providing solutions that can be employed to shift publicly funded scientific research results from limited access into open science. The Academies website for the study calls for the products of public research to be “findable, accessible, interoperable, and reusable (FAIR), with limited exceptions for privacy, proprietary business claims, and national security.” Conducted by an ad hoc committee under the Board on Research Data and Information, the study will culminate in a consensus report that will detail the findings and recommendations of the committee.
The focus of this open science study is to identify a strategy to make open science the default for all publicly funded research. The seven tasks specified within the study range from determining to what extent open science is already practiced in science and engineering, to identifying both barriers to and facilitators of open science, to recommending specific solutions (both existing and new) that would facilitate the implementation of open science on a broader scale (with specific focus on solutions for federal science agencies).
Within the science community, the idea of open science produces mixed responses. “The benefits [of open science] are straightforward—taxpayers would have access to results they have paid for, and people not affiliated with institutions that provide access can also participate in scientific inquiry,” says Shefford P. Baker, associate professor of materials science and engineering at Cornell University, and chair of the Materials Research Society (MRS) Publications Committee. Alan J. Hurd, executive advisor at Los Alamos National Laboratory and former president of MRS, agrees that open science has good aspects like greater transparency, access for those lacking the resources to pay for scientific articles, and the opportunity for scientific studies to be more widely read.
But Hurd characterizes the downside of open science as “frightening” and points out that it might lead to the requirement for user facilities to open up their databases or provide permanent storage of, and access to, data. Indefinite data storage on this scale would come with a hefty price tag, and neither researchers nor Congress are likely to want to cover the additional costs. Hurd also posits that “a few very dominant journal publishers may become even more dominant” if open science becomes the norm. This is primarily because producing quality journals requires capital, much of which is currently generated by selling access to content. Baker also points out that moving toward open access has already produced an increase in the number of predatory journals, saying “if people can simply pay to have their work published, there will be those who will be motivated to publish it, regardless of its veracity.”
Baker’s observation highlights another area of concern—the quality and authenticity of published results. According to a 2016 Nature survey of 1500 scientists, 90% of those surveyed believe there is a “reproducibility crisis” in scientific research, with 52% saying it is a significant crisis. The third Academies study, begun in December 2017, seeks to shed light on this issue. Reproducibility and Replicability in Science is the result of a Congressional mandate that reflects some policymakers’ diminishing trust in science. The committee in charge of the 18-month study consists of 15 experts spanning a range of science disciplines, and will highlight those areas of science that already have good data on reproducibility and replicability, and those that do not. The study will also review methods to improve reproducibility and replicability, present examples of best practices, and explore issues that hinder scientists’ ability to reproduce or replicate experiments and results across the breadth of the science enterprise.
Based on the Nature survey, reproducibility appears to be a problem across scientific disciplines, and both Hurd and Baker agree that materials research is no exception. Baker says, “It is not surprising that so many published results in materials research are irreproducible. Most interesting materials problems are complex, with many variables. Under high pressure to publish and little risk of being held accountable, many researchers simply conduct their work using their own distributions of dependent and independent variables. The results may be correct for that set of experiments, but cannot be reproduced.” Hurd agrees with Baker’s assessment that the pressure to succeed, often measured by the ability to produce high impact and/or large numbers of published results, is a driving factor of reproducibility issues within materials research. He also points out that open science might produce greater transparency and serve as one of the possible solutions to the issue of reproducibility in research.
Baker also stresses the importance of taking a serious look at the issue of integrity in scientific research, and what role it plays in the “reproducibility crisis.” According to Baker, shining the light on these issues and bringing the discussion more out into the open would be an excellent first step toward a solution. “I suspect that people [within the materials community] are just not aware of the magnitude of the problem, so don’t take the snippets that they do hear too seriously,” Baker says. He also points out that sometimes even when it is discussed, the importance of the issue is overlooked, saying, “the well-known joke that ‘typical results’ can be translated as ‘the only results like this we ever saw, but they agree with my predictions’ is probably true in more than a few instances.”
Integrity, transparency, and reproducibility in scientific research are multifaceted issues that will require changes to be implemented across the scientific enterprise, including the materials research community as it develops the critical technologies of the future.