OHAT Systematic Review
The NTP Office of Health Assessment and Translation (OHAT) develops literature-based evaluations to reach conclusions about potential human health hazards and to examine the state of the science. In 2012, OHAT began exploring and developing an approach for implementation of systematic review methodology to carry out these evaluations (Birnbaum et al. 2013). The output from an evaluation can vary including, but not limited to: NTP Monograph, state-of-the-science workshop report, or peer-reviewed journal publication. All evaluations follow a similar process, the OHAT Evaluation Process, with opportunities for external scientific, public, and interagency input.
Fact Sheet on Systematic Review
See the Fact Sheet on Systematic Review to learn more about the systematic review process.
Systematic Review MethodsHandbook for Conducting Systematic Reviews
The “Handbook for Conducting a Literature-Based Health Assessment Using OHAT Approach for Systematic Review and Evidence Integration” provides standard operating procedures for the implementation of systematic review in OHAT evaluations. The standard operating procedures are based on (1) lessons learned from developing protocols for two case studies for implementing systematic review, (2) consideration of public comments received on systematic review during the past two years, and (3) discussions with experts at other organizations and agencies working on applying methods of systematic review to environmental health and toxicology. The handbook is a living document and will be updated as methodological practices are refined and evaluated and strategies are identified that improve the reliability, ease, and efficiency of conducting systematic reviews.Risk of Bias Tool
The handbook for conducting systematic reviews presents OHAT’s tool for assessing study quality, or “risk of bias.” The tool applies a parallel approach to the evaluation of risk of bias for human and non-human animal studies to facilitate consideration of potential bias across evidence streams with common terminology and “domains” or areas of bias. The risk-of-bias domains and questions for experimental animal studies are based on established guidance for experimental human studies (randomized clinical trials). Detailed instructions for response are provided in the OHAT tool. The risk-of-bias rating tool will be refined and updated as practices evolve and empirical evidence is developed to support risk of bias elements. OHAT is currently refining the OHAT tool to assess potential bias for in vitro or mechanistic studies.
OHAT Systematic Review Publications
Systematic review and evidence integration for literature-based environmental health science assessments.
Rooney AA, Boyles AL, Wolfe MS, Bucher JR, Thayer KA. Environ Health Perspect. 2014 Jul;122(7):711-8. doi: 10.1289/ehp.1307972. Epub 2014 Apr 18.
Intersection of Systematic Review Methodology with the NIH Reproducibility Initiative.
Thayer KA, Wolfe MS, Rooney AA, Boyles AL, Bucher JR, Birnbaum LS.
Environ Health Perspect. 2014 Jul 1;122(7):A176-7. doi: 10.1289/ehp.1408671. No abstract available.
Implementing systematic review at the National Toxicology Program: status and next steps.
Birnbaum LS, Thayer KA, Bucher JR, Wolfe MS.
Environ Health Perspect. 2013 Apr;121(4):A108-9. doi: 10.1289/ehp.1306711. No abstract available.
Ongoing Methods Development Activities
OHAT is actively engaged with researchers in the systematic review and human health communities to identify strategies that improve the reliability and efficiency of the process for conducting and communicating the results of systematic reviews. The OHAT Approach is among the first systematic review methods designed to handle the breadth of data relevant to environmental health sciences (e.g., human, animal, and in vitro/mechanistic studies).
There are established methods for assessing individual study quality (e.g., risk of bias) and for evaluating confidence in the results of group of studies (or body of evidence) for human and animal studies. However, methods are less developed for mechanistic data, which include outcomes from in vitro, mechanistic, cellular, or genomic studies. Therefore, as an ongoing research effort, OHAT is developing a framework for mechanistic data that is conceptually similar to the approach for human and animal studies.
OHAT is currently refining the OHAT tool to assess potential bias for in vitro or mechanistic studies.
Past Methods Development Activities
Timeline of Outreach Activities in OHAT's Implementation of Systematic Review
The timeline below shows NTP activities associated with the development of the OHAT approach for systematic review. For more information about the activities, please see the “Events and Activities” webpage.
Webinar on Lessons Learned in Application of the OHAT Framework for Systematic Review and Evidence Integration to Case Studies
NTP held a web-based public meeting on July 31, 2014, 12:30-3:30 p.m. EDT focused on lessons learned during the case-study process. The first part of the webinar consisted of a series of topic or “lesson” focused presentations, followed by a short question-and-answer period. The second part of the webinar was a general discussion period when the public made brief comments on or asked questions about the application of the OHAT Approach to the case studies and lessons learned.
Draft OHAT Approach - February 2013
The Draft OHAT Approach for Systematic Review and Evidence Integration for Literature-based Health Assessments – February 2013 (“Draft OHAT Approach – February 2013”) provides a seven-step framework for conducting evaluations using principles of systematic review. The Draft OHAT Approach – February 2013 reflects consideration of input from a working group of the NTP Board of Scientific Counselors (BSC), the BSC, and the public (see for information, presentations, and minutes from the June 21-22 and December 11, 2012 meetings). This framework for conducting evaluations integrates within the OHAT Evaluation Process and includes multiple opportunities for external scientific, public, and interagency input.
- Federal Register Notice (78 FR 37 published February 25, 2013)
Draft Case-Study Protocols
To assist with determining if additional refinement or revision to the Draft OHAT Approach – February 2013 might be needed, OHAT plans to apply it to several case-study evaluations. The first two case studies evaluate the evidence regarding the association of (1) bisphenol A (BPA) exposure with obesity and (2) perfluorooctanoic acid (PFOA) or perfluorooctane sulfonate (PFOS) exposure with immunotoxicity. Prior to initiating these evaluations, OHAT has developed protocols that include specific elements for how the seven steps in the Draft OHAT Approach – February 2013 would be carried out. The protocols on BPA exposure and obesity and PFOA or PFOS exposure and immunotoxicity were released on April 9, 2013. Persons interested in the Draft OHAT Approach – February 2013 and the protocols are encouraged to access this website to stay abreast of the most current information.
- Download the Draft OHAT Approach for Systematic Review and Evidence Integration for Literature-Based Health Assessments – February 2013
- Draft Protocol for Systematic Review to Evaluate the Evidence for an Association Between Bisphenol A (BPA) Exposure and Obesity
- Draft Protocol for Systematic Review to Evaluate the Evidence for an Association Between Perfluorooctanoic Acid (PFOA) or Perfluorooctane Sulfonate (PFOS) Exposure and Immunotoxicity
NTP invited public comments on the Draft OHAT Approach – February 2013. Two protocols were released to illustrate the application of this framework: (1) BPA exposure and obesity and (2) PFOA or PFOS exposure and immunotoxicity. The deadline for submission of public comments was June 11, 2013. The comments are posted on the NTP website and persons submitting them are identified by their name and affiliation and/or sponsoring organization, if applicable.
Webinar on the Assessment of Data Quality in Animal Studies – March 2013
NTP held a web-based public meeting on March 20, 2013, 2:00-4:00 p.m. EDT focused on assessment of data quality in animal studies. The meeting began with invited presentations related to the assessment of study quality and was followed by a general discussion period.
Web-Based Informational Meeting - April 2013
NTP held a public, web-based, informational meeting on April 23, 2013, 12:00-4:00 p.m. EDT to provide an overview of the Draft OHAT Approach – February 2013, describe the contents in the two case-study protocols, and respond to questions from the public on any of the documents.
Webinar on OHAT Approach for Systematic Review - September 2013
NTP held a public webinar on September 26, 2013 from 1:00 – 4:00 p.m. EDT to gain additional clarity on issues raised in public comments. For each topic covered, there was a brief presentation by OHAT staff and then the opportunity for discussion in a question and answer session. The webinar also included a general discussion period for any additional comments or questions from the attendees.
- Meeting page for Webinar on OHAT Approach for Systematic Review
- NTP Listserv Announcement (August 6, 2013)
Future updates on the Draft OHAT Approach for Systematic Review and Evidence Integration for Literature-Based Health Assessments will be posted on this webpage. Persons interested in the Draft OHAT Approach – February 2013 and the protocols are encouraged to access this website to stay abreast of the most current information.
Individuals interested in receiving updates on this project are encouraged to register to the NTP Listserv.
Discussions with the NTP Board of Scientific Counselors
NTP presented and discussed the Draft OHAT Approach at several BSC meetings (listed below). A BSC working group was formed to provide input on the Draft OHAT Approach. Minutes, presentations, and other meeting materials from past BSC meetings.
- June 22, 2012: BSC meeting with presentation and discussion on Systematic Review and New Tools for Information Management and Data Display.
- August 28–29, 2012: BSC Working Group met to evaluate the suitability and transparency of the proposed approach for reaching evidence assessment conclusions from a body of evidence for non-cancer health effects.
- December 11, 2012: BSC Working Group released and discussed their draft report at the BSC meeting.
- December 11, 2012: BSC meeting with presentations and discussions on (1) NTP Approach for Systematic Review and Evidence Integration for Literature-based Health Assessments, (2) NTP BSC Working Group Report on the Draft NTP Approach, and (3) Implementation of the NTP Approach in Office of Hazard Assessment and Translation (OHAT) Evaluations.
- June 25, 2013: BSC meeting with presentation and discussion on Systematic Review and Evidence Integration for Literature-Based Health Assessments.