GUIDELINES FOR THE REVIEW OF REPORTS OF THE NATIONAL ACADEMIES OF SCIENCES, ENGINEERING, AND MEDICINE
Any report by the National Academies of Sciences, Engineering, and Medicine—including meeting summaries, signed papers, letter reports, or other study products—must be reviewed by a diverse group of experts other than its authors before it may be released outside the institution. This independent, rigorous review is a hallmark that distinguishes the Academies from many other organizations offering scientific and technical advice on issues of national importance.
The purpose of review is to assist the authors in making their report as accurate and effective as possible and to ensure that they and the Academies are creditably represented by the report published in both their names. Review not only fulfills the institutional obligation to exercise oversight, but also provides the authors with preliminary reactions from a diverse group of experts and, as a result, enhances the clarity, cogency, and credibility of the final document. Reviewers are asked to consider whether in their judgment the evidence and arguments presented are sound and the report is fully responsive to the study charge, not whether they concur with the findings.
The report review process is overseen by the Report Review Committee (RRC), made up of approximately 30 members of the National Academy of Sciences, the National Academy of Engineering, and the National Academy of Medicine. The process is managed by the division responsible for institutional oversight of the project. This division, in consultation with the RRC, appoints a group of independent reviewers with diverse perspectives on key issues considered in the report.
A draft report is sent to reviewers only after all authors have indicated that they are satisfied with its form and content. Reviewers receive the complete report (including front matter, preface, summary or executive summary, and all appendices), along with the statement of task and this brochure. Reviewers are asked to provide written comments on any and all aspects of the draft report, but to pay particular attention to the review criteria set forth in the final section of this brochure. The authors are expected to consider all review comments and to provide written responses, which are evaluated by the monitor (appointed by the RRC) and/or review coordinator (appointed by the division). A report may not be released to the sponsors or the public, nor may its findings be disclosed, until after the review process has been satisfactorily completed and all authors have approved the revised draft. Furthermore, once the review process has been successfully completed, no changes (other than minor editorial emendations) may be made to the approved text.
Confidentiality and Anonymity
To encourage reviewers to express their views freely, the review comments are treated as confidential documents and are given to the authors of the report with identifiers removed. After submitting their comments, reviewers are asked to return or destroy the draft manuscript and to refrain from disclosing their comments or the contents of the draft. The names and affiliations of participants in the review process will be made public when the report is released (usually by acknowledgment in the printed report), but their comments remain confidential. Even after release of the report, reviewers should not divulge their comments or any changes made to the draft manuscript. These restrictions safeguard the integrity of the institutional review process.
The rationale for any findings, conclusions, and recommendations should be fully explained in the report. This explanation might include references to the literature, analysis of data, or a description of the pros and cons of the range of alternatives and the reasons for preferring a particular option. Failure to document conclusions and recommendations adequately is the most common shortcoming of draft reports. Of particular concern are recommendations calling for organizational changes or budgetary increases within government agencies, for adoption of specific legislation, or for additional work for the Academies. In general, such recommendations should be avoided unless specifically called for in the study charge.
Summaries and Executive Summaries
Every report with consensus findings and recommendations should have a brief summary or executive summary that clearly and concisely communicates the main messages of the report to the intended audiences, including non-experts and readers who do not read the full report. The summary or executive summary may not distort or go beyond the content of the report. It must clearly identify the study charge. It need not include all findings, conclusions, and recommendations, but any that are included should also appear in the chapters that follow. An executive summary may not exceed 1,000 words; a summary may be no longer than 5,000 words.
Consensus and Dissent
The Academies’ committees strive for consensus, but on rare occasion-—despite extensive deliberations-—one or more committee members may not concur with the views of the majority. Matters of disagreement should be addressed forthrightly in the report. As a final recourse, a committee member may choose to prepare a brief dissent (no more than 5,000 words) succinctly describing the issues of contention and the arguments in support of the minority view. This statement should be included as an appendix to the draft report, with reference to it in the introductory text and the table of contents. A dissent may not address issues outside the study charge, comment on the committee’s deliberations, misrepresent the majority’s views, or contain other inaccuracies. Any questions regarding the appropriateness of material included in a dissent are referred to the RRC co-chairs. Although reviewers’ comments on the statement are given to its author for consideration, no formal written response is required.
The Academies’ reports cover a broad range of topics and appear in a variety of different forms. Although no rigid set of criteria is likely to be applicable to all reports, reviewers may find the following questions useful in formulating their comments. (Separate sets of criteria are used for workshop reports/summaries and proceedings.)
1. Is the charge clearly described in the report? Are all aspects of the charge fully addressed? Do the authors go beyond their charge or their expertise?
2. Are the conclusions and recommendations adequately supported by evidence, analysis, and argument? Are uncertainties or incompleteness in the evidence explicitly recognized? If any recommendations are based on value judgments or the collective opinions of the authors, is this acknowledged and are adequate reasons given for reaching those judgments?
3. Are the data and analyses handled competently? Are statistical methods applied appropriately?
4. Are sensitive policy issues treated with care? If the report contains recommendations pertaining to the reorganization of an agency or the creation of a new institutional entity, is it specifically called for in the statement of task and are the advantages and disadvantages of alternative options, including the status quo, considered? If the report contains a budgetary recommendation, is it specifically called for in the statement of task?
5. Are the report’s exposition and organization effective? Is the title appropriate?
6. Is the report fair? Is its tone impartial and devoid of special pleading?
7. Does the summary or executive summary concisely and accurately describe the key findings and recommendations? Is it consistent with other sections of the report?
8. Are signed papers or appendices, if any, relevant to the charge? If the report relies on signed papers to support consensus findings or recommendations, do the papers meet criterion 3?
9. What other significant improvements, if any, might be made in the report
In providing comments, reviewers are encouraged to distinguish issues they consider to be of general/major concern from other, less significant points.
REVIEW GUIDELINES FOR PROCEEDINGS OF A WORKSHOP AND PROCEEDINGS OF A WORKSHOP IN BRIEF
The products of convening activities, called “proceedings of a workshop” or “proceedings of a workshop—in brief,” reflect what transpired at the workshop and do not present the consensus view of the planning committee or workshop participants. The following questions may be used as a guide to formulating your comments.
1. Does the introductory material clearly explain the purpose and context for the proceedings? Does the proceedings cover what it says it does and (if needed) does it explain what it does not cover?
2. Is the content of the proceedings clear and comprehensible?
3. If you attended the workshop, does the content of the proceedings accurately reflect the presentations, discussions, and papers?
4. Are all views expressed in the proceedings appropriately attributed to individual participants or groups of participants in the workshop? Does the proceedings contain any statements that might be misconstrued as reflecting consensus judgments of the authoring committee or workshop participants as a whole?
5. Is the presentation of material balanced and fair? Are any sensitive policy issues treated with proper care? Is the title appropriate?
Support staff in your division reports offices will have copies of these brochures for distribution to your committees. Alternatively, please call Dalia Hedges x3155 if you need copies of the brochures.
REVIEW GUIDELINES FOR COLLECTIONS OF PAPERS OR PRESENTATIONS AT A WORKSHOP
Collected papers consist of individual workshop presentations or papers, each of which is attributed to its presenter(s) or author(s). The document should include introductory material describing the event in a non-substantive manner (e.g., location, sponsorship, purpose) without drawing any inferences or conclusions or summarizing the discussions. The following questions may be used as a guide to formulating your comments.
1. Is the content of the paper clear and comprehensible?
2. Is the paper of satisfactory quality, relevant to the goals of the workshop, and appropriate for inclusion in this compilation?
Report Review Committee Membership Roster
Current membership roster of the the Report Review Committee (RRC), composed of National Academy of Sciences, National Academy of Engineering, and the National Academy of Medicine members. The RRC oversees the institutional report review process.
RRC Membership Roster
Ellen Wright Clayton, NAM
Craig Weaver Professor of Pediatrics, Professor of Law
Center for Biomedical Ethics and Society
M. Granger Morgan, NAS
Department of Engineering and Public Policy
Carnegie Mellon University
Robert F. Sproull, NAE Vice President and Director [Retired] Oracle Labs
Maurice Falk University Professor of Statistics and Social Science
Carnegie Mellon University
Huda Akil, NAS, NAM
Distinguished University Professor and Quarton Professor of Neurosciences
Department of Psychiatry
University of Michigan
James O. Berger, NAS
Arts and Sciences Professor of Statistics
Department of Statistics Science
Bobbie A. Berkowitz, NAM
Dean and Professor
School of Nursing
Philip J. Cook, NAM
ITT/Sanford Professor of Public Policy
Sanford School of Public Policy
Susan J. Curry, NAM
Dean, College of Public Health
College of Public Health
The University of Iowa
Greg J. Duncan, NAS
Distinguished Professor of Education
School of Education
University of California, Irvine
Maryellen L. Giger, NAE
A.N. Pritzker Professor of Radiology and Medical Physics
Department of Radiology
The University of Chicago
Paul R. Gray, NAE
Executive Vice Chancellor and Provost, Emeritus
University of California, Berkeley
Bradford H. Gray*, NAM
The Urban Institute
David M. Karl, NAS
Professor of Oceanography
Department of Oceanography
University of Hawaii
Michael R. Ladisch, NAE
Distinguished Professor & Director, LORRE
Potter Engineering Center
Charles F. Manski, NAS
Board of Trustees Professor in Economics
Department of Economics
Bonnie J. McCay, NAS
Board of Governors Distinguished Service Professor
Department of Human Ecology
Rutgers, The State University of New Jersey
Robin K. McGuire, NAE
Lettis Consultants International, Inc.
Ali Mosleh, NAE
Distinguished Professor and Evelyn Knight Chair in Engineering
Henry Samueli Sch of Engineering & Applied Science
University of California, Los Angeles
Olufunmilayo F. Olopade, NAM
Walter L. Palmer Distinguished Service Professor of Medicine
The University of Chicago
Martin A. Philbert, NAM
Professor and Dean
School of Public Health
University of Michigan
Edwin P. Przybylowicz, NAE
Senior Vice President [Retired]
Eastman Kodak Company
Marcia J. Rieke, NAS
Regents' Professor of Astronomy
Department of Astronomy
University of Arizona
Stephen M. Robinson, NAE
University of Wisconsin-Madison
Sara Rosenbaum, NAM
Harold and Jane Hirsh Professor of Health Law and Policy
Department of Health Policy
Milken Institute School of Public Health
The George Washington University
Donald M. Steinwachs, NAM
Institute for Policy Studies
Johns Hopkins University
Executive Director, RRC
Administrative Coordinator, RRC
Associate Executive Director, RRC
Program Officer, RRC
Senior Report Review Officer, RRC