This paper is available in alternate formats: | Download Word | Download pdf |
The Work Group meeting was designed to elicit informed commentary regarding the feasibility and utility of an OSEP-sponsored process to identify and disseminate effective products and practices developed with IDEA-Part D support.
Lizanne DeStefano (University of Illinois), assisted RTP in arranging the meeting, including recruitment of the participants and developing the meeting agenda in conjunction with Lou Danielson and Renee Bradley of OSEP. The American Institutes for Research (AIR) hosted the meeting, handled logistical arrangements, and assisted in the preparation of this report.
The procedures that were used to facilitate the meeting and achieve RTP's goal for meeting participant interactions and commentary are described below.
The members of the Work Group were nationally recognized experts who were knowledgeable about review and dissemination practices. These experts, collectively, had previous experience with Federal review and dissemination mechanisms, knowledge of effective IDEA-Part D products and practices, research expertise, and familiarity with dissemination to local audiences. The Work Group consisted of seven external members and six from OSEP.
External participants and their institutional affiliations are listed below:
The following representatives from RTP/ OSEP also attended the meeting:
Attachment A contains contact information for both the Federal and non-Federal members of the Work Group.
Renee Bradley of RTP (OSEP) facilitated the work group meeting. The agenda consisted of the following principal activities:
The Work Group meeting closed with the members agreeing to respond to a draft summary report and participate in a session on this topic at OSEP's Research Project Director's Meeting in July (2001).
Prior to the meeting, Lizanne DeStefano, (University of Illinois), and Don Dailey, (AIR), conducted a thorough review of the literature on Federal efforts to identify effective practices in education and the Federal role in dissemination. Based on an extensive review of the literature, they created a briefing booklet of the most relevant documents for use at the meeting. In addition, members of the work group were sent, in advance of the meeting, a set of five papers. These papers were:
Work Group Members were also sent a memorandum restating the charge to the group and providing a framework for analysis and synthesis of the readings. Attachment B contains the full list of citations compiled for review at the meeting and contained in the briefing book.
After an introductory session in which Lou Danielson set the context for the work of the group and reiterated their charge, the members began their discussion of the feasibility and advisability of OSEP sponsoring a review and dissemination process for IDEA-Part D products and practices. Three guiding questions were posed by RTP and used to organize the discussion and commentary of the Work Group..
Guiding Questions for Work Group Discussions
Lizanne DeStefano presented a synthesis of information on Federal efforts to identify best practice in education over the last 30 years. A summary of these Federal initiatives is presented below, followed by an overview of the Work Group discussions regarding the benefits and challenges of OSEP considering such a review and dissemination process for IDEA-Part D products and practices. This section concludes with a review of the Work Group discussion of an appropriate role for OSEP in developing a new review and dissemination process.
There have been several major national initiatives, sponsored by various Federal agencies, to identify and disseminate effective practices, especially during the 1970s and 1980s. These Federal initiatives shared several distinguishing characteristics. For example, each of them was:
The Joint Dissemination Review Panel (JDRP) and its successor, the Program Effectiveness Panel (PEP), and their related dissemination structure, the National Diffusion Network (NDN), were clearly the most recognized and longstanding efforts of this kind.
The Work Group Members actively discussed the organization, function, and impact of the JDRP, PEP and NDN, relying on personal experiences, the readings, and Dr. DeStefano's presentation to inform the discussion. Topics were wide ranging and included (a) an analysis of changes in the political and practical context from the 1970s to present, (b) the types of entities eligible for review and qualifications of reviewers, (c) the criteria used to judge effectiveness and the composition and function of the JDRP, (d) the structure and function of the NDN, (e) perceived impact of the JDRP, PEP, and NDN, and (f) reasons for their demise. These analyses of the perceived strengths and weaknesses of the JDRP, PEP, and NDN initiatives set the context for the Work Group members to generate a number of possible benefits and challenges that seem likely to result from a development of a review and dissemination process initiated at OSEP.
Work Group members cited a number of possible benefits for OSEP. For example, they cited the benefit of explicit criteria for demonstrating effectiveness as useful to the field, independent of the role they play in the review process, because they represent a reaffirmation of respect for an evidentiary practice base and make public the high standards established for effective practice. The criteria would serve as a statement of professional values and standards for the field, and communicate expectations.
Benefits of a Review and Dissemination Process
The presence of a list of Federally-endorsed effective practices strengthens the capacity of local educators, family members, and policy makers to make informed decisions about practices that improve the quality of schooling and outcomes for students with disabilities. Members expressed concern that, currently, local educators and families do not have an easily accessible base of information from which they can make decisions as informed consumers. As a result, there is a strong sense among the members that opportunities exist for programs lacking evidence of effectiveness to be successful in marketing their products. Equally frustrating to members is that effective practices are not reviewed, endorsed, and promoted in a systematic way that distinguishes them from ineffective practices.
The Work Group believes a review and dissemination process has potential for strengthening the linkages among IDEA-Part D supported research and innovation, technical assistance and dissemination, and local improvement. Members talked about how OSEP has developed a wide range of technical assistance and dissemination programs that could be used for transmitting useful information about validated practices to local educators and families in friendly language. The problem is that too often the connection is not made between the research community and those responsible for TA/dissemination. A review and dissemination process could serve as a focal point for building this capacity and focusing dissemination on research-validated practices which schools desperately need.
The publication of a set of Federally-endorsed products and practices makes a powerful statement about the impact of IDEA-Part D supported research and innovation projects and underscores the importance of Federal support for research and development. Implicit in a review and dissemination process, with its emphasis on positive changes in student outcomes, is an affirmation of the principal of accountability, which, of course, is a major focus of today's policy discussions and initiatives. Efficient and effective use of resources is a continuing issue of concern among policymakers, citizens, and parents. It is important to demonstrate the impact resources have on helping schools become better equipped to improve learning and other outcomes for students with disabilities.
Work Group members also talked about possible challenges for OSEP (see side bar: Challenges of a Review and Dissemination Process). For example, they discussed complications involved in disseminating the results of reviews. The audiences and mechanisms for dissemination in special education have changed since JDRP. More students are being educated in general education classrooms by general educators. Reaching these audiences will require a different set of strategies than traditionally used for communicating with special educators. Furthermore, the dissemination structure for reaching special educators has changed. For example, NDN used a system of state contacts that helped schools to implement approved products and practices. This is no longer in place. The current technical assistance and dissemination structure has great potential for reaching diverse audiences, but it is complex. Using it for this purpose could be expensive.
Challenges of a Review and Dissemination Process
Members raised the issue of how the field continues to model improvement through programs, yet the idea of "program" as the entity to be reviewed may be misleading and may affect the usefulness of the results. It is more frequent that smaller units such as practices or interventions are transported, rather than entire programs. The review process could be used to identify best practices within programs, though the structure and methodology for accomplishing this has not been discussed.
The Work Group expressed concern about resources. One issue of critical concern is that the cost of operating a review and dissemination process may exceed its benefits or the level of support that OSEP can provide. Members talked about the challenge of conducting the full range of activities involved in both review and dissemination. Some expressed support for starting slowly and splitting different aspects of the process in critical areas. Related to this issue, members discussed how the current length of funding for a research or model demonstration project is not sufficient for getting to the level needed to validate effective practice. The nature of most projects is such that more time is needed to produce sufficient evidence of effectiveness.
Another issue of concern is motivation among developers. There may not be sufficient motivation for developers to apply for approval. The incentive system for doing so is unclear. In response, some members cited the need for marketing the process through a public relations campaign. They talked about the need for building momentum for the review process and thinking of ways to stimulate momentum. If endorsement for the review gained momentum and prestige in the field, especially among schools and practitioners, this would provide sufficient incentive for involvement.
Members also discussed how the review system would have to reflect inclusiveness when it comes to recognizing legitimate research paradigms. To promote its integrity and clarity of purpose, the system would also need to identify paradigms that are not appropriate. A strongly focussed review process would have to publicize both exemplars and non-exemplars of legitimate paradigms.
Another challenge cited by members is that a review process based on obtaining significant group effects may be antithetical to special education, which has always espoused individual progress. This creates a potential lack of fit between the review process and the cognitive theories on which several interventions are designed. To the extent a review process does not adequately account for the impact of an intervention on students with disabilities individually, it may encourage approval for practices with significant group effects where problems occurring with individual students are masked. In turn, interventions making progress with individual students may be undervalued if they do not combine this with significant group results. In a related vein, measurement issues and the creation of reasonable controls and comparison groups are intensified with a special education population. Although these concerns present considerable challenges, they should not necessarily be viewed as reasons to abort the initiative.
The Work Group strongly supported the view that Federal agencies, such as OSEP, can play a critical role in identifying effective products and practices. They believe Federal agencies support research and evaluation that can validate the effectiveness of different products and practices. According to the Work Group, these agencies should be playing an increased leadership role in advocating for more widespread use of proven products and practices that can be used to improve educational results for children of different ages and with different special needs.
Members of the Work Group, however, expressed concern about the Federal role in dissemination, with particular concern about the cost of widespread dissemination. Developing a comprehensive dissemination system is not perceived by some of the members the group as currently feasible. Target audiences are diverse, and unifying the existing technical assistance and dissemination structure around best practices endorsed by the review process would be challenging. Despite these concerns, other members believe marketing the review process and linking its endorsements to assistance and dissemination is at the core of research to practice, and fundamental to achieving the benefits cited earlier about the review process such as increasing information available to schools as consumers. Still, members of the Work Group are cognizant of the time needed to accomplish this, and the level of financial resources needed is a critical reality.
Overall, Work Group members strongly supported a Federal role in identification of effective products and practices. They unanimously recommended that OSEP continue to explore the feasibility of establishing a review process for IDEA-Part D products and practices and urged the agency to move quickly in response. President Bush and his Secretary of Education have gone on record repeatedly as valuing research-validated practices in education. A review and dissemination process is a timely response to the President's agenda.
Funding for NRCLD is provided by the U.S. Department of Education, Office of Special
Education Programs. Award No. H324U010004. Opinions expressed herein are those of the National
Research Center on Learning Disabilities and do not necessarily represent the position of the U.S.
Department of Education.
© 2007 National Research Center on Learning Disabilities
Last Updated: November 19, 2007
Contact: nrcld@ku.edu