Copy

View this email in your browser

September 2021 Newsletter

Welcome to the September newsletter from the Wing Institute.  Schools are fully in session and learning to solve the myriad of problems related to implementing newly adopted interventions.  This highlights the necessity of thoughtful implementation practices.  In this month’s newsletter, we continue the focus on implementation beginning with a discussion of usable interventions.  Also, in this newsletter you will find reviews of four recently published research articles as well as an original paper by the Wing Institute.

Wing Institute Commentary

Implementation Spotlight: Usable Interventions
 

Every implementation effort begins with exploring options for solving a problem.  This search begins with defining the problem to be solved, identifying for whom the intervention is intended, and the context for implementing it.  Once these questions have been answered, then the review of the available evidence for effective interventions begins.  As is often the case, the characteristics of the participants, settings, and implementers in the research setting does not exactly match the characteristics of the participants, settings, and implementers in the practice settings.  Everyone considering adopting an intervention must make judgements regarding the importance of the differences between the research and practice settings and what adaptations must be made to implement in the practice setting to make the intervention usable.  
 
These considerations are part of creating a contextual fit between the intervention and the setting.  All of this effort is done in the name of identifying an effective intervention that is also usable.  A usable intervention is one that can be implemented within a specific context, making use of the available staff, resources, and materials to implement the intervention.  Interventions that require additional staff, resources, and materials are not as likely to be effective if they are adopted because the infrastructure of the system cannot support the implementation.
 
As is often the case in research settings, the implementers are highly trained and closely supervised by the researchers.  The potential implementers and supervisors in practice settings usually do not have the same training, experience, and supervision.  The challenge for the practice team is to consider what would be required in terms of training and supervision to ensure that the quality of implementation is sufficient to obtain benefits from the implementation effort. One of the factors influencing the decision to adopt an intervention are the costs associated with the training, support, and supervision of the implementers.  To the extent that the costs are judged to be too great the intervention is not usable.
 
An additional consideration in determining if an intervention is usable within a specific context is the time required to implement.  For example, if a reading intervention under consideration for adoption requires 90 minutes per day to achieve benefit, and the school context has allotted only 45 minutes per day, then the adoption committee will either have to revise the school schedule so there is the necessary time to implement with fidelity and achieve benefit, or adapt the intervention to fit within the existing schedule.  To the extent that there is a mismatch between the time recommended to implement and the available implementation time, the intervention is not usable.  Too often the decision is to adopt and reduce the daily implementation time to fit the existing schedule.  This is essentially reducing the dosage.  In making this adjustment, the decision may be made to not implement parts of the curriculum that are, in fact, necessary to achieve benefit.  Making these decisions reduces the effectiveness and perhaps completely neutralizes the effectiveness of the intervention.  The challenge for the research community is to evaluate the components of the intervention that are necessary to achieve benefit.  Practice settings have a great many resource constraints so they must find interventions that are efficient and effective.  There is very little to be gained by developing an intervention that is effective but cannot be implemented with sufficient fidelity within the context of actual practice settings.  
 
Finally, a usable intervention is one that fits the values, beliefs, and experiences of those responsible for implementation.  If an intervention is incompatible with the values, beliefs, and experiences, it is less likely to be adopted and used.  For example, if a classroom management intervention package includes praising students but the implementers have beliefs that praising students will lessen their intrinsic motivation to perform, then the quality of the implementation is likely to suffer.  Implementers are more likely to implement those components of the classroom management package that are consistent with their values, beliefs, and experiences.
 
This is a brief discussion of usable interventions.  From this discussion, adopting an intervention that is effective and usable necessitates the consideration of a large number of factors requiring thoughtful judgement.  Since those adopting the intervention are not always the people implementing it, it is important to gain input from those who will be implementing the intervention.  The acceptability of the intervention to those implementing it can increase the probability of high-quality implementation.  This is even more likely if the implementers had a role in selecting the intervention to adopt.  To enhance the adoption of a usable intervention, it is important for the site to create an implementation team to attend to all of the issues related to adoption and high-quality implementation of an intervention.  We will be discussing implementation teams more fully in a future newsletter. - Ronnie Detrich -

Wing Institute Original Work

How effective is principal evaluation?
 

In this month’s newsletter, we have one original paper from the Wing Institute by Jan Donley reviewing the state of the field of principal evaluation (https://www.winginstitute.org/quality-leadership-principal-evaluation).  Principals play an important role but indirect role in influencing student outcomes.  Given their importance in school improvement, it is important to know that these leaders are performing well.  As described in this paper, it is clear that there is not an agreed upon approach to evaluating principal performance.  To date, there is not agreement about what aspects of principal performance should be evaluated.  In some instances, principal evaluation has not included student outcomes as one of the components of the evaluation system.  In these instances, the emphasis has been more on process measures.  Is the principal doing the right thing as defined by the evaluation system?  The effects of what the principal is doing are not evaluated.  More recently, there has been a movement toward more results-based accountability systems.  Even though principal accountability is increasingly gaining favor, there are considerable differences within states and across states as to how to measure principal effectiveness.
 
In terms of implementation science, principal evaluation is important because leadership is one of the core implementation drivers in the Active Implementation Frameworks (Fixsen, Blase, & Van Dyke, 2019).  In addition, principals are a sub-component of the implementation driver, Organizational Drivers.  Principals play a significant role in clarifying the values and vision related to adopting an innovation, establishing the processes for implementing the innovation, and insuring that everyone is implementing the steps necessary to achieve full implementation of the innovation.  Once the innovation is fully implemented, it is the principal’s responsibility to maintain high quality implementation of the innovation. Read More
 
Citation
 
Donley, J., Detrich, R., States, J., & Keyworth, (2021). Principal Evaluation Oakland, CA: The Wing Institute. https://www.winginstitute.org/quality-leadership-principal-evaluation

News

How are Effective Programs Disseminated and Scaled?
 

One of the persistent problems in education and other human service disciplines is the research to practice gap (some would call it a chasm).  In an effort to disseminate an effective bullying program (Free2B), Leff and colleagues applied the logic of Diffusion of Innovations (Rogers, 2003).  This logic proposes that innovations are more likely to be adopted if the innovation has (1) a relative advantage over current practices, (2) is easy to use, (3) is compatible with the values, beliefs, experiences of the users, (4) can be implemented on a trial basis before large scale implementation, and (5) the opportunity for others to observe implementation and the effects of implementation.  Leff and colleagues followed these recommendations in implementing the Free2B anti-bullying program in 40 middle schools.  The authors concluded that it was an attractive alternative to many anti-bullying programs because the intervention was delivered in a school assembly that schools were already providing, so it required no additional time allocation.  Additionally, the video format made the delivery very easy compared to school-wide programs that are more time and resource intensive.  The students reported that it addressed important topics. Prior to implementation, Leff and colleagues presented pilot data to key stakeholders at the state’s Office of Safe Schools who were able to leverage adoption by schools across the state.  In addition to measuring adoption they also measured the impact on students and founds positive effects across all measures. Read More

 
Citation: Leff, S. S., Waasdorp, T. E., Paskewich, B. S., & Winston, F. K. (2020). Scaling and Disseminating Brief Bullying Prevention Programming: Strengths, Challenges, and Considerations. School Psychology Review, 1-15.

 

What are Effective Classroom Behavior Management Practices to Reduce Disruptive Behavior of Middle School Students?
 

Middle school students are more likely than elementary or high school students to be disruptive (Erikson & Gresham, 2019).  This presents difficult problems for classroom teachers trying to provide instruction and maintain order in the classroom.  It has been estimated that 2.5 hours per week are lost to disruptive behavior each week (Education Advisory Board, 2019).  This level of disruptive also contributes to teacher turnover with 39% of teachers reporting that disruptive behavior was one of the primary reasons for resigning (Bettini et al., 2020).  To address challenges presented by middle school students, Alperin and colleagues (2021) completed a systematic review to identify programs that had a positive effect on disruptive behavior and the characteristics of those programs.  They identified 51 studies that met their inclusion criteria.  Of those 51 studies, 40 of them specified the function of behavior (gain attention or escape demands) that the program addressed; 16 of the studies included a home-based component with 7 of the studies providing parent training; 22 of the interventions had a manual guiding the implementation of the intervention; and encouragingly, 42 of the studies assessed intervention implementation.  Effect sizes for seven of the studies were computed for intervention that involved class-wide intervention strategies.  The effects ranged from small to large across the studies.  Fourteen of the studies evaluated skill acquisition for small groups or individuals and the effect sizes again ranged from small to large.  Seven of the studies evaluated reinforcement strategies for reinforcement-based interventions for small groups or individuals and reported effect sizes that ranged from small to large.  Two studies evaluated interventions for escape from demands for small groups or individuals.  Both of these studies reported large effect sizes.  The data from this study are important as they can provide guidance to educators seeking to reduce disruptive behavior of middle school students.  Ultimately, the educators will have to consider the contextual fit for each of these interventions for the settings in which they work.  This study narrows the range of options to those that have some demonstrated level of effectiveness rather than leaving the educator to choose from all available options. Read More
 
 
Citation: Alperin, A., Reddy, L. A., Glover, T. A., Bronstein, B., Wiggs, N. B., & Dudek, C. M. (2021). School-Based Interventions for Middle School Students With Disruptive Behaviors: A Systematic Review of Components and Methodology. School Psychology Review, 1-26

Can Technology Improve the Fidelity of Implementation?
 

Fundamental to any intervention outcome is the fidelity of implementation of the intervention.  The ultimate goal of implementation science is to assure that innovations are implemented well enough for students to benefit.  Failure to implement well can minimize the effectiveness of even the most powerful intervention.  One of the challenges involved in insuring high quality implementation is that most approaches are resource intensive and often are not seen as feasible in school settings even though failure to achieve adequate implementation fidelity may result in a very poor benefit to cost ratio.  One possible alternative is to utilize technology to reduce the resource demands.  Fallon and colleagues (2021) conducted a systematic review to evaluate the effectiveness of technology-based supports to promote implementation fidelity.  For the purposes of this review, “electronically delivered implementation supports (EDIS) was support delivered to an implementer electronically (e.g., via email, social media, video conferencing) for the purpose of improving educators’ implementation fidelity of a student intervention.”  Fifteen studies met inclusion criteria and were judged to be of sufficient methodological rigor to warrant further analysis.  The electronically delivered implementation supports ranged from video modeling, electronically delivered performance feedback, emailed intervention prompts, coaching via video conference, and online training modules.  All of the studies were based on single participant designs.  Since there are no agreed upon methods for calculating effect sizes for single participant designs, the authors calculated several different effect sizes (Tau-U, Standard Mean Difference, Hedges’ g, and a variation of Hedges’ g.  In most of the studies, the effect sizes ranged from moderate to large regardless of the calculation method used.  After completing the review, the authors provided guidance to educators about when to use the various methods of electronically delivered implementation supports.  This article is a valuable resource to any educator considering implementing an intervention but is concerned about the resource requirements required for insuring high quality implementation.  This article suggests that technology-based alternatives can be effective in supporting implementation and may reduce the overall demands on resources. Read More
 
Citation: Fallon, L. M., Collier-Meek, M. A., Famolare, G. M., DeFouw, E. R., & Gould, K. M. (2020). Electronically Delivered Support to Promote Intervention Implementation Fidelity: A Research Synthesis. School Psychology Review, 1-16.

Can Real Time Performance Feedback Improve Instructional Practice?
 

Performance feedback, including real time performance feedback, has been implemented across many contexts including educational settings.  Sinclair, Gesel, LeJeune, and Lemons (2020) evaluated 32 studies that met their inclusion criteria to determine the effectiveness of real-time performance feedback in improving the instructional practices of educators. Interestingly, all but two of the studies utilized bug-in-the-ear technology in which the teacher wore an ear piece and a coach provided immediate feedback to the teacher as implementation of an intervention was occurring.  In this review, teachers implementing both academic instruction and behavior management interventions were considered.  Based on this review, the authors concluded that real time performance feedback was an evidence-based practice and could be used as a method for improving the performance of pre-service teachers, teachers, and paraprofessionals.  The bug-in-the-ear technology offers several advantages.  First, it is less intrusive than other methods for providing real time feedback.  With current technology, the coach can view implementation in the classroom without being in the classroom.  Technologies such as Go Pro and Swivl have sufficient flexibility for the coach to get a good sample of what is happening in the classroom.  A second advantage is that because the feedback is immediate it is more likely to be effective compared to when the feedback is delayed.  Finally, the bug-in-the-ear technology is time saving because the feedback is delivered in real-time.  Brief follow-up meetings to discuss issues related to the intervention can be scheduled.  Interestingly, bug-in-the-ear technology has been around for decades but has been under-utilized in educational settings.  An analysis of the barriers to utilizing this technology more broadly is warranted.  The potential for impact on implementation of an intervention is significant. Read More
 
Citation: Sinclair, A. C., Gesel, S. A., LeJeune, L. M., & Lemons, C. J. (2020).  A review of the evidence for real-time performance feedback to improve instructional practice. The Journal of Special Education54(2), 90-100.

References

References

 
  • Bettini, E., Gilmour, A. F., Williams, T. O., & Billingsley, B.(2020). Predicting special and general educators’ intent to continue teaching using conservation of resources theory. Exceptional Children, 86(3), 310–329. https://doi.org/10.1177/ 0014402919870464

 

  • Education Advisory Board. (2019). Breaking bad behavior: The rise of classroom disruptions in early grades and how districts are responding. http://pages.eab.com/rs/732-GKV-655/images/ BreakingBadBehaviorStudy.pdf

 

  • Erickson, M., & Gresham, F. (2019). Measuring teachers’ perceptions of student behavior using the systematic screening for behavior disorders in middle school students. Journal of Emotional and Behavioral Disorders, 27(2), 119–128. https:// doi.org/10.1177/1063426618763110

 

  • Fixsen, D. L., Blase, K., & Van Dyke, M. K. (2019). Implementation practice & science. Active Implementation Research Network.

 
  • Rogers, E. M. (1962). Diffusion of innovations. New York: Free Press of Glencoe.

Twitter
Facebook
Website
Copyright © 2021 Wing Institute, All rights reserved.


Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list

Email Marketing Powered by Mailchimp