Spreading Community Accelerators through Learning and Evaluation Findings from the SCALE Formative Evaluation January 15, January 15, PDF

Please download to get full document.

View again

of 295
0 views
PDF
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Document Description
Spreading Community Accelerators through Learning and Evaluation Findings from the SCALE Formative Evaluation January 15, January 15, 2017 Principal Investigator: Abraham Wandersman Project Director:
Document Share
Document Transcript
Spreading Community Accelerators through Learning and Evaluation Findings from the SCALE Formative Evaluation January 15, January 15, 2017 Principal Investigator: Abraham Wandersman Project Director: Jonathan P. Scaccia The SCALE Evaluation Team Michelle Abraczinskas, Kassandra Alia, Brittany Cook, Gareth Parry, Rohit Ramaswamy, Amy Reid, Jonathan Scaccia, Victoria Scott, Abraham Wandersman With contributions from: Holly Hayes, Morgen Palfrey, Marie Lina Excellent April 27, 2017 Contents Acknowledgements... 1 Spreading Community Accelerators for Learning and Evaluation... 2 Formative Evaluation Executive Summary... 2 SCALE Processes... 2 SCALE Drivers... 3 SCALE Formative Evaluation... 3 Some Key Evaluation Results... 3 How Community Capability was Developed... 3 What Communities Accomplished... 4 Progress Developing a System of Spread... 6 Recommendations... 6 Afterward... 7 Chapter 1. Introduction... 8 Evaluation Overview...11 Structure of Report...12 Chapter 2. Synthesis Evaluation Methods...14 Readiness...14 Methods to Assess Readiness in SCALE...14 Chapter 3. Overview of Prior Evaluation Reports...16 Chapter 4. SCALE Support System...19 Section Introduction...21 Evaluation Approach...23 Support Provided to Communities...23 Tools...24 Training...28 Coaching/Technical Assistance...35 Quality Assurance/Quality Improvement...46 Support Provided to SCALE Coaches...49 Factors Impacting Delivery of the Coach Role...49 Support Received by Coaches...50 Support Provided to the Implementation Team...51 Chapter 5. Community Delivery System Summary...54 Community Readiness...59 Comparisons using the Readiness Innovation-Configuration Map (RICM)...63 SCALE Application Reanalysis...63 Readiness Findings from Case Studies...68 Progress on Drivers of Change...68 Develop leadership capability to transform and improve within and across communities 69 Knowledge and Skills for SCALE tools and methods...74 Sustainability...80 Creating vibrant relationships and functional networks...84 Evidence of team functioning...87 Extent Aims in the Driver Diagram are based on a Vision of Change for the...90 Extent goals have been supported by data...90 Evidence of peer to peer connections...92 Create and improve the inter-community spread system Use of Technology to facilitate improvement and Spread...94 Spread of Ideas...99 Biggest threats or challenges to success...99 Use of Formative Evaluation Tools and Results...99 Chapter 6. Summary of Deep Dive Case Studies Background SCALE Experiences Chapter 7. Pathways to Pacesetter (P2P) Initiative Introduction Evaluation Approach Community Delivery System Supports Delivered to Communities Tools Training Technical Assistance/Coaching Synthesis and Translation System Facilitators (SCALE Communities) and Professional Coaches Delivery of Facilitation/Coaching Supports Delivered to Facilitators/Coaches Chapter 8. Conclusions and Recommendations SCALE Findings P2P Findings Reflections and Recommendations Appendix A. Formative Evaluation Plan Overview Appendix B. Tools and methods used in the SCALE evaluation Appendix C. Synthesis Methods Appendix D. Deep Dive Case Studies Appendix E. Definitions of Readiness (R=MC2) in SCALE Appendix F. Plan-Do-Study-Act (PDSA) Workgroup Log Appendix G. Community Readiness Interviews Appendix H. Aggregated CHILA session ratings Appendix I. SCALE Evaluation Team SCALE FORMATIVE EVALUATION 1 Acknowledgements The work documented in this report represents a deep, rich, and meaningful collaboration of multiple stakeholders including representatives from: the SCALE Implementation Team (Institute for Healthcare Improvement, Community Solutions, Communities Joined in Action, and the Collaborative Health Network), the Robert Wood Johnson Foundation, and SCALE communities. We are deeply indebted to the leadership and openness of Soma Stout (SCALE P.I.) in setting an ambitious agenda and providing ideas and enthusiasm for a collaborative approach to evaluation and evaluation use. We also acknowledge the important contributions of: Institute for Healthcare Improvement: Marie Schall, Aleya Martin, Niñon Lewis, Hayley Browdy, Elizabeth McDermott, Laura Howell, Laura Baker, Correy Denihan, Marianne McPherson, Mark Bradshaw, Kush Badshah, Tam Duong Community Solutions: Paul Howard, Garen Nigon, Karina Mueller, Jana Pohorelsky Communities Joined in Action: Laura Brennan, Tanisa Adimu, Beverly Tyler Network for Regional Healthcare Improvement: Sarah Woosley, Margy Weinbar, Collaborative Health Network: Janhavi Kirtane, Emily Dietsch Independent: Ziva Mann, Catherine Craig SCALE communities: Zachary Desmond, Fran Mullin, Teal VanLanen, Drew Martin, Shemekka Coleman, Lena Hatchett, Becky Henry, Kate Ebersole, Megan Albertson Special contributions to this report were made by Kassandra Alia, Jonathan Scaccia, Amy Reid, Michelle Abraczinskas, and Morgen Palfrey. The SCALE evaluation was assisted by contributions from Amrita Dasgupta, Ariel Domlyn, Samantha Hartley, Steven Taylor, Melek Yildiz Spinel, Jason Reed, Stacy Wright, and Sarah English. Finally, we acknowledge the insightful support and guidance provided by colleagues at RWJF, including Laura Leviton (program evaluation project office), Robin Mockenhaupt, and Hilary Heishman. The SCALE Evaluation Team April 2017 SCALE FORMATIVE EVALUATION 2 Spreading Community Accelerators for Learning and Evaluation Formative Evaluation Executive Summary The first phase of Spreading Community Accelerators through Learning and Evaluation (SCALE) was a Robert Wood Johnson Foundation (RWJF)-funded initiative led by the Institute for Healthcare Improvement (IHI) and key community partners 1 ; it ran from January 2015 to January SCALE s aim was to build the readiness and capability of 24 community coalitions to improve health, well-being, and equity (see Figure E.1 for the location of the communities). Figure E.1 SCALE communities SCALE Processes The SCALE initiative was designed to test new approaches to accelerate transformational community change by building community capability to learn how to improve and to spread improvement ideas. These approaches, aligned with RWJF s vision of promoting a Culture of Health for all people by changing social, policy, financial, and community-level structures, included: 1. Focus on Transformation emphasizes building skills for how people relate to themselves, relate to one another, and apply techniques for improvement. 2. Failing Forward recognizes that mistakes and challenges will happen along the community health improvement journey, but that these failures represent critical opportunities to learn and accumulate knowledge. 3. Co-Design by having SCALE intervention activities collaboratively developed with meaningful input from the Evaluation Team, RWJF, and the SCALE communities. The codesign spirit carried over to the evaluation itself to ensure that research methods and techniques aligned with the needs of all stakeholders. 4. Including community members with lived experience in order to ensure that community voices were meaningfully represented in program design and implementation. 5. Focus on equity emphasizes the community capability to recognize social and health disparities between groups within their communities and to prioritize improvement efforts that address inequity. 1 Key Partners included: Community Solutions, Communities Joined in Action, the Collaborative Health Network SCALE FORMATIVE EVALUATION 3 SCALE Drivers SCALE was intended to accelerate improvement in communities that had been identified during an application process as having high readiness (being willing and able) to improve. The Theory of Change to accomplish this consisted of the following drivers: 1) Developing leadership capability to transform and improve within and across communities, 2) Creating vibrant relationships and functional networks between communities that accelerate trust, learning and achievement of a shared goal 3) Creating and improving an inter-community system to spread good and promising ideas. To build capability on these drivers, communities enrolled in SCALE attended four face-toface training sessions, called Community Health Improvement Leadership Academies (CHILAs), that were complemented by an elaborate support system consisting of peer-to-peer and expert coaching, webinars, resources and tools. Additionally, a supplementary program, Pathway to Pacesetter (P2P), was created for lower readiness communities that were not part of the original SCALE cohort. This program aimed to see whether SCALE approaches could be adapted to support communities at earlier stages of readiness. Select SCALE resources were available at no direct cost to many P2P communities. SCALE Formative Evaluation A collaborative formative evaluation approach was used to foster learning and adaptation during the implementation of SCALE. We implemented a mixed-methods, multilevel evaluation design to monitor: a) how support was provided to communities and b) how communities progressed on their community health improvement journey (Chapters 4-7; Appendices A-C). A key objective of the evaluation was to emphasize evaluation use by having evaluation findings be available in a timely, understandable, and actionable way to help improve the provision of support for enhancing community activities. Evaluation results were gathered continuously and collaboratively analyzed by Evaluation, Implementation, Community, and RWJF stakeholders, with program changes made based on the findings. Some Key Evaluation Results How Community Capability was Developed SCALE provided communities with a comprehensive support system involving training, technical assistance, tools, and quality improvement/quality assurance. These supports were co-designed by stakeholders in order to facilitate transformational change along the three primary drivers (Chapter 4). Community Health Improvement Leadership Academies (CHILAS) were in-person, multi-day training sessions designed to foster relationships between communities, build skills in community health improvement leadership, and facilitate learning across communities. Across all four CHILAS, communities highly rated the overall quality and content. The CHILAs were pivotal in building trust and relationships within and between communities, as indicated by qualitative reports that persisted across CHILAs and sense of community measures On CHILAs I also loved the space of open and honest dialogue that was created.being able to discuss and debrief with my team- my home team and at times others whether peer community team, or just small groups. Being able to learn and discuss how to apply right away is valuable. HIPMC (pacesetter community) SCALE FORMATIVE EVALUATION 4 made post-chilas. They were highly effective at creating excitement and motivation for implementing improvements. Communities reported statistically significant gains in prepost surveys for nearly all skills that were provided at CHILAs. Feedback from each CHILA led to co-designed changes in subsequent CHILAs. Coaching was provided directly by improvement experts to communities and in a group format (Peer Community Teams). Both formats were viewed positively with the direct coaching getting higher scores. Qualitative results revealed variability in coaching content and styles, and may reflect some of the challenges that coaches faced at times with clarity of deliverables and timelines. Capability of the coaches improved over time as gaps in training and support identified through interviews with coaches conducted during the formative evaluation were addressed. Overall, however, findings suggested that coaches were effective at building connections between communities. Other support methods like webinars, online courses, and a community health-specific social media platform were also used to facilitate improvement and spread of ideas. These were useful but not as much as the coaching, and attendance in the webinars decreased over time (from about 60 attendees in mid-2015 to about 15 by the end of SCALE). The mentor community role was not seen as an effective means of support. What Communities Accomplished A large and complex initiative like SCALE demands a lot from communities; and the SCALE communities, despite feeling overwhelmed at times, rose to the occasion. Since t he design of the support system was taking place at the same time as implementation, there were occasions where communities, like the coaches, felt the need for a roadmap for understanding milestones and expectations. But overall, communities reported sign ificant progress on building improvement capability. As would be expected in a 24-community study, there was considerable variability in progress. As SCALE evolved, the Community of Solutions skills emerged as a way to organize the key capabilities that communities were expected to acquire. This framework, shown in (Figure E.2) emphasized skill building in five areas: On Coaching [Our coach] is always there for us and is consistently giving me ideas for who in our peer community I should contact with questions or ideas I have I d say all the credit for helping the quality of our community health plan would go directly to her. ---BCCHC (pacesetter community) Leading from within skills included the development of a common, learning-oriented value system with norms and principles of openness, trust and inclusion. The Habits of the Heart session presented in the first CHILA was rated as the most valuable session across all four CHILAs. Figure E.2 Community of Solutions Skills Communities adopted the language and SCALE FORMATIVE EVALUATION 5 principles of failing forward and mentioned them frequently. The evaluation results showed that these concepts resonated and were appreciated by communities. Leading together included how communities worked together to build the trust and governance processes to share resources and assets to promote improvement. By the end of SCALE, communities reported developed meaningful relationships with each other. Leading for outcomes pertained to how communities adopted and implemented methods to locally address complex problems. There was a major focus on assessing and using readiness (motivation, innovation-specific capacity and general capacity; Appendix E) to support progress toward improvement and equity goals. Using a comprehensive, mixedmethods approach, communities reported high readiness for engaging in SCALE methods to work toward outcomes. Over the course of SCALE, communities showed significant gains in their reported readiness to implement community health improvement projects. Switch Thinking was a unique set of concepts to be used in healthy community coalitions and it was used by many communities. While many communities made initial progress on applying quality improvement methods to community health issues, specifically by beginning to use Plan-Do-Study-Act (PDSA) cycles, there were significant challenges to fully implementing and documenting outputs of this work. A highly-structured quality improvement method (the Action Lab) was introduced in May 2016 to help accelerate progress. While only five communities fully completed their Action Lab by the time core SCALE funding ended in January 2017, the communities that completed it reported accomplishing their equity-focused aims. The progression from readiness to demonstrated use takes time and these results are not unreasonable for a two-year project. Leading for equity included how communities addressed social and health disparities between groups in their communities. Feedback from communities during CHILA 2 directly contributed to greater emphasis on how to improve community-level equity. All communities reported progress involving people with lived experience as community champions in their coalitions. There was variability in the extent to which these champions were engaged as well as in the roles they played that evolved over time. Leading for sustainability included how communities identified pre-existing resources, fostered new resources (people and capacities), and built a culture of change within their community. While communities reported that they had acquired skills related to developing beneficial relationships and maintaining engagement many also expressed concerns about maintaining their progress and momentum due to external constraints such as resources or funding. Indicative of their ongoing commitment to sustainability, the vast majority of communities (~88%) made a formal commitment to continue working together as part of future waves of SCALE SCALE FORMATIVE EVALUATION 6 Progress Developing a System of Spread Bright Spots are programs, initiatives, organizations, communities, or individuals who are achieving exceptional results. This concept was introduced early in SCALE and communities found the concept extremely useful. About 70% of communities identified Bright Spots relevant to their work, traveled to learn from them, and began to adopt and adapt what they had learned. P2P findings provided initial data on how communities can serve as peer coaches. P2P data also showed the importance of face-toface interactions in promoting relationships and spread between communities. On Applying Bright Spot Learning We had visited the [Community Center]. This place was in shambles (so to speak) and did not appeal to the eye; however, this did not matter. It taught us that we don't have to have the perfect place or even have everything ready. What matters is building the leadership capacity of our youth. [We are also] learning from OK City Public Health Department; we have built off of their community health worker model to develop a community nurse activator role at our community wellness center. The service is free to community members --- Live Algoma (pacesetter community) Recommendations While we detail SCALE-specific recommendations that can be carried forward in Chapter 8, our evaluation of SCALE has illuminated findings which are applicable to communities seeking transformational change using improvement methods. Content and Relationships. Our results show that the multidisciplinary capability-building approach undertaken during CHILAs that jointly emphasized the tools and methods of improvement and personal values, relationships, and trust are critical in building motivation, engagement around the improvement process, and perseverance when the going is tough. Both CHILA content and relationship-building processes are likely to be an indispensable part of any future community capability building initiative. Ongoing support through formative evaluation is crucial because community transformation is complex and dynamic. Transformational change in communities is a complex and non-linear process. Therefore, clarifying and documenting what is learned along the way is critical to the achievement of progress. This includes capturing data about what is being improved and data on growth in motivation and skills. Assessing change in community processes and systems should be a key evaluation component that is used in an ongoing fashion to foster positive and open relationships among all stakeholders. To that end, clear processes are needed to facilitate the systematic integration of evaluation findings in ways that can strengthen implementation and support. Further, learning through self-evaluation is a part of good improvement practice and should be a goal for each community. Measurement. Objective output or outcome measures need to be prioritized when using improvement methods. However, data on such measures cannot be obtained as easily and as often in community settings compared to the industry or facility contexts where these methods were first developed. Clear guidelines on how to use multiple sources of data in addition to output measures (e.g., local process data, or qualitative assessments) to practically assess improvement should be created. In add
Similar documents
View more...
Search Related
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks