LIAS Effectiveness Study

By Sam Piha

Sam Piha

The Learning in Afterschool & Summer (LIAS) project was launched in 2010. In the summer of 2015, the LIAS staff conducted a LIAS Effectiveness Study to gauge our impact on the field and how best to further the project goals. The objectives of this LIAS Effectiveness Study were to:

• Learn how effective the LIAS campaign was in reaching and impacting expanded learning stakeholders and program leaders;
• Learn how effective the individual LIAS strategies were;
• Improve the LIAS campaign going forward; and
• Share with the larger field any learnings that may be helpful to future efforts to improve expanded learning program quality.

We collected 91 surveys completed by Regional Leads, those who oversee or

work directly with youth, or provide technical assistance. We also conducted follow up interviews with 20 afterschool leaders across the state. In terms of geography, most of the respondents are located in Regions 3, 4, 9, and 11. These four regions contain 57% of the total California ASES and 21st CCLC programs. (Source: California Afterschool Network). 

Below is a summary of findings from our study. To view the study Executive Summary, click here. To view the full report, click here. 

LIAS effectiveness: 

• The LIAS project has successfully raised awareness of the LIAS principles across the state.
• On average, 74% of respondents reported their contact with LIAS materials. Of those, 67% on average, rated their experience as a 4 or 5 on a 1-5 scale of effectiveness, with 1 being the lowest.
• On average, only 30% of respondents reported an involvement with LIAS support activities. Of those, on average, 79% rated their experience as a 4 or 5 on a 1-5 scale of effectiveness, with 1 being the lowest.
• Of those surveyed, over 84% reported that the LIAS materials and supports impacted their programs. They reported that the LIAS project either changed the way they talk about their programs (49%) or effected their program design (35%).

LIAS Strengths: Respondents reported that the language of the LIAS learning principles are clear and understandable (nearly 84%) and the LIAS learning principles can easily be applied to program design and improvement (nearly 69%). 

LIAS Weaknesses: A fair number of respondents (22%) thought that the LIAS project can do more outreach to educate practitioners about the LIAS learning principles. Nearly 46% of respondents replied, “No comment”. 

Challenges and barriers to program improvement: Respondents named staff turnover (92%), insufficient funding (86%), and insufficient time (80%) as the largest barrier or challenge. 

Takeaways
Objective 1: Learn how effective the LIAS campaign was in reaching and impacting expanded learning stakeholders and program leaders

• LIAS outreach – The LIAS project has been very successful in spreading the word on the LIAS principles. The newly designed website and LIAS blog have been well received.  

• LIAS impact – The LIAS project reportedly has impacted many by changing the way they talk with others about their program activities and/or changing actual day-to-day practice. Program leaders want more guidance on how to apply LIAS and character building strategies into practice. 

Objective 2: Learn how effective the individual LIAS strategies were

• LIAS materials – The LIAS postcard has been widely distributed across the state and well received by program leaders. However, some of the LIAS materials such as the webinar, are not reaching a large audience. 

• LIAS training opportunities – The LIAS orientations and introductory trainings are an effective educational outreach strategy. However, they do not go far enough in changing practice at the program level. The LIAS learning communities have been well received and judged as effective ways to impact program quality. However, they are not widely available due to issues of cost, commitment, and geography. 

Objective 3: Improve the LIAS campaign going forward

• Diversify audience – It would be useful if the LIAS project could reach out to diverse audiences that impact young people’s learning. These additional audiences include youth program providers outside of the “afterschool circle”, principals and educational administrators, instructors in higher education, museums, libraries, and Parks & Recreation departments. 

• Access to online tools – Because of issues of cost and distance, and because young workers grew up in the digital age, we have been advised to ensure that materials and tools include short videos of actual program practice, are accessible via the Internet, be multimedia, and viewable on smartphones and tablets. 

• Integrate new concepts - materials should include the new state Quality Standards. They should also integrate 21st century learning skills, social emotional learning, building character skills, non-cognitive skills, growth mindsets, and foundations for young adult success. There is also the demand that programs respond to the new state Quality Standards for Expanded Learning Programs and the Common Core State Standards. While it is true that some of these are overlapping, they still present an overwhelming set of “lists”. The LIAS project has been advised to create an approach that integrate these “lists” and the Quality Standards.

Objective 4: Learnings that may be helpful to future efforts to improve expanded learning program quality

• Cost as a barrier - Due to rising labor costs and stagnant grant awards, programs have cut the amount of resources for professional development and program tools. For all of us that work to improve program quality, we have been advised to consider the issue of cost – program materials and training - and even offer stipends to training participants in exchange for a commitment to attendance and program improvement efforts. 

• Other barriers to program improvement – Any effort to improve program quality requires educational outreach that is ongoing due to staff turnover. In addition to continuing these efforts, we have been encouraged to target program leaders, who have longer tenures than frontline staff.