PLSA Program Evaluation
Why evaluate the PSLA?
The purpose of conducting a program evaluation of the PSLA is to identify strengths, weaknesses, and potential areas of improvement in the PSLA curriculum.
Each semester, students' inability to correctly identify source types in LIB 301 surfaces as a priority concern.
While we have made improvements to the LIB 301 curriculum by including the PSLA, we need to measure the impact this program actually has in improving student learning.
Evaluation Questions
1. Is the length of time between Module 3 and the PSLA an impediment to students’ ability to transfer knowledge of source types throughout the semester?
2. Did embedding instructional content from Module 3 throughout the course at specific points of need improve the rate at which students are able to correctly identify source types in their initial PLSA submissions?
3. Do instructors perceive that embedding source type instruction at the point of need (Module 4, 5, 6, & 8) improves the rate at which students are able to correctly identify source types?
4. Do instructors perceive that the inclusion of a reflective journal or a discussion forum where students could describe the characteristics of a particular source type support students’ ability to transfer the concept of source type identification throughout LIB 301?
CIPP Evaluation Model
- Context - What needs to be done
- Input - How should it be done
- Implementation - is it being done
- Product - did it succeed
Using the Logic Model evaluation chart (below), we evaluated the existing inputs, outputs (implementation), and outcomes (products) of our current curriculum. This defined the context in which the PSLA operates.
By evaluating changes to our inputs and outputs we are able to determine the extent to which our program is succeeding.
Data Collection & Analysis Plan
Data was collected on initial PSLA submissions for scholarly and popular articles.
- Initial submissions represent impact of the curriculum on student learning
- Subsequent submissions are not being included as they represent the influence of instructor intervention
- Submissions from students who did not complete both PS3 and PS4 (or the Annotated Bibliography in Fall 2012/Spring 2013) are excluded.
Data was collected from sections 0, 03, and 05 of LIB 301.
- Sections have been offered each semester
- Sections represent a cross sample of DCP students
Data was collected from the following semesters:
- Fall 2012 - First semester LIB 301 offered (Annotated Bibliography)
- Spring 2013 - First semester PSLA implemented (PS 3 and Annotated Bibliography)
- Fall 2014 - First semester PS 4 implemented (PS 3 and PS 4)
- Fall 2014 - Pilot semester for redesigned Module 6 curricular unit (PS 3 and PS 4)
Data Findings
Fall 2012 - Fall 2014
This statistic is representative of students who pass the course, as submissions by students who did not submit all components of the PSLA were removed from our final data set.
Fall 2015 - Pilot of Redesigned Module 6 Curricular Unit
This statistic is representative of students who are projected to pass the course, as submissions by students who did not submit all components of the PSLA were removed from our data set.
Spring 2016 - Second Implementation of Module 6 Curriculum
The statistic is representative of students who actively participated in the PSLA. Students who did not submit an attempt were excluded from the data set.
Recommendations
Implement the Curriculum Redesign across all PSLA Modules
The foundation of the curriculum redesign is to embed Module 3 source type content throughout the course engaging students at their point of need.
Transition alignment from ACRL Standards to ACRL Framework
The redesigned Module 6 curricular unit was specifically aligned with the new Framework. As we implement the full redesign across the other three curricular units, we will transition our alignment from the old ACRL Standards to the new ACRL Framework.
Migrate to new content delivery system (piloted in Module 6)
Students feedback suggests that students prefer the new delivery model over the traditional slideshows used in other course modules.
To continue to meet the learning needs of all our students, we will also offer traditional slideshows with the new Learning Modules.
Continue CIPP Evaluation
Continuing the evaluation process will allow us to measure the full impact of the new design across the full PSLA program.
Additional Recommendations from Round-table Discussion
Add Reflective Writing Component
The PLC will research and design opportunities for reflective writing experiences in LIB 301. The instruction team will review and offer feedback on reflective writing components prior to implementation.