PLSA Program Evaluation

Why evaluate the PSLA?

The purpose of conducting a program evaluation of the PSLA is to identify strengths, weaknesses, and potential areas of improvement in the PSLA curriculum.

Each semester, students' inability to correctly identify source types in LIB 301 surfaces as a priority concern.

While we have made improvements to the LIB 301 curriculum by including the PSLA, we need to measure the impact this program actually has in improving student learning.

Evaluation Questions

1. Is the length of time between Module 3 and the PSLA an impediment to students’ ability to transfer knowledge of source types throughout the semester?

2. Did embedding instructional content from Module 3 throughout the course at specific points of need improve the rate at which students are able to correctly identify source types in their initial PLSA submissions?

3. Do instructors perceive that embedding source type instruction at the point of need (Module 4, 5, 6, & 8) improves the rate at which students are able to correctly identify source types?

4. Do instructors perceive that the inclusion of a reflective journal or a discussion forum where students could describe the characteristics of a particular source type support students’ ability to transfer the concept of source type identification throughout LIB 301?

CIPP Evaluation Model

A four stage, cyclical evaluation model focusing on continual program improvement.

  1. Context - What needs to be done
  2. Input - How should it be done
  3. Implementation - is it being done
  4. Product - did it succeed

Using the Logic Model evaluation chart (below), we evaluated the existing inputs, outputs (implementation), and outcomes (products) of our current curriculum. This defined the context in which the PSLA operates.

By evaluating changes to our inputs and outputs we are able to determine the extent to which our program is succeeding.

Big image

Data Collection & Analysis Plan

Data was collected on initial PSLA submissions for scholarly and popular articles.

  • Initial submissions represent impact of the curriculum on student learning
  • Subsequent submissions are not being included as they represent the influence of instructor intervention
  • Submissions from students who did not complete both PS3 and PS4 (or the Annotated Bibliography in Fall 2012/Spring 2013) are excluded.

Data was collected from sections 0, 03, and 05 of LIB 301.

  • Sections have been offered each semester
  • Sections represent a cross sample of DCP students

Data was collected from the following semesters:

  • Fall 2012 - First semester LIB 301 offered (Annotated Bibliography)
  • Spring 2013 - First semester PSLA implemented (PS 3 and Annotated Bibliography)
  • Fall 2014 - First semester PS 4 implemented (PS 3 and PS 4)
  • Fall 2014 - Pilot semester for redesigned Module 6 curricular unit (PS 3 and PS 4)

Data Findings

Fall 2012 - Fall 2014

On average 18% of initial attempts at choosing the correct source type for scholarly and popular articles is incorrect each semester.

This statistic is representative of students who pass the course, as submissions by students who did not submit all components of the PSLA were removed from our final data set.
Big image

Fall 2015 - Pilot of Redesigned Module 6 Curricular Unit

For Fall 2015, 6% of initial attempts at choosing the correct source type for scholarly and popular articles were incorrect after piloting the new curriculum.

This statistic is representative of students who are projected to pass the course, as submissions by students who did not submit all components of the PSLA were removed from our data set.
Big image

Spring 2016 - Second Implementation of Module 6 Curriculum

For Spring 2016, 10% of initial attempts at choosing the correct source type for scholarly and non-scholarly articles were incorrect in the second semester of implementing the newly designed curriculum.

The statistic is representative of students who actively participated in the PSLA. Students who did not submit an attempt were excluded from the data set.

Big image


Implement the Curriculum Redesign across all PSLA Modules

Based on data, the curriculum redesign piloted in Module 6 has reduced the percentage of incorrect source type submissions.

The foundation of the curriculum redesign is to embed Module 3 source type content throughout the course engaging students at their point of need.

Transition alignment from ACRL Standards to ACRL Framework

LIB 301 is currently aligned with the ACRL Standards for Information Literacy. In 2015, ACRL developed their new Framework for Higher Education.

The redesigned Module 6 curricular unit was specifically aligned with the new Framework. As we implement the full redesign across the other three curricular units, we will transition our alignment from the old ACRL Standards to the new ACRL Framework.

Migrate to new content delivery system (piloted in Module 6)

In implementing the redesign, we developed a more engaging course delivery model using the Blackboard Learning Module tool, info-graphics, and instructional videos (Ainsworth, 2010).

Students feedback suggests that students prefer the new delivery model over the traditional slideshows used in other course modules.

To continue to meet the learning needs of all our students, we will also offer traditional slideshows with the new Learning Modules.

Big image

Continue CIPP Evaluation

The PLC recommends continuing the CIPP evaluation through the first year after the full implementation of the curriculum redesign.

Continuing the evaluation process will allow us to measure the full impact of the new design across the full PSLA program.

Additional Recommendations from Round-table Discussion

Add Reflective Writing Component

Based on emerging trends in library research and at the University, the library instruction team recommends incorporating reflective writing assessments in LIB 301.

The PLC will research and design opportunities for reflective writing experiences in LIB 301. The instruction team will review and offer feedback on reflective writing components prior to implementation.

Evaluation of Reflective Writing Component

The library instruction team and PLC recommend conducting a CIPP evaluation of the reflective writing component as it is incorporated in LIB 301.