theBookmark Feature:
Evaluation:
Transformation as well as Accountability
By
Dianne Oberg
Evaluation
of School Library Education
Learning
to be a teacher-librarian is really a lifelong process of learning.
As part of that lifelong process, formal coursework has a role to
play in developing and maintaining professional competence, for
initial professional learning and for continuing professional
learning. Every program coordinator or instructor involved in
professional education must at some time wonder about the quality of
the coursework that he or she has developed and delivered. Have the
courses made any difference in the way that students think about and
carry out their professional work? Does learning through coursework
lead to proficient professional practice? Can leadership be developed
through an academic program? How can a program keep current with the
changes in schooling and in society?
In
the Teacher-Librarianship by Distance Learning (TL-DL) program at the
University of Alberta (and in its earlier face-to-face iterations),
many different kinds of evaluation strategies have been used to
assess the quality, content, and impact of the program and courses.
An important insight from my doctoral work was that program
evaluation can serve more than accountability purposes: it can help
those involved to understand the program more deeply, to become more
committed to the program, and to enable transformation of the program
(Oberg, 1994). In this article, I will outline some of the program
evaluation strategies we have used. I will try to show how evaluation
has helped to transform, in small ways and in large ways, the school
library education that we offer at the University of Alberta.
Finally, I will make some recommendations as to how these approaches
might be applied in school library settings.
Evaluation
Strategies
At
least four different approaches have been used to evaluate school
library education at the University of Alberta (see Table 1, for
details):
Evaluation
of Courses and Instructors
Evaluation
of Impact of Courses
Evaluation
of Program Quality
Evaluation
of Program Content
Table
1: Evaluation of School Library Education at the University of
Alberta
|
|
What?
|
How?
|
By Whom?
|
Example
|
More
Information
|
|
1
|
Course
content
Instructor
quality
|
Computer-scored
questionnaires
|
Current
students
|
UofA’s
USRI
|
Link
to USRI on <http://www.
ualberta.ca>
|
|
2
|
Impact of
learning on practice
|
Surveys
|
Current
students
|
Impact
of online
courses
|
Oberg,
1998
|
|
3
|
Program
quality
|
Self-study
Focus
groups
|
Practitioners
Consultants Instructors
|
CSLA
Qualifications
for Teacher-librarians,
1979
|
Easton &
Oberg, 1993; Oberg & Easton, 1995
|
|
4
|
Program
content
|
Self-study
Surveys
Consultation
|
Instructors
Practitioners
Graduates
|
AASL
Standards for the 21st-Century Learner, 2007
|
Branch
et al.,
2009a,
2009b
|
Evaluation
of Courses and Instructors
At
most universities, students are routinely asked to evaluate the
quality of courses and instructors. At the University of Alberta,
this practice is referred to as Universal Students’ Rating of
Instruction (USRI). Computer-scored questionnaires, with mandated
response items and space for comments, are administered to students
near the end of all courses. The numerical results are reported to
the instructor and the instructor’s supervisor and are
accessible to all members of the University community via the
University’s website. The focus of this practice is
accountability to students, ensuring that course content is
appropriate and that instructors are of high quality. The information
generated is based on student perceptions. Because of this,
University policy supports a multifaceted approach to evaluation of
courses and instructors and cautions against heavy reliance on USRI
results only. Nevertheless, meeting students’ learning needs is
a mandate of the University, and USRI results are taken very
seriously.
Over
the years, USRI results have indicated that, with few exceptions,
students rate highly the course offerings in school library education
at the University of Alberta in terms of both course content and
instructor quality. As gratifying as this is, however, the
information that is more useful for making improvement in both
curriculum and pedagogy comes from the students’ USRI comments.
Some examples of changes that have come about because of student
course evaluations include: reorganizing program content to make
courses more “stand alone” in order to better meet the
needs of students who could not take courses in the recommended
sequence; scheduling courses to meet teachers’ and
teacher-librarians’ academic year (i.e., no courses offered in
May-June Spring term); and developing a course on new and emerging
technologies.
Evaluation
of Impact of Courses
Course
evaluation gets at the short term satisfaction of students with
course content and instructor quality. Rarely, however, are students
asked to evaluate the long term impact of their courses (or
programs). This is a critical consideration: What if students react
positively to the learning experience but do not apply what they have
learned in their professional practice? This question was central to
an impact study conducted by the program coordinator, with the
assistance of an evaluation researcher at the University of Alberta
(Oberg, 1998).
Students
who had been involved in the first two online courses (Fall and
Winter of the 1996-97 academic year) offered through the
Teacher-Librarianship by Distance Learning Program were invited to
participate in the study. Responses were received from 15 out of 29
participants (a response rate of 52%).
The
courses were intended to provide opportunities for learning that
could improve students’ professional practice. For 13 out of
15 participants, this was very much the case. Participants were asked
if they had noticed any changes in their professional practice that
they could attribute to their distance learning experiences. Four
responded “no” or did not answer the question. Those who
responded “yes’ added such comments as:
“I use technology with my
pupils more often.”
“I
facilitate others in the use of communication technologies as a
result of having had a variety of distance learning experiences.”
“[I]
refer to e-journals and e-mags more, subscribe to several educational
listservs.”
The
use of technology in the courses also was intended to develop
students’ knowledge of and ease with communication
technologies. For 10 out of 15 participants, this was very much the
case. Participants were asked if they had noticed any changes in
their use of communication technologies that they could attribute to
their distance learning experiences. Four responded “no”
and added comments such as “I have been exploring email and
teleconferencing etc. over time!” Those who responded “yes”
added comments such as:
“I
am more comfortable about using the Internet and CD-ROMs and am able
to integrate this into the courses I teach.”
“I
am a more frequent user of the Internet to search for information. I
am more confident about teaching others (teachers, students) about
how to use the Internet.”
Because
these students had taken the courses at least a year before the
survey, they were able to assess the impact of their coursework over
the longer term. Their responses indicated that the students were
putting into practice what they were learning.
Evaluation
of Program Quality
Evaluation
of programs is necessarily more complex than evaluation of courses.
Program evaluations are generally multifaceted activities which occur
over a year or more and examine overall program quality—program
delivery as well as program content. Often program evaluations
involve self-study by the program providers, comparison of program
elements to national or international standards, and interviews by
outside evaluators of those involved in and served by the program.
MLIS programs in Canada are regularly reviewed and accredited by the
American Library Association Committee on Accreditation (COA) working
in conjunction with the Canadian Library Association. There is no
Canadian equivalent of the U.S. National Council for Accreditation of
Teacher Education (NCATE) for reviewing MEd and Diploma in Education
programs. That means that school library education programs offered
by Faculties of Education in Canada are rarely reviewed and revised
at the program level with attention to overall program quality.
Two
program-level evaluations have been conducted for the school library
education program at University of Alberta in the past two decades.
In the early 1990s, the U of A program was primarily a Diploma
program, delivered face-to-face and on campus. The program
coordinator, with the assistance of a program evaluator from the
ministry of education, conducted a program evaluation using a
self-study which involved analysis of student evaluations and
comparison with national standards as well as findings from a series
of focus groups which involved representatives from the groups served
by the program (Easton & Oberg, 1993; Oberg & Easton, 1995).
Six themes, representing areas critical to school library education,
emerged from the program evaluation (listed here in order of
frequency of mentions):
taking
a leadership role – learning leadership skills in order to
develop and articulate a philosophy and plan for the school library
program;
developing
a school library program – enhancing expertise in curriculum
and pedagogy and knowledge of the research process, of learning
styles and thinking skills, and of children’s and young adult
literature;
managing
the program – developing sufficient technical skills to be
able to teach them to others and to be able to implement
efficiencies;
coping
with technology – learning how to keep up with new
technologies and how to help others to use new technologies for
instructional purposes;
sequencing
of learning – recognizing and providing for the different
learning needs that teacher-librarians have throughout their careers
requires flexible and adaptable programs; and
modeling
and mentoring – socializing new teacher-librarians into the
profession through practice-focused and collaborative pedagogies.
For
the ten years that followed, program improvements were made, guided
by these themes, strengthening these aspects within established
courses and developing three new courses, EDES 549 Leadership in
Information Literacy, EDES 545 Information Technology for Learning,
and EDES 542 Inquiry-based Instruction.
Evaluation
of Program Content
By
the turn of the new century, the U of A program in school library
education had grown substantially, particularly in numbers of MEd
students, and had been redeveloped as an online program. The themes
identified in the program review of the mid-1990s continued to be
critical ones for school library education but two had increased in
importance: taking a leadership role and coping with technology. The
latter had become so demanding that a second technology course,
focusing on Web 2.0 tools, was added to the program.
The
2008 program review was multifaceted, involving self-study, a survey
of MEd graduates, comparison with standards, and consultation with
practicing professionals. The focus of the 2008 program evaluation
was on program content since the overall program quality continued to
be high, as measured by admission grade point averages, student
course evaluations, student publication rates, and employer
testimonials.
The
self-study and survey of graduates were conducted by the program
coordinator, with the assistance of a graduate research assistant.
The self-study involved closely examining the learning outcomes of
all the courses, looking for redundancies, gaps, and relevance to
current professional practice. The survey was sent to the graduates
MEd of the online TL-DL program; responses were received from 26 out
of 29 graduates (a response rate of 90%). They were asked about how
well the MEd program prepared them for their work and what trends and
issues they saw emerging in their work. Five of the themes reported
here echo those identified in the program evaluation carried out 15
years earlier:
taking
a leadership role -- the MEd empowered graduates to take on greater
leadership roles in their schools and districts;
developing
a school library program – becoming a better teacher was part
of becoming a teacher-librarian, the importance of learning how to
teach the inquiry process and of supporting literacy as well as
information literacy;
coping
with technology – new and emerging technologies, how to
integrate them into teaching and learning,
sequencing
of learning – recognizing the need for continuing professional
development, to keep up with research and to deal with changing
technology; and
modeling
and mentoring – the program was transformational, both
personally and professionally, through sharing ideas with others
from around the world, being part of a community of learners.
The
graduates reported that the MEd program had prepared them well for
their work, that the online format provided them with the flexibility
needed to succeed while managing diverse work and home
responsibilities, and that the course assignments provided them with
opportunities to tailor their learning to their unique situations and
professional learning needs. They reported that they would highly
recommend the program to others.
Another
strategy for assessing program content is to compare the program
content to professional standards such as the new American
Association of School Librarians (AASL) Standards for the
21st-Century Learner.
We were assisted in this work by Dr. Marcia Mardis of Wayne State
University School of Library and Information Science, who joined our
Faculty for three weeks as a Fulbright Senior Specialist. The
Fulbright program supports faculty exchanges between the United
States and Canada (and many other countries as well). Dr.
Mardis was instrumental in the writing of the AASL and is a prolific
researcher and conference presenter in the area of school libraries.
Based on the self-study, the graduate survey, and the AASL standards,
the program content was re-visioned and a curriculum document was
prepared for feedback from the field. Dr. Mardis joined the program
coordinator and instructors in exploring the foundations of the new
AASL standards and the new TL-DL curriculum with a group of
teacher-librarians, instructors, and consultants.
The
new curriculum proposes seven courses which combine and re-imagine
topics from our current program and add new topics. For example, we
saw a need for a new course in multiple literacies with an emphasis
on reading. We also realized, both from the graduate survey and from
our own reading and thinking, that the leadership role of the
teacher-librarian needs to be the integral part of every course. We
also recognized that providing choice in assignments was making
learning meaningful for our students . We conceptualized the work of
the teacher-librarian in terms of six competencies—coordination,
consultation, communication, curriculum, collaboration, and
community—which are exercised in the teacher-librarian’s
work within the professional community as well as within the school
community.
In
the next few years, this new curriculum will be developed and new
courses will be approved by departmental, faculty and university
level committees. Plans are also in the works for an instructor
handbook that explains why we do what we do as well as what we do—the
need for this has grown as we have moved into a distributed faculty
model where instructors are geographically remote and where
orientation to the culture and beliefs of the TL-DL program cannot be
developed through face-to face activities.
Applications
in School Library Settings
Evaluation
of Courses and Instructors
Evaluation
of the programs and services provided by the school library and the
school library staff serves both accountability and transformational
purposes. Evaluation can indicate the extent to which students and
teachers perceive that they benefit from those programs and services
(accountability), but it can also help to shape those programs and
services and enhance the understanding of and commitment to those
programs and services for both library staff and library users
(transformation).
An
important research study exemplifying both accountability and
transformation purposes of evaluation is the Student
Learning through Ohio School Libraries (The Ohio Study) where
students and teachers in 39 Ohio
schools were asked through an online survey of 48 items to respond to
the question “How does the school library help?” (Todd &
Kuhlthau, 2005a, 2005b). A major study of this sort is beyond the
resources of most schools or school districts, but there are simple
but effective alternatives. Two examples are: district satisfaction
surveys and school-based surveys or feedback groups.
Most
school districts these days have some version of an annual
“satisfaction survey” which is completed by students,
teachers, and parents. If your district does use this evaluation
strategy, it is very worthwhile to try to get a question or two added
to the survey related to library programs and services. Even if you
are unsuccessful at first, the lobbying for such a question can be an
important way to develop a better understanding of library programs
and services within the district administrative group.
School-based
evaluation also can be a powerful tool for accountability and
transformation. For my examples here, I am indebted to a principal of
a rural school of about 300 K-12 students that I am fortunate to know
well (K. L. Riise, personal communication, January 15, 2009). Last
year, in preparation for developing the school’s three-year
plan, he and his vice-principal surveyed each class in the school,
beginning with grade one, asking the students questions such as “What
makes our school good?” and “What should we do more of to
make it better?” The data from the surveys were analyzed and
shared with teachers and support staff and with the Parent Council.
This year, he is having student representatives, two from each class,
grades 7-12, meet to give feedback on the school’s course
offerings and “Expect Respect” program, asking students
to identify over the course of several feedback sessions what they
would like more or less of in the courses/program and what other
issues need to be addressed in the courses/program to make their
school a better place for them as learners. Both of these approaches
to program evaluation could be easily adapted to evaluating all or
part of the library’s suite of programs and services
Evaluation
of Impact of Courses
Impact
evaluation for the school library focuses on the concept of
‘value-added’ and can be designed to identify the
contribution of school library inquiry activities to student
learning. Here is where it is important to find out from the students
what they have learned. For example, inquiry projects should result
in students developing a deep understanding of a topic, knowing how
the process of inquiry works, and appreciating the importance of
their learning. To discover to what extent inquiry projects had had
an impact on the three aspects of student learning, elementary
students from Grades 1-6 in the Library Power project (Oberg, 1999)
were asked at the end of their projects:
Could
you tell me about your project, about how you used the books and
computers? What worked well, what caused problems? (“Construction
of Knowledge”)
How
did you get started? What did you do in the middle? How did you
finish? How did you feel at each of these points? (“Disciplined
Inquiry”)
What
did you learn, what stands out in your memory? Did you share your
project outside of school? How is your project like things people do
outside of school? (“Value Outside of School”)
A
similar approach which could be used with secondary students is the
School Library Impact Measure, also called the Student Learning
Impact Measure or SLIM (Todd, Kuhlthau & Heinstrom, 2005). At
three points in the inquiry process, students are asked to complete
Reflection Sheets which ask similar questions:
Take
some time to think about your topic. Now write down what you know
about it.
How
interested are you in this topic? (“Not at all” to “A
great deal”)
How
much do you know about this topic? (“Nothing” to “A
great deal”)
Thinking
back on your research project, what did you find easiest to do?
Thinking
back on your research project, what did you find most difficult to
do?
What
did you learn in doing this research project? (This question is only
asked at the end of the project.)
Analysis
of student responses to interview questions or reflection sheets will
be a demanding and time-consuming exercise for the teacher-librarian
and teachers involved but they will be able to see how students have
developed their knowledge and understanding of curriculum content as
well as information handling and process skills that are important
for school, work, and beyond.
Evaluation
of Program Quality
A
program evaluation focusing on program quality normally is a long
term project, often completed over several years and involving a
variety of activities. It is helpful to begin with a framework to
guide those activities such as a school accreditation process or a
provincial or national standards document. A comprehensive program
evaluation should be undertaken only with strong administrative
support and with access to outside expertise such as a district
consultant. However, value can be derived from undertaking a partial
program evaluation, carefully planned to limit the resources needed.
For example, a self-study might address the scope of library-based
instructional activities over only a single term or academic year to
assess what percentage of students and teachers have been involved in
these activities. A comparison to standards might be limited to one
aspect of the library program or services, such as facilities or
collection; a survey of graduates might be designed to address
only the impact of the library’s instructional activities on
the graduates’ current work or study activities.
Evaluation
of Program Content
A
program evaluation focusing on program content could be broad or
narrow in scope and could be one-time or ongoing. A self-study might
be designed to analyze the learning outcomes addressed through
library-based instruction activities over a term or over several
years.
The
learning outcomes addressed through library-based instruction
activities might be compared to the learning outcomes in one or more
curricula: in Alberta, the Information and Communication
Technology (ICT) Program of Studies (Alberta Learning, 2000-2003)
provides an excellent starting point for determining the extent to
which the library’s instructional activities are meeting
general curriculum objectives. Another approach might be the use of
focus groups of classroom teachers and/or department heads which
might consider what learning outcomes should be addressed through
library–based instructional activities. For best results (rich
discussions and frank observations), focus groups are best organized
and facilitated, not by the teacher-librarian but by a third party,
that is, an outside evaluator such a teacher-librarian colleague from
another school or a district learning consultant.
Evaluation
as Accountability and Transformation
In
my doctoral research in the 1990s, I spent almost two years in a
school district trying to understand how a school district went about
changing its school libraries. During that time, there was a major
shift in the way that people in the district thought about three
aspects of the school library: from a collection in a facility,
isolated from the curriculum, and directed by support staff, to an
instructional program, integral to the curriculum, and directed by
teaching staff. This transformation of perception occurred during the
evaluation of the district’s libraries. This experience taught
me an important concept: program evaluation can address more than
decision-making or problem solving concerns (accountability); program
evaluation can influence people’s thinking about a program and
develop support for the program (transformation).
Contact
information:
Dianne
Oberg, 403-492-5416, <doberg@ualberta.ca>
Teacher-Librarianship
by Distance Learning website, <http://www.quasar.ualberta.ca/tl-dl/>
References
Alberta
Learning. (2000-2003). Information and communication technology
program of studies. Retrieved January 12, 2009, from
http://education.alberta.ca/teachers/program/ict.aspx
Barranoik,
L. K. (2004). Meaningful
research projects: Perspectives from high school students and
their teacher.
Unpublished doctoral dissertation, University of Alberta, Edmonton,
AB.
Branch,
J., deGroot, J., Mardis, M., GallowaySolowan, D., & Barranoik, L.
(2009a). What is the right degree for teacher-librarians in the 21st
century? MSLA Journal. Retrieved January 12, 2009, from
http://www.manitobaschoollibraries.com/therightdegree.htm
Branch,
J., deGroot, J., Mardis, M., GallowaySolowan, D., & Barranoik, L.
(2009b). Education for teacher-librarians in the 21st
century: The University of Alberta’s new Master of Education
program. The Medium. Retrieved January 12, 2009, from
http://www.ssla.ca/medium_articles/medium_winter2009.htm
Easton,
E., & Oberg, D. (1993). Focus groups in the evaluation of a
school library education program. Alberta Learning Resources
Journal, 12(1), 18 25.
Oberg,
D. (1994). Rethinking program evaluation: Lessons from a case study.
The Canadian School Executive, 14(5), 21-24).
Oberg,
D. (1998). The impact of distance learning on professional practice.
Teacher Librarian Today, 4(2), 32-34.
Oberg,
D. (1999). A Library Power case study of Lakeside Elementary School,
Chattanooga, Tennessee. School Libraries Worldwide, 5(2),
63-79.
Oberg,
D. & Easton, E. (1995). Focus group interviews: A tool for
program evaluation in school library education. Education for
Information, 13, 117-129.
Todd,
R. J. & Kuhlthau, C. C. (2005a). Student Learning through Ohio
School Libraries, Part 1: How effective school libraries help
students. School
Libraries Worldwide,
11(1), 63-88.
Todd, R. J. & Kuhlthau, C. C. (2005b).
Student Learning through Ohio School Libraries, Part 2: Faculty
perceptions of effective school libraries.
School Libraries Worldwide,
11 (1), 89-110.
Todd,
R., Kuhlthau, C., & Heinstrom, J. (2005). SLIM Toolkit. New
Brunswick, NJ: Center for International Scholarship in School
Libraries, Rutgers University. Retrieved January 12, 2009, from
<http://cissl.scils.rutgers.edu/index.html>
