PHASE+4+EVALUATION+(Review)

=PHASE 4 - Evaluation (Review)=

Evaluation is understood as "the collection, analysis, and interpretation of evidence about the effects and outcomes of a selected activity or form of provision. It includes both intended and unintended outcomes and should support the making of judgements about the activity or the provision" (Thorpe 2001). This evaluation process is part of a cyclical model which feeds key learning from the evaluation back into the design of the learning system as depicted in the following diagram:



Evaluation is always a complex process that requires careful planning against key selection criteria. John Owen (2006) notes that the list of items to be analysed is usually too long and serious discussion needs to take place to reduce the list to a series of issues and questions that are realistic and have focus. As Thorpe (2001, p131) points out, it is better to decide on priority areas and get general reactions on the first attempt, rather than trying to find out everything and risk being overwhelmed by too much data. Subsequent evaluations of the course (which may happen on a yearly basis) can then focus on more specific feedback in key areas as the course is further developed. The evaluation plan below has been devised based on the areas that Owen (2006) outlines as imperative to any evaluation plan.

Object of the Evaluation
Initially, the object of this evaluation is the effectiveness of the transition of this course, from a F2F delivery to a blended learning experience. Using various methods of data collection as several points throughout the transition process, we will bring together the reflections from teachers and learners. The feedback will be on the different aspects of the course experience, including content, design, learning and teaching activities, and assessment. These reflections will be crucial when other courses/units of study need also to be redeveloped from F2F delivery to online blended learning. For example, this could assist in building a Frequently Asked Questions (FAQs) section for online learning. Much of the data collected can then be used for ongoing evaluations of the course with the focus possibly changing over time.

**Purpose of the Evaluation**
The purpose is to evaluate the effectiveness of innovative teaching, particularly in a high school setting. Thorpe (2001, p.128) notes that “once course materials are in use, practitioners should be able to answer the questions "Have we achieved what we set out to achieve, and why?"

Primary Audience
Owen (2006) notes that there is often a key small group within an organization for whom the evaluation has particular meaning. In this instance, the primary audience is the team of teachers who are responsible for the design, delivery and ongoing development of the learning program. The evaluation will provide them with feedback in order to gauge the effectiveness of their teaching of this subject in a blended learning environment. A secondary audience is school leadership such as principals and department heads who want to evaluate the effectiveness of the initiative and learn from its implementation.

Key Issues and Questions
Benigno and Trentin (2000, p. 262) state that there are two aspects to evaluation: "the first is evaluation of learning, and the second is evaluation of the participants' performance". The following issues and questions aim to incorporate these two aspects and can be asked at any point in the delivery of the course:

1. Content
We must ask whether the learners understand the course content. This focuses on the effectiveness of activities throughout the course and looks at test results, participation and learners' attitude (Simonson, 2012, p. 353).

2. Design
We need to consider the learners access to technologies which focuses on the "blended nature of the course" (Bath & Bourke, 2010, p. 61). The design of the course must incorporate the aspects of the learners accessibility to technologies as well as their usability of the technologies. This can be extrapolated via learner profiling and once the learning is underway, teachers can evaluate whether learners are using the course or resources as intended (Thorpe, 2001, p.126).

3. Learning and Teaching Activities
The lesson-to-lesson activities for both teacher and learner must be considered. In this blended environment, many activities focus on collaboration which should be closely monitored through the learning process. In this case, the teacher's classroom performance in this new environment must also be a key component to evaluation. Is the teacher using the materials (technologies and media) to its best advantage so that learners can fulfill the requirements of the unit? The learners' level of comfort in using those technologies should be a part of the evaluation process. The issue of workload should be monitored throughout the learning period. Students and teachers need to be aware that " where several media are in use, the effort of understanding what each medium carries and how best to use it may add to the total study time required." (Thorpe, 2001, p. 128)

4. Assessment
In this blended environment, it is important to focus on teacher-learner interaction regarding assessment. With the ability to respond asynchronously, for example, via email or discussion boards, it is possible for teachers to provide prompt feedback as well as students to raise concerns and questions promptly. Much of this communication can be documented without effort (e.g. posts on a discussion boards) and can be used as evaluation data later on.

Assembly of Evidence
Data for this evaluation will be collected from four key areas based on the 4Q Model of Evaluation (Bath & Bourke 2010; Smith 2008):

**1. Self-evaluation**
The teacher reflects upon the teaching and learning experience with particular reference to what they do and why they do it. Teachers can obtain self-reflective data in a variety of ways. Some of these questions are about learning theories and teaching strategies that Robson (2000, p. 166) calls the "community context".
 * Self-reflective questions: What is my espoused theory of learning? Is my current practice in keeping with my theory? Am I using technology in a pedagogically sound way? How can my theory help me teach more effectively?
 * Reflection on personal work experience: Was there an increase or decrease in workload? Was it more difficult or easier to manage the course?
 * Technology and media training for students and teacher - was it difficult? time-consuming? cost-saving?
 * Keeping a teaching journal which utilises a 3-stage process which involves recalling experiences, reflecting and learning.

**2. Peer evaluation**
The teacher draws on the experience and insight of his/her peers according to specific aspects outlined by the teacher. A colleague may sit in on the a lesson with blended learning and then provide feedback to the teacher verbally or in written form. A colleague can also provide feedback on the way in which learning materials are used in a lesson with a focus on how they were used in a blended learning environment.

3. Student Learning
Bath and Bourke (2010) state that "sometimes it can be very helpful to get feedback from students about their learning as a way of getting feedback about your teaching" (Bath & Bourke, 2010, p. 65). A series of questions can be asked of students at various points of learning and can range from asking them about what confused them, which learning experiences were easier to understand and what new things they learnt after specific lessons. In the spirit of the blended learning environment, it is best to collected learners responses through both online and face-to-face discussion and surveys. Evaluation of student work will also provide insight into student learning. Assessment tasks, quizzes and in-class/online activities provide opportunities to assess the quality of learning that is taking place. Students can complete a self-assessment before an assessment task and afterwards. Homework completion rates will give an indication of student motivation and learning. LMSs offer new resources for evaluation. Transcripts of online discussions and question-and-answer interactions between learner and teacher via the LMS can offer an insight into "the effectiveness with which the learning technologies have been used" (Thorpe, 2001, p.130).

**4. Student Experiences**
It is important to give students the opportunity to offer their feedback on the course. Informal feedback from learners is a very good indicator on how well the course is design, delivered and experienced. Initially, the focus on this informal feedback should involve the activities that were impacted by the blended learning changes. The teacher must allow students to offer their responses to activities that now use technologies and media in its delivery method. Questionnaires can be filled in by students that may include questions such as "Do feel the technology and/or media was appropriate for the task?" and "In what can the task be improved with or without technology and media?" Online questionnaires can also be ideal in evaluating teaching and methods of teaching.

**Dissemination**
It is the responsibility of the teachers to gather this information and to reflect upon it. The four dimensions of the 4Q model give plenty of data for meaningful reflection by teachers, particularly as the course transitions to a blended mode of delivery. Feedback should be reported to both students and school management (Teaching and Learning Committees, Heads of Department) so that all participants are informed of the changes that are going to be made as a result of their feedback and so that there is adequate accountability.

Codes of Behaviour
Ethical issues that need to be considered during the evaluation process. Students must feel comfortable with giving honest responses in surveys. The anonymity of online surveys and questionnaires goes a way in providing this environment. In a secondary school environment, the ideal situation would be for students to be able to speak candidly and informally about their reactions to the blended course, however, this may not always be possible. The teacher needs to ensure adequate space between themselves and students when collecting of data ((Bath & Bourke 2010). The data collection for this blended course may be invaluable for other course redevelopment, it may be worthwhile investing in recruiting the teacher's peer to conduct discussions and interviews with the students as " it is easier to be honest with a disinterested observer than with the course designer." (Thorpe, 2001, p.132)

Timeline and Resource Implications
It is important that data is collected while the teaching and learning experience is still fresh in everyone’s minds. This will occur in the latter stages of the learning experience and before final marks are allocated. It will also be important that there be feedback early in the course to give time to make course adjustments early in the piece, in particular after a first-time activity or before and after an assessment task. It will also be helpful to allow students to complete pre-learning student surveys to determine the type of learner they are and to document their expectations of the course.

< Previous Page (Implementation) | Next Page (References) >