Part+3+-+How+will+you+evaluate+this+course

Evaluation is understood as "the collection, analysis, and interpretation of evidence about the effects and outcomes of a selected activity or form of provision. It includes both intended and unintended outcomes and should support the making of judgements about the activity or the provision" (Thorpe 2001). This is a complex process that requires careful planning against key selection criteria. John Owen (2006) notes that the list of items to be analysed is usually too long and serious discussion needs to take place to reduce the list to a series of issues and questions that are realistic and have focus. Owen identifies a number of important dimensions to any evaluation plan:
 * 1) **Specifying the evaluand** (i.e. the object that is the focus for the evaluation) We need to specify here exactly what is the object of our evaluation. Are we evaluating the transition from F2F course to blended learning? Are we evaluating the design and implementation of the course? Are we evaluating student learning? Are we evaluating teacher experience? So, what is the object or focus of the evaluation?
 * 2) **Purpose of the evaluation.** What is the fundamental reason for doing the evaluation? Is it to aid program development? Or to improve the implementation of the program? Are we trying to monitor program outcomes? Or are we trying to determine the impact of the program?
 * 3) **Primary audiences.** Who will receive the information? Owen (p69) notes that there is often a key small group within an organization for whom the evaluation has particular meaning. Is this the teachers of the course who want to gauge the effectiveness of their teaching in this particular subject? Is it school leadership such as principals and department heads who want to examine the effectiveness of the initiative? Is it management of the department?
 * 4) **Evaluation Resources.** What person power and materials are available? Someone must plan and set up appropriate data management systems. Who is this and what have they done?
 * 5) **Evaluation focus.** Which elements of the program will be investigated?
 * 6) **Key issues and questions.** This needs to select the most important aspects of the program for examination.
 * 7) **Assembly of Evidence.** What is the strategy for collecting and analysing evidence? These should be aligned with the questions that are being addressed.
 * 8) **Dissemination.** How will the information about the evaluation be disseminated? How will the findings and conclusions be disseminated? Is there a need for recommendations? Who will create them?
 * 9) **Codes of behaviour.** Are there ethical issues that need to be addressed during the evaluation process?
 * 10) **Budget and timeline.** An indication of the amount of resources needed for the evaluation should be included.

__**SUGGESTED METHOD OF EVALUATING THE COURSE**__

Pre-learning student surveys: - Allow students to complete __**surveys online**__ (possibly through something like Survey Monkey) to determine the type of learner they are and to document their expectations of the course

Survey after key tasks/assessment - e.g. after a first-time activity, after an assessment task

Self-assessment - students to complete a self-assessment before an assessment task and afterwards.

Measuring motivation? - Maybe by looking at homework completion rate

Data collection: - Transcript of online discussions and q&a from LMS - survey results - assessment task results - homework completion rate - Enrollment levels for the next year - tricky because course enrollment can be according to popularity (in and out of fashion).

Educator self-assessment - personal workload - did it increase/decrease? more difficult or easier to manage course? - training for students and educator - was it difficult? time-consuming? cost-saving?



__**Part 3 - Evaluation**__

Some important points from the Deakin Reading: __**2.2 Evaluating the use of learning technologies by Mary Thorpe**__

The definition of evaluation: "evaluation is the collection, analysis, and interpretation of evidence about the effects and outcomes of a selected activity or form of provision. It includes both intended and unintended outcomes and should support the making of judgements about the activity or the provision." (p.126)

"Once course materials are in use, practitioners should be able to answer the the questions "Have we achieved what we set out to achieve, and why?" " (p.128)

"Adding learning technologies to the resource base of learning programs may increase the study time required in two ways. First, where several media are in use, the effort of understanding what each medium carries and how best to use it may add to the total study time required. Second, these technologies typically contain optional resources, so learners may spend time learning more than they need to - or simply get hooked on an optional extra and spend longer using it than the course designers intended." (p. 128)
 * The issue of increased workload when adding technologies**

LMSs, and the emphasis on collaborative learning, are forcing learners to put time management high on their priorities. Educators need to determine "what makes for the most effective online support for learners." (p. 129)
 * Changing roles of learners and educators**

__**DATA COLLECTION FOR EVALUATION PURPOSES**__

"A great potential advantage of Web-based media is that they can be changed relatively easily. Whereas the aim for print-based materials was to get a course right first time, the new technologies enable a more rapid response to learner needs and content updating. This leads to regular surveys of learners so that preferences can be incorporated in the next year's changes." (p.129-30)
 * Online surveys and interviews**

"A general message can be sent to all students at the start of a course explaining not only its aims, but also how those aims will be evaluated. Regular requests for feedback can be mailed to participants at the end of key sections of a course, and evidence of effectiveness can in theory be available in a matter of days. Thus a rich source of data about learners' reactions from start to finish of their learning is provided by evaluation activities built into the course." (p.130)

"Online delivery also offers much easier access to learners for purposes of small-scale and qualitative studies. An interview can be carried out online in the form of structured question-and-answer sessions, thus avoiding the time-consuming routine of arranging interview times, travel, and transcription of interviews. By printing out the responses of interviewees, the practitioner has a record from which to carry out content analysis and use verbatim quotations, with, of course, the permission of the interviewee." (p.130)

LMSs offer a new resource for evaluation: "a permanent record of the transactions of the whole course, which we can retain and analyze to evaluate the effectiveness with which the learning technologies have been used." (p.130)
 * Transcript of online discussions**

"Surveys and quantitative data have a key role, but can only be used effectively if the issues and learners' perceptions are well understood. So it can be productive to carry out qualitative interviews first, and then follow up with a survey if necessary. Learners should feel comfortable with the interviewers and be able to offer their views honestly. Younger learners may be more prepared to be open about reactions with their peers than with their tutors, so it may be desirable to recruit and train peers to conduct interviews: it is easier to be honest with a disinterested observer than with the course designer." (p.132)

< previous page (Part 2) | working area home