Saturday, May 2, 2009

Weeks 7 and 8 - Draft of evaluation plan

Hi there,
Here is a draft of my evaluation plan. Would appreciate any feedback or suggestions.

Draft of evaluation plan
Introduction
This document describes the evaluation plan of a planned online software training course for staff at Manukau Institute of Technology (MIT). Software courses taught may include Microsoft Office applications, eg Word, Excel, PowerPoint, Outlook, Publisher, Visio, and some MIT web-based applications, eg the portal, the intranet site, etc. It will be developed mainly by the Staff Computer Trainer, who is working in conjunction with the Manager of MIT Short Courses. Together they are looking at developing online software training, the MIT Software Trainer for the staff at MIT, and the Manager of MIT Short Courses for industry training. The evaluation, which is a Needs Analysis, will be conducted in May 2009. This evaluation is concerned with the online training for MIT staff.
The Needs Analysis is based on the Eclectic-Mixed Methods-Pragmatic Paradigm, using the Mixed Methods Evaluation Model.

Background
The MIT Staff Software trainer currently runs face to face courses and one on one training on Microsoft Office applications and some MIT web-based applications. She believes that there is a need for online training to cater for those learners who:
· Cannot attend face to face courses due to timetabling constrictions;
· Prefer self-paced learning;
· Possibly to provide blended learning to complement the face to face training.

Purpose:
The purpose of this Needs Analysis is to obtain accurate information that confirms that there is, in fact, a need for this type of elearning, and to optimise the design and delivery of the course by establishing the needs of the learners in terms of content and learning styles – leading to the types of activities that will engage them.

Limitations:
One limitation could be the time involved in carrying out this evaluation, on top of normal (already heavy) workload.

Audiences
The Needs Analysis will be carried out on a cross-section of MIT staff. Experts will also be involved in the evaluation process, including the Manager – MIT Computer Short Courses; the Learning Technology Centre Manager; the Learning Technology Centre Advisor, the Software Support Manager.

Decisions:
I cannot foresee any negative outcome of this evaluation. From informal questioning, there does appear to be an interest in this elearning course, and I expect this evaluation to confirm that, and also to give constructive feedback as to the content and design of the course.

Questions:
These questions have been taken from the Elearning Guidelines for New Zealand:

SD2: Do students have any choice in terms of what they learn, the particular resources they will study and/or the learning activities they will engage in?
i. How will students like to use this elearning course? eg self-paced learning, blended delivery,
ii. What resources will best engage the learner, whilst providing for maximum learning outcomes eg audio visual demonstrations, text-based instructions, interactive activities, formative and/or summative assessments (online tests)?
iii. Which software training courses will best be suited to online training?
iv. How will I monitor what is working well and what isn’t (related to the design of the modules)?

ST1: Do you have a way to identify student needs and respond to them?
i. Are the staff at MIT interested in online software training?
ii. Of those that are interested in online software training, are they familiar with Emit (Blackboard). If yes, do they like using Blackboard; if no, how will training on the use of Blackboard be provided?
iii. What is an effective method(s) for receiving constructive feedback from the learners on their needs, eg what further online training they would like?
iv. What systems will be used to ensure that the learner is learning at an appropriate level, and relevant to the needs of their role?

Methods:
I will be using the Multiple Methods Evaluation model, which will allow me to use different methods of evaluating individual aspects (triangulation) and evaluates a range of factors within each aspect (bracketing). This method is well suited to the complexity of instructional design. (here I will insert a table which demonstrates the relationships between the Questions, and the methods used to evaluate them)


Sample:
The participants involved in this evaluation will be:
· A random selection of MIT staff (volunteers)
· Manager, Learning Technology Centre, and Learning Advisor, Learning Technology Centre
· My manager – Software Support User Manager
· Manager, MIT Computer Short Courses
· Other MIT lecturers and trainers with an interest in eLearning

Instrumentation
· A sample learning module to be prepared on Emit for testing by focus group
· A usability questionnaire to be prepared for staff who are involved in the focus group and testing of the learning module, in order to evaluate the design of the module, and what resources/activities best hold the learners focus and meet learning outcomes.
· A questionnaire to be prepared for MIT staff


Logistics:
I will be responsible for the implementation, analysis and reporting of the evaluation

Time Line
Implementation – 1 week
Analysis – 1 week
Reporting – 1 day


Budget
Captivate software for creating audio visual demonstrations - $200
My time

Appendices
A. Information sheet for participant (yet to be created)
B. Questionnaire - Usability of learning module (yet to be created)
C. Questionnaire – general questionnaire on the viability of online learning for software training (yet to be created

3 comments:

  1. Michelle this is a very comprehensive draft plan. You have obviously thought this through very carefully. Your guidelines are really good and the sub-questions too.

    I would like to discuss the sub-questions for ST1, because a little bit of re-wording could be beneficial to keep them broader - they have become a bit too specific. For example, some suggestions below:
    i. Are the staff at MIT interested in online software training? Suggest this becomes: What aspects of online software training interest staff?
    ii. Of those that are interested in online software training, are they familiar with Emit (Blackboard). If yes, do they like using Blackboard; if no, how will training on the use of Blackboard be provided?

    Change to something like: How familiar are staff with using an online learning management system and what sort of training is needed?
    iii. What is an effective method(s) for receiving constructive feedback from the learners on their needs, eg what further online training they would like? Not sure about this one.

    Maybe something like: How would staff prefer to provide information about their training needs?
    iv. What systems will be used to ensure that the learner is learning at an appropriate level, and relevant to the needs of their role?
    How will you find this out?

    For SD2 - iv How will I monitor what is working well and what isn’t (related to the design of the modules)? We need to talk about this one - I am not sure if you need it.

    Perhaps you might like to change students to staff in the guidelines particularly as you are working with staff on the evaluation.

    If you want to link to your matrix and questionnaires you can either load them on Google documents and publish as a web page (please add me as a collaborator so I can give feedback) or add then to Internet Archive and put the link on your blog. http://www.archive.org/index.php

    Re your time and having to develop the online module. Is it possible for you to create just a small part of the course to show people? otherwise it could all become too onerous and fraught timewise and within the timeline of this course.

    Please contact me for a time to briefly discuss your plan. I do not see too much out of kilter so it will be just to tidy up any loose ends and discuss some of the sub-questions.

    ReplyDelete
  2. Brief, concise, and to-the-point. I really like the Plan Michelle. Wonderful. Looking forward to see the results.

    ReplyDelete