Tuesday, June 2, 2009

Initial thoughts after evaluation

Findings from the Expert Review of the Sample Learning Unit
I think that the Expert Review was the most valuable feedback that I received. The experts (7) sampled the learning module and then completed an evaluation on Blackboard. They had a 15 minute break while I reviewed the results and then as a group we had further discussions to clarify and expand the ideas.

One aspect that was highlighted was that many people don't read instructions on the internet, or skim there way through. A couple of people rushed through the module and didn't even notice some of my activities, eg video demo! So instructions need to be kept to a minimum, and those instructions should be VERY obvious, brief and simple.

Another topic that was well discussed was the use of audio in the learning module. I thought I had prepared for this by adding text captions, but apparently if the computer doesn't have sound, then the video demo doesn't open at all. Although all MIT staff do have sounds on their computers, some may be accessing the learning model from home. Not sure if this issue is just an MIT issue, or maybe related to the software that they are using (they used Camtasia, I am using Captivate at present) Will have to check this out further.

I am quite keen to use audio in my learning module, as research shows that this does add to the learning experience, and caters for different learning styles. Also, 83% of those surveyed on my staff questionnaire preferred audio visual demonstrations. One option suggested was to create one video demo with sound and one without, then staff could select the appropriate medium. If any of you had any experience with this, would appreciate any feedback.

See results of the Expert Review of the Sample Learning Unit on http://spreadsheets.google.com/ccc?key=r19WdTxDb12ig_9iM7S931A

Another question that I brought to this focus group was "What method should be used to find out the training needs of the staff?

I currently use a hard copy of a Training Needs Analysis to recommend training for new staff during their induction training. Training Needs Analysis is only carried out on existing staff if requested by staff member or manager.

After much discussion it was agreed that an Online Training Needs Analysis would fit the bill. Could easily be created in Wimba Create and possibly placed in Intranet site, alongside Health & Safety online quizz (for new staff).

For staff booking on computer courses, a short pretest could be emailed to them when they register, to check that they are at the right level for this course, or to recommend another course - this could easily be done in Wimba Create.

Could also put TNA on Blackboard (staff could self-enrol).

MIT Staff Questionnaire
This questionnaire was completed by 18 staff consisting of FreeB Community Facilitators and Call Centre staff. There were no surprises here, the results were what I expected. The questionnaire was done in SurveyMonkey and I was really impressed with how easy it was to use and download results.

I used a 5 scale Likert questionnaire. On one question "I would need training in the use of an LMS (Blackboard) 10 out of 18 responded "Neutral" - not very helpful, left me wondering. On reflection I would choose a 4 level scale in future.

The preference was for self paced learning 61%, followed by blended delivery 56%. This was to be expected with the group of staff that I chose - they are from offsite locations or in roles where they not easily attend face to face training.

The resources that these staff thought would be most useful to their learning were audio visual demo's, and interactive exercises.

Results of survey taken by 18 MIT staff, consisting of FreeB Community Facilitators and Call Centre staff can be viewed on http://spreadsheets.google.com/ccc?key=rar-Fhxk14bjim7ebihrEAg.

MIT-wide survey
As part of a wider survey, 35 selected MIT staff (managers or their representatives across all departments)were asked if they had any interest in online software learning for themselves or their staff. 26 said "yes", 1 said "no", and 4 said "maybe/unsure"

Presentation of Data
I am thinking of presenting the data in graph format because I think that this visual is more effective than reading figures. They would be a combination of either bar charts or pie charts depending on the data.

All for now,



  1. Michelle you have extricated some excellent information from your participants.Re: "preferred audio visual demonstrations." Yes providing options is good to increase accessibility. Another way is to offer an MP3 option and both the video with sound and Mp3 could be downloaded to cell phones and ipods or mp3 players.
    The findings from the expert review are promising. Now you need to collate the responses from the survey into a Table of the number of responses for each rating on the Likert scale. Then calculate % frequencies as per the example on the course wiki. (http://wikieducator.org/Evaluation_of_eLearning_for_Best_Practice#Weeks_twelve_and_thirteen:_Analysis_of_data_and_results)

    I agree the four part Likert scale removes the uncertainty you mention which came with the reponses to the question. However it also indicates that they are not pushed about having training, and if you were interviewing you could follow this up. However in your discussion you can speculate or surmise that this may be because they are happy to try things out themselves which is a good thing and indicates a certain level of confidence with the technology, or they are unsure if they need training at the moment or not and may do in the future.

    "question "I would need training in the use of an LMS (Blackboard) 10 out of 18 responded "Neutral" - not very helpful, left me wondering. On reflection I would choose a 4 level scale in future."

    I will give more detailed feedback on your results in Google docs if you can please add me as a collaborator. I agree graphs do make it easier to "see" the findings.

  2. Thanks Michelle for sharing your results. It has been good to review yours as you have used frequencies and percentages - now need to do that with mine. I agree with you that graphs are so much easier to read than alot of writing, you can really see at a glance what the results were.

  3. Michelle, in the expert review of sample learning unit, all looks very positive. Te AV demo comes high in the list of ressources helpful for learning. It was interesting to see in the MIT staff questionnaire that there were a substantial part of the people surveyed who felt that Blended Learning is the preferred option (55%). May be they do not feel ready for a full on-line class or are unsure of the support offered. What do you think?
    The AV demo comes again high in the list of resources requested.


  4. Hi Herve
    Thanks for your comments. There were actually more staff that were interested in self paced learning, than blended learning. The questionnaire results showed 11 for self paced learning, and 10 for blended delivery. The percentages don't make much sense in this question, cos they were asked to select all that apply. In hindsight, maybe not a good idea in this case...

  5. HI Michelle,

    Very interesting results,

    I also agree with your comment:
    "question "I would need training in the use of an LMS (Blackboard) 10 out of 18 responded "Neutral" - not very helpful, left me wondering. On reflection I would choose a 4 level scale in future."

    I had also used 5 level scale and the neutral option didnt really help in the evaluation unless a interview was carried out following the survey.

    Good luck with your report.