Profile of Learners
The formative evaluation of this program was conducted with 4 students who are part of the target learner audience. Two females and two males evaluated the program by participating in three of the modules in the program and completing an evaluation. The ages and characteristics of learners are presented in the table below.
Learner |
Gender |
Age/Grade |
General Characteristics |
| Learner #1 |
Female |
16/10th Grade |
Caucasian; has not studied the Book of Mormon in Seminary |
| Learner #2 |
Male |
17/11th Grade |
Caucasian; has not studied the Book of Mormon in Seminary |
| Learner #3 |
Female |
15/9th Grade |
Caucasian; has not studied the Book of Mormon in Seminary |
| Learner #4 |
Male |
17/11th Grade |
Caucasian; has not studied the Book of Mormon in Seminary |
Assessment
I observed all of the participants as they took part in the program. I looked for how they were interacting with the tutorial, and tried to detect any problems or obstacles that the learners were having. After participating in the program the learners were asked to fill out a likert-type survey regarding the following statements:
1. The program was easy to use.
2. I was able to understand the information presented in the program.
3. The program made me excited to learn more about the Book of Mormon.
4. There were too many videos and pictures, and I had trouble focusing on important information.
5. I learned more about the Book of Mormon than I knew before I went through this program.
6. The information presented was too easy for me and the program was boring.
7. I would rather use this program to learn about the Book of Mormon rather than read about it in the student manual.
8. This program could be enhanced by adding:
9. My favorite part of the program was:
10. My least favorite part of the program was:
Do you have any other suggestions about this program: |
For each question, a Likert scale response was used, and learners were able to choose their response by circling choices ranging from 5 (Strongly Agree) to 1 (Strongly Disagree). The open-ended questions provided space for comments and suggestions. Each question was aimed at specific evaluation areas: usability, content, reading level, etc.
Procedures
On December 10, 2004, the four learners participated in the program and filled out the survey. It took place on the computer in my home.
The students were brought into my office one by one and I observed them navigating the program. They sat at the computer on my desk and I stood behind them and to the right, out of their line of sight. I noted any problems or difficulties that popped up as the students went through the program. Each student tried to ask me questions, but I did not answer them. After completing the program I gave the student the survey and left the office.
Results
Results from the survey are presented below. For each multiple choice question, an average score is presented, followed by discussion of the responses from the open-ended comments and questions.
Key:
| Strongly Agree: |
5 |
| Agree: |
4 |
| I Don't Know: |
3 |
| Disagree: |
2 |
| Strongly Disagree: |
1 |
Question |
Average Score |
| 1. The program was easy to use.
|
4.25 |
| 2. I was able to understand the information presented in the program. |
4.50 |
| 3. The program made me excited to learn more about the Book of Mormon.
|
5.00 |
| 4. There were too many videos and pictures, and I had trouble focusing on important information.
|
1.75 |
| 5. I learned more about the Book of Mormon than I knew before I went through this program.
|
5.00 |
| 6. The information presented was too easy for me and the program was boring.
|
2.25 |
| 7. I would rather use this program to learn about the Book of Mormon rather than read about it in the student manual.
|
4.75 |
The average scores indicate that the learners felt that the program was helpful overall, and the level was appropriate for their abilities and experience with this topic. It also seemed to effect the students' motivation to learn about the Book of Mormon.
During the observation of their navigation I noticed that the students seemed to skim the textual information presented in the timeline section. They also were quite attentive during the video aspects of the program. None of the learners used the help portions of the program and they didn't read the instructions. They figure things out through experimentation. I noticed that two of the students would click on home and then the timeline button rather than clicking on the timeline button that was provided. Also, they all tried to click enter on the Jeopardy game to answer the question.
Revision
Problem |
Revision Plans |
| Two mentioned that it took too long to load |
I need to redo the program so that one small file loads the different pages of the program from an external file |
| Three mentioned that it would be helpful for keyboard commands to navigate the program |
Add keystroke commands in Flash |
| All mentioned that they would like a way to get the right answer after getting a wrong answer in the Jeopardy game |
I don't know if I want to revise this. My thinking is that it encourages them to go back into the program. |
| Three mentioned that a video on every page would be cool |
Due to lack of time and resources this won't be happening |
| One student mentioned that the pages with text and a picture were quite boring |
Animate the graphics on those pages |
Overall, this evaluation aided me in seeing where students were having difficulty, which aspects they liked and disliked, and practical considerations that I can add or change to make the program more effective. The students seemed to really enjoy the program and commented on how cool it was.
|