Unit 18 Simulation Analysis

Figure 18: The scenes arrangement
The image above describes the overall scenes arrangement and scenario flow of the system.
The user:
• Will find himself in front of the main menu in the first place which contains the links for the simulation and the conversation scenes.
• The user has the choice to choose to start with the simulation or the conversation which in the normal case he will start with simulation but he has the choice anyway.
• We’ll go with the normal case in which the user chooses the simulation first of all, he will then move to the next scene that will contain two links: the first one for the first sub-simulation named ”Attract and Repel” and the second one for the second sub-simulation named “Electric Force”.
• After playing
the simulation, the user can move to the conversation when he gets back to the main menu.
• Once he greets the robot, the latter will start the conversation by asking the 1st question.
• The user can’t move to the next question unless he answers correctly the current one.
• The user can type “help” if he gets stuck and he will receive a hint to get him close to the answer.
3. Validation:
3.1. Experimentation design:
The experimentation was run in the middle school of Raguada with 9th grade students. Originally the students were to get assigned randomly to one of the following three conditions
• Simulation only: the students will be exposed to the learning concept through simulation only
• Conversation only: the students will be exposed to the learning concept through conversation only
• Full system: the students will be exposed to the learning concept through simulation and then conversation.
But due to the restricted number of students (24) and to avoid disrupting regular class schedule more than that. We had to change the procedure of the experimentation. So instead of three independent samples, we were obliged to conduct the experiment on two dependant samples. So the students were assigned randomly to two conditions instead of three:
• Simulation only then full system: the first group of students will be exposed to the learning concept through simulation only in the beginning then through the full system later.
• Conversation only then full system: the second group of students will be exposed to the learning concept through conversation only in the beginning then through the full system later.
In the beginning of the day of the experimentation, all the students took a pre-test designed to assess the students’ knowledge about the basics of the electrostatic force. Students were told that they will be using three different tools to learn about the basics of the electrostatic force. In the end, all students took a post-test equivalent to the pre-test and then they filled out a questionnaire on their impressions and satisfaction of the system.

Figure 19: The students during the experimentation

Figure 20: Procedure of the experimentation for the 1st group

The first group and that is composed of 11 students, take first of all a pre-test. Then they take 15 minutes to play the simulation. After that they retake the same test they took in the pre-test. Then, they move to use the full system (35 minutes). And in the end, they take the same test for the 3rd time. Thus, we calculate the learning gains after using the simulation and after using the full system and compare between them.
The second group and that is composed of 13 students, take first of all a pre-test. Then they take 20 minutes to use the intelligent tutoring system. After that they retake the same test they took in the pre-test. Then, they move to use the full system (35minutes). And in the end, they take the same test for the 3rd time. Thus, we calculate the learning gains after using the intelligent tutoring system and after using the full system and compare between them.

3.2. Learning impact:

Table11. Average of the pre-test, the test, the post-test and the learning gains for the 1st experimentation Pre-test Test Post-test Learning gain1(pre-test/pots-test1) Learning gain2(pre-test/post-test2)
Simulation only/Full system 1.727 2.090 5.386 0.363 3.659

Table12.
Average of the pre-test, the test, the post-test and the learning gains for the 2nd experimentation Pre-test Post-test1 Post-test2 Learning gain(pre-test/pots-test1) Learning gain(pre-test/post-test2)
Conversation only/Full system 0.615 2.615 5 2 4.153

As we can see there is a difference between the learning gain after using simulation only and the learning gain after using the full system with the favor to the full system. Also, there is a difference between the learning gain after using the chatting robot only and the learning gain after using the full system with the favor to the full system.
But is this difference significant? To know the answer to that, we used a paired-samples t-test. The t-test is one of the most used statistical methods to compare between the means of two groups and to know whether the difference between them is statistically different or not. (Guili Zhang, 2009)
For the first group our hypotheses are going to be:
H0: Null hypothesis: There is no significant difference between the learning gains after using simulation only and after using the whole system.
H1: Alternative hypothesis: There is a significant difference between the learning gains after using simulation only and after using the whole
system.
As for the second group, the hypotheses are:
H0: Null hypothesis: There is no significant difference between the learning gains after using the chatting robot only and after using the whole system.
H1: Alternative hypothesis: There is a significant difference between the learning gains after using chatting robot only and after using the whole system.
After running the test on excel, we obtained the results below: Figure 24: The result of the t-test for the 1st experimentation

Figure 25: The result of the t-test for the 2nd experimentation
Interpretation:
The results included a lot of data, some that’s obvious (like the number of data items). But what really matters is the alpha value, the p-value, the t-value and the critical t-value.
Our alpha value that we chose is 0.05.
For the first group, if we compare p-value with alpha value, we’re to going to find that p-value=0.000145 < alpha=0.05 and also that t-value=5.424 > t-critical value=1.812 so in this case we can reject the null hypothesis that states that there is no significant difference between the learning gains after using simulation only and after using the whole system in favor of the alternative hypothesis
For the second group also, if we compare p-value with alpha value, we’re to going to find that p-value=0.035 < alpha=0.05 and also that t-value=1.974 > t-critical value=1.782 which means we can reject the null hypothesis that states that there is no significant difference between the learning gains after using the chatting robot only and after using the whole system in favor of the alternative hypothesis
3.3.
User’s satisfaction and technology acceptance:
Technology acceptance is a factor that determines the success of the technology since the technology is of no value, if we may say, if the users don’t accept it and don’t use it. A lot of models and theories have been developed to measure the user’s acceptance of technology. We mention:
- Cognitive Dissonance Theory (CDT) (Festinger, 1957)
- Task Technology Fit Model (TTF) (Goodhue & Thompson, 1995)
- Theory of Reasoned Action (TRA) (Fishbein & Ajzen, 1975)
- Technology Acceptance Model (TAM) (Davis, 1989)
Technology Acceptance Model TAM is one of the most common and used models for the sake of determining the technology acceptance and many studies concentrated on expanding it since it was proven by experience that it has a high validity. (T. Ramayah et al,
2002)