2009年度 学術交流支援資金報告書

プロジェクト No.: 1-5

研究課題名: Projective Pedagogical Agents: Developing Metacognitive Skills in Math Using Physical and Virtual Technological Artifacts

代表者：徳田英幸

プロジェクト No.	1-5
研究課題名	Projective Pedagogical Agents: Developing Metacognitive Skills in Math Using Physical and Virtual Technological Artifacts
代表者	徳田英幸

Abstract

This research examines whether children can model a pedagogical agent’s behavior/skill by monitoring, and eventually turn that behavior/skill inward for themselves. More specifically, would modeling a computer agent and catching their math mistakes, help children learn the skill of self-monitoring solving math problems. A testing ground called Projective Pedagogical Agent, “ProJo,” will be developed and used as an environment where children can practice monitoring, and improve performance. The research also plans to compare virtual and physical objects as agents. Physiological devices may be used as additional ways to measure reaction to different cues and social behavior when engaging with the agent.

Introduction

There are several ways that people learn (Okita & Schwartz, 2006). Learning can occur by comparing ourselves to others. For example, comparing test scores helps people learn where they stand academically. Another way is observing others helps develop a better understanding of the self. For example observing others solve a math problem helps you learn how to solve problems. A third way is controlling other’s behaviors may help control your own behavior. An example may be learning to self-monitor by monitoring others, where catching other’s calculation mistakes may help people catch their own calculation mistakes. An ideal TBO for this learning environment would be an interactive pedagogical agent designed to solve math problems that would engage the students in catching mistakes and monitoring agent behavior.

Self and Other Study: Learning to Self Monitor by Monitoring Projective Pedagogical Agents

Children may find it difficult to be attentive to their own mistakes when they are concentrating on a

problem or task (Markman, 1977). However, they may find it relatively easy to find inconsistencies in other people’s work. An on-going study examines whether children learn to self-monitor by monitoring others. The task of catching the agent’s calculation mistakes, may help the child learn the skill of monitoring, making it easier to eventually self-monitor. An ideal TBO for this learning environment would be an interactive pedagogical agent that openly displays its reasoning while solving math problems. Children can then correct the agent’s mistakes and continue to monitor agent behavior (See Figure 2).

Figure 1. Addition and multiplication math trick used during the treatment session

Figure 2. ProJo environment where child monitors ProJo for calculation mistakes

In the first study, thirty-one elementary school students between the ages 10- to 11- years participated in a one-to-one sixty-minute session. The session included two within-subject treatment sessions of self-monitoring and self-other monitoring. A new technology called Projective Pedagogical Agent, “ProJo,” provided a testing environment where students practiced monitoring the agent’s performance on the math skill while checking for potential mistakes. One treatment session taught the reordering skills for addition, and another treatment session taught divisibility rules for multiplication problems (See Figure 1). For example, the addition math skill on reordering involved taking the problem 23+35+17+72+15+8= and reordering the numbers 23+17+35+15+72+8=, so that the addition process can be grouped (23+17)+(35+15)+(72+8)=, making the calculation 40+50+80= easier for students. Each treatment session was either addition or multiplication. The self-monitoring treatment consisted of six trials with one problem per trial, where students solved math problems on their own. In the self-other monitoring treatment, students took turns with ProJo. The students had three trials solving the problem on their own, and three trials monitoring ProJo solve problems. ProJo would say, “I think I learned a lot by watching you play. Let me try. I don’t like mistakes, so if you can catch me when I make a mistake that would be great, since I wouldn’t want you to lose a point because of my mistake.” ProJo would make a few intentional “slow motion” mistakes so the student could monitor and catch (or not catch) the error (See Figure 2). ProJo solved problems where sometimes the answer would be correct and other times incorrect. The students were responsible for monitoring and catching any mistakes ProJo made. The relative effects of self-other monitoring were evaluated using calculation time and accuracy measures during the treatments, as well as pre-test and post-tests. Each treatment session consisted of one pre-test question to check for prior knowledge, and a post-test that consisted of four problems varying in difficulty levels.

Figure 3. Slow down effect between conditions when solving easy, medium and hard math problems.

Figure 4. The External first effect where monitoring improved write out problems

The study was a first attempt to examine the question of whether monitoring a task as simple as watching a computer agent play a math game elicit better learning. The results showed that students under the self-other treatment slowed down their calculation speed after monitoring “others”. Three different calculation times are shown in figure 3, the calculation time of the “other” agent, the calculation time of students in self-treatment, and the calculation time of students in self-other treatment. The students in the self-treatment with no monitoring took less time even when the difficulty level increased, whereas students in the self-other treatment took more time to solve the problems. Students that monitored the agent slowed down their calculation speed immediately after monitoring. The posttest had four problems with two parts (“In Head” and “Write Out”) where students were first asked to calculate the problems in their head, and then write out how they solved the problem in their head. Interestingly the results showed that self-other treatment improved the student’s accuracy on write out problems but not in-head problems. Self-other treatment showed a relative advantage when calculations were written-out, but a relative disadvantage when done in the head (See Figure 4). This meant that monitoring “others” affected subsequent paper and pencil task more than in-head problem solving. One possible interpretation was that their own external representation of writing the calculations out was a form of representing the “other” that helped them apply monitoring. Students may learn to monitor external behaviors first, and then learn to monitor their own thinking through this write-out process. Being able to monitor one’s own outward performance would presumably be a way to internalize the process of problem solving and monitoring. Another interpretation might be that students monitored when calculating problems in their head, which caused them to slow down, and temporarily perform poorly. However, this is speculative. The initial thought was that students in the self-other treatment would “slow down” when solving problems on their own. Instead, students in both treatments solved problems faster. The difference was how quickly they solved the problems. The students in the self-other treatment showed a more gradual decrease in time than the self-treatment (See Figure 5B). The students in the self-treatment increased calculation speed immediately after the first trial. Students in both treatments spent the most time on the initial problem, possibly because it was the first one they solved. The effect of accuracy and time was short lived where the two treatments showed little difference by the posttest. Comparing calculation time and accuracy across the trials, results showed that students in the self-other treatment gradually picked up speed and improved in accuracy over time (Figure 5A). Students in the self-treatment immediately picked up speed and did worse over time.

Figure 5. (A) Accuracy Score on trials where children solved math problems on their own, (B) Time taken to solve math problems on their own

The second study built on findings from the first study. Forty elementary school students between the ages 10- to 11- years participated in a one-to-one forty-five minute session. The students participated in one treatment session, the self-monitoring treatment or the self-other monitoring treatment. Self-other treatment was compared directly to self-treatment as a between subjects design. This new design increased the number of trials and gave a better estimate for long-lasting effects.

The second experiment also involved adding a working memory measure. Working memory refers to the ability people have to temporarily encode, manage and retrieve information in carrying out cognitive tasks. For example, mental arithmetic or other cognitive tasks that involve learning, reasoning, and comprehension activities use working memory.

Monitoring one’s own problem solving was thought as a dual task demand (monitoring as one task, and solving the problem as another task) that placed a heavy burden on working memory. Therefore, a strong working memory would facilitate the ability to monitor, especially for the students in the self-other treatment that had the opportunity to practice on an external plane. A slightly different prediction was that monitoring somebody else while solving a problem increases the burden on working memory more. This would presumably occur because the individual needs to monitor the computer agents’ problem solving process while also computing on their own to check the agent’s answer. The computer agent displays the reasoning behind the problem solving so the student can follow the agent’s “lead” and check the answer. In other words, the computer agent’s performance collapses monitoring and problem solving. The overall prediction was that working memory would not influence effectiveness of other monitoring, but it would facilitate the acquisition and ability to monitor oneself in the self-other treatment.

Students also took a digit span task to see if working memory capacity affected their learning. The standard digit span task involved an examiner reading a list of random numbers and asking the student to recall the numbers in order. After the digit span task, students were randomly assigned to their treatment condition. The tasks were all multiplication problems with ten trials in each treatment. In the self-other treatment, the student took turns with the agent, and solved five trials on their own and monitored the agent for five other trials. The students were assigned to high and low working memory conditions posthoc for statistical analyses.

Regardless of working memory level, students varied in accuracy across trials. The only descriptive effect to note was that the high working memory students in the self-other treatment took more time, and showed the lowest accuracy in the beginning (See Figure 6 circled area), and later on took less time and increased accuracy (See Figure 7 circled area).

Figure 6. First few trials during the treatment session. High working memory (WM) students in the self-monitoring condition take more time, and perform the worst.

Figure 7. Last few trials during same treatment session as seen in Figure 6. High working memory (WM) students in the self-other condition take less time and perform higher accuracy.

The slow down was relatively stable for the students in the self-other treatment (See Figure 7). Interestingly, high working memory students in the self-other treatment sustained the slow down for longer than the high working memory students in the self-treatment. The same slow down pattern was found in the first study. In both studies, students early on take more time and show the lowest accuracy. One possible interpretation is that students are working the hardest on internalizing the other monitoring into their own problem solving, which cashed out on later problems. The results are not sufficiently strong to guarantee this interpretation, but they are consistent with the pattern seen in the first study. The high working memory students in the self-treatment took less time to solve the problem than low working memory students in the same treatment. The high working memory students in the self-other treatment took more time than students with low working memory in the same treatment. One possible interpretation for this effect was that students with a high working memory were trying to self-monitor after they monitored the agent. The burden of completing this dual task might have only been attempted by students with high working memory (See Figure 7).

Prior to the second study, a math test was conducted to see what kinds of calculation mistakes students made. This pre-test data was used for ProJo to make similar mistakes as students. There were five pairs of problems. The students in the self-treatment solved all problems on their own, while students in the self-other treatment took turns with the agent. During monitoring, the agent ProJo would always make three mistakes and correctly solve two problems. The three mistakes included two similar mistakes as the child (mix-up procedure and losing track in procedure), and one general calculation mistake (different from the student’s mistake). The results showed that similar mistakes were difficult to catch, while students had no trouble catching general mistakes. The students also had no trouble acknowledging when the ProJo solved the problems correctly.

Figure 8. Overall accuracy score in monitoring ProJo solving problems where some are solved correctly and others incorrectly.

Figure 9. Accuracy score in catching mistakes of ProJo that are similar to their own, or general

mistakes separated by children with low and high working memory (WM).

For the mix-up problems, students found it easier to solve it on their own than monitor the agent’s mistake. For the general calculation problems, students were able to catch the agent making a mistake more than they could catch their own mistake. Overall, students with low working memory did well compared to students with high working memory (See Figure 9). For the lose track problems, students found it easier to solve it on their own than catching the agent’s mistake. In summary for general calculation problems, students are better at catching errors than solving the problems on their own. For similar mistakes of mix-up and lose track problems, the students are worse at catching errors, and do better solving the problems on their own. One possible explanation for this result is that the general calculation errors are matters of fact. Students can remember their math facts and compare them to the agent’s computation. In contrast, for the mix-up and lose track problems, the errors are procedural, and students need to follow the procedure and check it against their own answer. This may be more difficult, because procedures are new, whereas math facts are well known. An alternative explanation is that the calculation errors are different from the mistakes the students would make, so it is easy for them to catch. The limitation of solving problems with procedural mistakes was that monitoring could be worse than solving the problem on your own. Students still seemed to benefit from monitoring general calculation problems. This was an interesting finding, but further exploration needed. The critical limitation was that the ProJo agent was successful at replaying errors, but unsuccessful in making the students notice their own mistakes.

Overall, the evidence that children learn to self-monitor by monitoring others was suggestive but not definitive. The two studies demonstrated four effects. The “Slow-Error effect” and “Late-Gain effect” found that students in the self-other treatment slowed their problem solving and made more errors after initially monitoring the agent (See Figure 6), and more so for the high working memory students. These results suggested students were trying to monitor themselves, but because monitoring added new complexities to the task, they were inefficient. However, over time, they became faster and more accurate than students in the self-monitoring condition, which suggested they had learned to self-monitor through a U-shaped curve of performance (Karmiloff-Smith, 1979).

The “External First effect” occurred in the first study (See Figure 4). On the post-test children in the self-other treatment showed a relative advantage on problems when they could write out the answers, but not for problems done in their head. In the second study, children with high working memory in the self-other treatment exhibited a post-test advantage for problems that they had to solve completely in their head. In the first study, perhaps writing the calculations out became an external representation of the “other” and helped them apply monitoring. Students may learn to monitor their external behavior first, before they can monitor their own thinking. In the second study, students with higher working memory may have been able to better manage the dual cognitive demands of completing the problem and monitoring their thinking. Therefore, they did not need to depend on the external representation of working out the problem on paper to apply self-monitoring.

The “Catch You not Me effect” indicated that students found it difficult to catch mistakes of the agent that were similar to their own, but they were successful at catching the mistakes if the errors were unlike their own (See Figure 8). The pattern was the same for children with both high and low working memory. Even though students did not always catch the agent when the mistakes were similar to their own, seeing ProJo’s errors still had a modest benefit for student learning. This was seen in the post-test where the students solved problems on their own, away from ProJo. If students made an error on their own problem, and then monitored ProJo make a similar error, the students were less likely to make the error on the post-test. If the students had not made an error, but saw ProJo make a mistake on a similar problem, this tended to hurt the student’s post-test performance. Students in the self-monitoring treatment who made an error, but did not get to monitor ProJo make a similar error, were likely to make a similar error on the post-test. These tentative results suggest a practical hypothesis that if a student gets a problem right, then the student should solve another problem on their own. If the student gets a problem wrong, have them monitor an agent. A more sophisticated version of ProJo could be designed where the system makes real-time decision to have the student monitor an incorrect ProJo, a correct ProJo, or simply continue working on their own.

The studies provided initial evidence that self-other treatment using a TBO ProJo may be an effective way to help students develop self-monitoring skills and learn better. The studies were short in duration and used only mental math techniques as the content. The studies were designed to see if there were any short-term effects of self-other monitoring. A larger intervention that lasts several months could determine if more sustained practice at monitoring over several topics would lead to stronger effects and improved learning.

The present measures of Slow-Error effect and Late-Gain effect fall short in determining whether the students were monitoring or just copying behavior. Nevertheless, the short duration and small sample studies did provide a template for larger more ambitious studies, and the results appeared to indicate that self-other monitoring a computer agent was changing behavior in ways that merits follow up.

Discussion

The pedagogical agent ProJo, which was a learning environment used to train students to self-correct by monitoring the agent make mistakes. ProJo helped students learn to monitor by explicitly displaying the reasoning during calculation. Children found it relatively natural to catch agent’s errors, more so than their own. ProJo provided a testing ground to practice monitoring. ProJo was also an environment where self-other monitoring could be compared directly to self-monitoring. Some promissory evidence showed that self-other monitoring could be an effective way to develop metacognitive skills. The present measures fall short in determining whether the students are monitoring or just copying behavior. Nevertheless, the short duration and small sample studies did provide a template for larger more ambitious studies.