When AI Helps Students Learn and When It Doesn't: A Groundbreaking Study by an Emory Professor

When AI Helps Students Learn And When It Doesn’t: An Emory Professor’s Groundbreaking Study Time for a quiz. Which of these four learning environments, involving artificial intelligence — or not — leads to the best academic outcomes for university students? AI-generated content and AI delivery AI-generated content and human delivery Human-generated content and human delivery Human-generated content and AI delivery To find the answer, Rajiv Garg, a professor at Emory Goizueta Business School (Atlanta, USA), enlisted nearly six dozen students from various disciplines for an in-class experiment. What he found might surprise you and change your mind about AI’s potential in the classroom – for MBAs, B-school students, and everyone else. Results show huge benefits of AI in the classroom Garg, an associate professor of information systems and operations management at Emory Goizueta University, conducted a pilot study from last semester until last week, comparing AI-generated courses with human-generated courses. First, he had a master’s in data science graduate create and deliver an online course on the database creation language SQL. Then he asked ChatGPT to create a course as well. Then he had the graduate deliver the AI-created course and an AI avatar deliver the human-created course. In the end, students took four courses: an all-AI course, an all-human course, an AI course with a human voice/avatar, and a human course with an AI voice/avatar. Garg and his team ran the experiment with 40 students last semester and another 30 this spring. For the courses where AI content was delivered, the team created an avatar (via HeyGen) and fed it a script. Each group in Garg’s study completed the course and took a short quiz at the end. The result: Garg and his team found that student performance was highest in the course where human-generated content was delivered by an AI voice/avatar. Second best was an all-human-generated course, followed by an all-AI-generated course – and worst was an AI-generated course with a human voice/avatar. The most important finding: Course content created by humans – experts and even average instructors – is still better than AI-generated content. But the content can be delivered more effectively by an avatar. “I found that students who took an all-human-generated and delivered course scored, on average, 5.7% higher on quizzes than students who took an all-AI-generated and delivered course,” Garg tells Poets&Quants. “Furthermore, students who took a hybrid human-AI course scored, on average, 4.3 additional points compared to an all-human-generated course. Finally, students who took a hybrid AI-generated and human-delivered course scored, on average, 2.7 points less than in an all-AI-generated and delivered course. Thus, human-generated content is superior to AI-generated content for higher education, while AI-generated delivery – voice and avatar – can improve student learning.” It was not the result he expected. “For one hypothesis, I was unsure whether AI or human content would be better, as we are still exploring the quality of generative AI, but I expected human delivery – voice and avatar – to perform better than AI voice and avatar. I was proven wrong – although I am not particularly surprised, as my research also shows that voices can play a role in information seeking.” Quiz results in Rajiv Garg’s study AI-generated content and AI delivery 79.2% AI-generated content and human delivery 76.5% Human-generated content and human-generated content 84.9% Human-generated content and AI delivery 89.2% Why did B-school students learn best when the content of an online course was human-generated but delivered by an artificial voice and avatar? According to Garg, this is partly due to a common practice among students in online programs – watching lectures at 1.5x speed – and what this does to the modulation of a teacher’s voice. “If you watch content at 1.5x speed and your voice is modulated, it sounds worse,” Garg says. “But if your voice is flat, you can still understand it well. So I gave them the freedom to change the speed, but I didn’t think they would. In the post-course survey, I talked to some students and asked, ‘Why do you think this AI voice was better for your learning?’ And they said they listen at 1.5x speed and it’s so easy to understand what it’s saying. I thought, ‘I’ve never thought about that.’” There are other issues, he adds, including the impact of accents on learning. And while it’s possible – and quick – to simply use an avatar provided by a system like HeyGen, students respond better to an avatar created from the visuals of an existing teacher. The avatar can be designed to match the expert’s appearance, mannerisms, and voice, including their accent. The benefits are manifold: Instructor avatars can deliver content in any language – from Mandarin Chinese to French, from Arabic to Hebrew – reaching more students in their native language. This is exactly what Garg’s bos