Learning with GenAI

Although studies consistently show that students are more capable of producing functional code with an AI assistant, there is conflicting evidence on how students perform when the AI assistant is taken away.  In one study, students who learned to program with GenAI performed better than students who learned without GenAI, even when GenAI was later taken away.  However, new meta-cognitive challenges emerge for students writing software with the aid of GenAI, and students report feeling over reliant on the tools. 

GenAI in Software Engineering

GenAI has been used in a software engineering course focused on understanding, managing, and testing large code bases. Students more often used the GenAI chat interface rather than the code-completion interface, demonstrating the rapid pace of change in how students interact with GenAI.

GenAI as an Instructional Assistant

In a number of studies, GenAI has been shown to be valued by students as a tutor. Precisely how GenAI should behave is an open area of inquiry, with some studies providing a GenAI assistant trained not to give away answers, and research offering students answers that are usable only when they engage with the code in a meaningful way.

GenAI on Intro CS Assignments

A 2023 paper demonstrated that large language models were already performing well on introductory computer science assignments, and the AI has dramatically improved since that time. You can generally assume that any reasonable GenAI today can solve your intro CS assignments.

GenAI on Programming Exams

Studies show that while older models like Codex performed reasonably well on programming exams, more recent models like GPT-4 can achieve scores comparable to top students, raising concerns about the validity of unproctored exams.

Novice Interaction with GenAI

Research indicates that novice programmers often struggle to effectively communicate with LLMs, missing crucial details in prompts and making superficial changes during revisions, highlighting the need to teach prompt engineering skills.

Metacognitive Challenges with GenAI

A study observed that students using LLMs faced different metacognitive challenges compared to those without, although task completion rates increased significantly. A key concern raised is whether students taught with LLMs can succeed without them, suggesting potential over-reliance.

Engaging with AI-Generated Code

Exploratory research has investigated various techniques designed to increase students' cognitive engagement with AI-generated code, aiming to slow down the process and encourage deeper understanding before integrating the code.

Metacognitive Demands of GenAI Use

Working with GenAI systems imposes significant metacognitive demands on users, requiring monitoring (e.g., self-awareness of goals, adjusting confidence) and control (e.g., metacognitive flexibility, task decomposition), akin to a manager delegating tasks.