NAACL 2025 Outstanding Paper Award

Congratulations to Insource Team Lead Ryan Knight for Outstanding Paper Award recognition at NAACL 2025

Insource Team Lead Ryan Knight was recognized as a co-author of an Outstanding Paper at The Nations of Americas Chapter of the Association for Computational Linguistics (NAACL) 2025!

Ryan and his coauthors received the award for research into the practical applications of AI in education. The paper, titled, “DrawEduMath: Evaluating Vision Language Models with Expert-Annotated Students’ Hand-Drawn Math Images“, evaluated how well AI can understand handwritten student work in math.

Working out math problems by hand remains an essential part of learning math for children everywhere. There is a large body of research showing that students learn better through handwriting. Many teachers are relying on handwritten math even more in the age of AI, to ensure that students understand the work they are doing.

However, teachers are left with a hard choice: have students do deep thinking, handwritten work, and evaluate that work themselves, or use Ed Tech products to have students do work on computers that is automatically graded.

Vision language models (VLMs) are AI models that can understand images. VLMs have the potential to be game-changing for handwritten math work, allowing teachers to get the best of both worlds: deep student thinking and technology to help save teacher time.

In order for teachers and ed tech companies to trust VLMs, we need a way to evaluate their performance. In order for AI researchers to improve VLMs on education use cases, they need stronger signal about where the models need to improve.

The DrawEduMath benchmark is a way to test how well VLMs can understand handwritten math work. The benchmark leveraged an open source dataset of handwritten math images from the ed tech platform ASSISTments. The team hired expert teachers to annotate the image with their insights about what the images show about student thinking. The benchmark compares the agreement of VLMs to the expert teacher insights on specific questions about what’s happening in the image.

The result is a leaderboard of VLM performance over time. There remains significant gaps in VLM’s ability to go beyond literal descriptions of what’s happening in the image to identify pedagogical concepts and student misconceptions.

The winning team includes:

• Sami Baral (corresponding author), Worcester Polytechnic Institute
• Li Lucy, University of California Berkeley
• Ryan Knight, Insource Services, Inc.
• Alice Ng, Teaching Lab
• Luca Soldaini, Worcester Polytechnic Institute
• Neil Heffernan, Worcester Polytechnic Institute
• Kyle Lo, Allen Institute for AI (Ai2)

The 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL) awarded 10 Outstanding Papers out of more than 1000 accepted papers.

Please join us in congratulating Ryan and his collaborators on this outstanding achievement! We can’t wait to see the continued impact of their important work in education technology and AI.