ITiCSE 2021 Paper: CS1 Reviewer App

 ITiCSE 2021 is here! And this year, I’ve got a paper with Anshul Shah, Jonathan Liu, and Susan Rodger. The first two were undergraduate students. They are now both graduated and going on to grad school to study computer science education! Anshul is going to UCSD and Jonathan to Chicago.

[Also posted on medium.]

The paper is “The CS1 Reviewer App: Choose Your Own Adventure or Choose for Me!” and is a tools paper. The paper introduces an app that Anshul originally developed with a partner in their Fall 2019 database class. We’ve been developing the app ever since with help from other undergraduate students through Duke’s summer CS+ program. Jonathan started helping last Fall. Below is a screenshot, the abstract, and a discussion of changes from the last six months since we submitted the paper. You can also check out the tool yourself at https://cs-reviewer.cs.duke.edu/

Abstract

We present the CS1 Reviewer App — an online tool for an introductory Python course that allows students to solve customized problem sets on many concepts in the course. Currently, the app’s questions focus on code tracing by presenting a block of Python code and asking students to predict the output of the code. The tool tracks a student’s response history to maintain a “mastery level” that represents a student’s knowledge of a concept. We also provide an option of answering auto-generated quizzes based on the student’s mastery across concepts. As a result, the tool provides students a choice between creating their own learning experience or leveraging our question selection algorithm. The app is supported on traditional webpages and mobile devices, providing a convenient way for students to study a variety of concepts. Students in the CS1 course at Duke University used this tool during the Spring and Fall 2020* semesters. In this paper, we explore trends in usage, feedback and suggestions from students, and avenues of future work based on student experiences.

* Addendum: And Spring 2021 too now.

Updates since submission

Bayesian Knowledge Tracing (BKT). The auto-generated quizzes choose questions based on a student’s topic’s mastery score (lower mastery meant more likely to pick a question from that topic). The mastery score was initially a rolling average of a student’s performance for each topic on a quiz (details in the paper). We’ve now changed the mastery score calculation to use BKT (Catherine Yeh and Iris Howley created a visualization to help understand it too), which we believe to be a more accurate and versatile measurement of a student’s topic mastery.

Wrong answers tagged with why incorrect. All questions are machine-generated, i.e., we wrote code to write questions (like question templates with randomized mad-libs parts). Some templates could have more potential options than shown (questions currently have one correct answer and three incorrect), so it also randomly chose which ones to use. This past semester we updated the app’s backend to associate a “tag” with every wrong answer option. This tag(s) represented why the option was wrong. It’s an application of my prior work from ICER 2017. A wrong answer could have multiple tags. The intention behind it is to aggregate how the students are doing at the tag level instead of only correctness per topic.

More questions. We are also constantly writing more question templates and using them to add more questions to the app. We’ve added recursion with lists for Spring 2021 (see screenshot below). And this summer we are in the process of writing booleans, if…elif…else, and for loops. One idea is to open source the code for the question templates. We designed them to easily export into whatever format you need. My original plan included using the code for the app and generating questions for my learning management system to create quizzes for my students. If you are interested in the project, let me know in a comment or email me!


Comments

Post a Comment

Popular posts from this blog

How I track my to-do list

This is also on medium . I use Trello  to track all the things I need to do. Trello, at its core, is just a to-do list management tool. It is versatile to many organization strategies and is great for a single person or a team to organize the tasks that need doing. However, this versatility can be very daunting as a novice because you are just starting to figure out a system to keep track of what you need to do. And a novice rarely does well in an environment where there are many ways to do the same thing, a.k.a. a versatile system. A beginner does not have enough experience to determine what technique is better than another. Moreover, a novice finds it challenging to keep track of the many ways to do the same thing. So, if you are a novice, here are my suggestions for how to use Trello to track your personal to-do list (how I use it to organize my research team(s) and teaching team(s) I'll leave for a different blog post). First, I'll go over the fundamental way I use Tre...
Read more

GPS Syndrome

A term I made up and have been using for a while is "GPS syndrome." And it turns out that enough people have picked up this term that I feel like I need to post an "official" definition. I most commonly use the metaphor when talking about and with teaching assistants (TAs) that overhelp students. So this blog post is mostly from a google doc I wrote for new TAs for one of their training sessions. Definition " Scaffolding refers to a variety of instructional techniques used to move students progressively toward stronger understanding and, ultimately, greater independence in the learning process. The term itself offers the relevant descriptive metaphor: teachers provide successive levels of temporary support that help students reach higher levels of comprehension and skill acquisition that they would not be able to achieve without assistance. Like physical scaffolding, the supportive strategies are incrementally removed when they are no longer needed, and the...
Read more

How I Track The To-do List For My 200+ Student Class

Image
This is also  posted on medium . This is post #4 of a 5 post series on how I organize the teaching staff for my 200+ student class. This post discusses how I track all the class’s tasks. I start with an overview of Trello , the tool I use to track tasks. Next is a discussion on my overall task management approach. I then move on to how I use Trello to structure the task lists, common and recurring tasks, and a task’s life cycle. Finally, I close with the advantages I see for having an externalized task list. The other posts in this series are as follows (I’ll update with links as I post): Overview Teaching staff roles How I communicate with my teaching staff How I track the to-do list (You are here) How I run the head staff meetings Terminology Rather than require you to read my teaching staff roles post , here is a quick refresher: Teaching Associate (TA+)  — A full-time department staff member (not a student) assigned to my class. She serves as my right-hand person with many and...
Read more