How I test my exams

In my post, "How I track my to-do-list", I linked to my publically accessible Trello project templates. That Trello board has my "Creating an Exam" card list. This list has cards referring to how I test my exam, but I didn't elaborate on that process. So I thought given the season (of exams) I'll write a post about it.

First, I want to acknowledge Jan Plane at the University of MD, College Park, where I want to undergrad. I served as her undergraduate TA and, as her TA, I learned her process of testing exams. The bulk of my system I learned from her with some adaptations to serve my purposes.

Two Rounds of Testing

I have two rounds of testing that basically look identical. I have two of my TAs, usually from my head staff (head undergrad TAs and grad TAs) that take the exam with a timer. They note:

  1. How long each coding problem (or problem set if they are quick questions) takes them to read and answer.
  2. Any typos, confusing wording, etc.

After each round, I first go through their feedback and, if needed, I fix the wording, add examples, and rearrange questions. Then, I go through the TAs' answers. If their answer is incorrect, I go back to that TA and ask why. Usually, they read the problem too fast. Other times the problem wording is confusing and needs to be fixed. I also note differences in their coding solutions to take into account for the grading rubric.

Each round has a different set of TAs. I have two rounds so that the second round is checking the changes I made due to round 1. Changes are almost always needed after round 1. On rare occasions, I will also make a new problem between rounds 1 and 2. Adding a problem usually happens because a TA rightly points out that I've forgotten to test the students on a critical concept. The second round is generally more about getting the timings and final checks for wording.

Exam Finalization

After both rounds, I finalize the exam in terms of length and number of points. For the exam's length, I take the average time across all TAs for each problem, with maybe removing a time if it's an extreme outlier. Grad TAs are the most likely to have an outlier timing because they, unlike the undergrad TAs, have not taken the class. I multiply the average by 3 to get an estimate of how long the 95th percentile student will likely take on the exam.

I then remove problems (my exam drafts are almost always too long) until my total time is under how much time the students will have for the exam. If a problem had a large variety of answers from the TAs, that makes it more likely to be removed. A wider range of answers by TAs usually means an even wider array from students (who are even more inventive), which will make the problem harder to grade. Some of the problems I remove are saved for future exams.

My general rule of thumb for assigning points is 1 point per minute of the exam. I do have a minimum number of points for each kind of problem to make sure I have enough points for the grading rubric. Otherwise, it's about how many minutes I think the 95th percentile student will take rounded. It's not an exact science. What I'm attempting to do is to fairly allocate effort (measured by time) with points earned.

Timeline

My overall philosophy is to give everyone and myself enough time to do everything that is needed and not require work to get done on weekends. Therefore, I only count business days and pad in days where I can. So the overall days for each phase are:

  • 2.5 business days for round 1 testing by my TAs
  • 0.5 business days to address round 1 feedback
  • 2.5 business days for round 2 testing by my TAs
  • 0.5 business days to address round 2 feedback and finalize the exam
  • 2 business days for copying the exam (ours is still on paper)

And yes, this adds up to 8 business days and round 1 gets a full draft of the exam. This means I have a close to finished draft two weeks before my students take the exam. Moreover, creating an exam takes me about 13 hours, which I usually disperse over two weeks because large blocks of uninterrupted time don't exist for a professor. This means I start working on my exams a whole month before my students get it, and I already feel late if I haven't begun brainstorming by then.

Conclusion

I hope this helped give you ideas on ways to become more confident in your exam's level of rigor. And I acknowledge that a process like this requires planning, which is something we all struggle with. For myself, I enjoy planning (it's sometimes how I procrastinate). And planning is a way for me to reduce the risk of getting frazzled in a job that pulls me in many different directions.

Do you have a process for testing your exam? What do you do?

Comments

Post a Comment

Popular posts from this blog

How I track my to-do list

This is also on medium . I use Trello  to track all the things I need to do. Trello, at its core, is just a to-do list management tool. It is versatile to many organization strategies and is great for a single person or a team to organize the tasks that need doing. However, this versatility can be very daunting as a novice because you are just starting to figure out a system to keep track of what you need to do. And a novice rarely does well in an environment where there are many ways to do the same thing, a.k.a. a versatile system. A beginner does not have enough experience to determine what technique is better than another. Moreover, a novice finds it challenging to keep track of the many ways to do the same thing. So, if you are a novice, here are my suggestions for how to use Trello to track your personal to-do list (how I use it to organize my research team(s) and teaching team(s) I'll leave for a different blog post). First, I'll go over the fundamental way I use Tre...
Read more

GPS Syndrome

A term I made up and have been using for a while is "GPS syndrome." And it turns out that enough people have picked up this term that I feel like I need to post an "official" definition. I most commonly use the metaphor when talking about and with teaching assistants (TAs) that overhelp students. So this blog post is mostly from a google doc I wrote for new TAs for one of their training sessions. Definition " Scaffolding refers to a variety of instructional techniques used to move students progressively toward stronger understanding and, ultimately, greater independence in the learning process. The term itself offers the relevant descriptive metaphor: teachers provide successive levels of temporary support that help students reach higher levels of comprehension and skill acquisition that they would not be able to achieve without assistance. Like physical scaffolding, the supportive strategies are incrementally removed when they are no longer needed, and the...
Read more

Revisit: How I organize the teaching staff of my 200+ student class

 When it comes to anyone’s advice on how to run your own course, your “mileage may vary.” So, I thought it would be interesting to revisit my series on how I organize my teaching staff in my 200+ student class . The first post of that series is my most viewed post on Medium. I published that series during the second half of 2020, and times have changed since then (**cough** pandemic **cough**). In addition, I’ve moved on from that introductory computer science (CS1), and I primarily teach an elective data science (DS) course, which has a high variability in enrollment. In this post, I revisit that series from the perspective of what I still do versus do not between my CS1 and DS courses. [ Also posted on medium. ] Current Course’s Context Before I start my comparison, I need to go over how the context is different since it influenced many of the differences between the two courses. My DS class has a semester-long project and no labs/discussions. The class also uses Python, whi...
Read more