How I test my exams

In my post, "How I track my to-do-list", I linked to my publically accessible Trello project templates. That Trello board has my "Creating an Exam" card list. This list has cards referring to how I test my exam, but I didn't elaborate on that process. So I thought given the season (of exams) I'll write a post about it.

First, I want to acknowledge Jan Plane at the University of MD, College Park, where I want to undergrad. I served as her undergraduate TA and, as her TA, I learned her process of testing exams. The bulk of my system I learned from her with some adaptations to serve my purposes.

Two Rounds of Testing

I have two rounds of testing that basically look identical. I have two of my TAs, usually from my head staff (head undergrad TAs and grad TAs) that take the exam with a timer. They note:

  1. How long each coding problem (or problem set if they are quick questions) takes them to read and answer.
  2. Any typos, confusing wording, etc.

After each round, I first go through their feedback and, if needed, I fix the wording, add examples, and rearrange questions. Then, I go through the TAs' answers. If their answer is incorrect, I go back to that TA and ask why. Usually, they read the problem too fast. Other times the problem wording is confusing and needs to be fixed. I also note differences in their coding solutions to take into account for the grading rubric.

Each round has a different set of TAs. I have two rounds so that the second round is checking the changes I made due to round 1. Changes are almost always needed after round 1. On rare occasions, I will also make a new problem between rounds 1 and 2. Adding a problem usually happens because a TA rightly points out that I've forgotten to test the students on a critical concept. The second round is generally more about getting the timings and final checks for wording.

Exam Finalization

After both rounds, I finalize the exam in terms of length and number of points. For the exam's length, I take the average time across all TAs for each problem, with maybe removing a time if it's an extreme outlier. Grad TAs are the most likely to have an outlier timing because they, unlike the undergrad TAs, have not taken the class. I multiply the average by 3 to get an estimate of how long the 95th percentile student will likely take on the exam.

I then remove problems (my exam drafts are almost always too long) until my total time is under how much time the students will have for the exam. If a problem had a large variety of answers from the TAs, that makes it more likely to be removed. A wider range of answers by TAs usually means an even wider array from students (who are even more inventive), which will make the problem harder to grade. Some of the problems I remove are saved for future exams.

My general rule of thumb for assigning points is 1 point per minute of the exam. I do have a minimum number of points for each kind of problem to make sure I have enough points for the grading rubric. Otherwise, it's about how many minutes I think the 95th percentile student will take rounded. It's not an exact science. What I'm attempting to do is to fairly allocate effort (measured by time) with points earned.

Timeline

My overall philosophy is to give everyone and myself enough time to do everything that is needed and not require work to get done on weekends. Therefore, I only count business days and pad in days where I can. So the overall days for each phase are:

  • 2.5 business days for round 1 testing by my TAs
  • 0.5 business days to address round 1 feedback
  • 2.5 business days for round 2 testing by my TAs
  • 0.5 business days to address round 2 feedback and finalize the exam
  • 2 business days for copying the exam (ours is still on paper)

And yes, this adds up to 8 business days and round 1 gets a full draft of the exam. This means I have a close to finished draft two weeks before my students take the exam. Moreover, creating an exam takes me about 13 hours, which I usually disperse over two weeks because large blocks of uninterrupted time don't exist for a professor. This means I start working on my exams a whole month before my students get it, and I already feel late if I haven't begun brainstorming by then.

Conclusion

I hope this helped give you ideas on ways to become more confident in your exam's level of rigor. And I acknowledge that a process like this requires planning, which is something we all struggle with. For myself, I enjoy planning (it's sometimes how I procrastinate). And planning is a way for me to reduce the risk of getting frazzled in a job that pulls me in many different directions.

Do you have a process for testing your exam? What do you do?

Comments

Popular posts from this blog

How I track my to-do list

How I Track The To-do List For My 200+ Student Class

GPS Syndrome