Soft, hard, and late deadlines (or another failed experiment)

After my side channel experiment that wasn't worth keeping, it's probably no surprise that I did yet another crazy experiment! This one has a bunch of the usual elements. A naive teacher with delusions of lots of energy who thought this was a good idea (that's me). A class structure for who that teacher (me) wants the students to be, not how typical human students would behave.

This time I messed with the deadline structure in my CompSci216 class, Fall 2021. This class is data science for students that have finished a CS2 level course or have enough programming and statistics that they'll be able to make up for having less programming experience.

[Also posted on medium.]

I knew going in that it wasn't the greatest of ideas. But I also thought it was an experiment worth trying (just like the side channel experiment). Moreover, I couldn't think of a way to change it to increase the likelihood of success, so I ran with the idea anyway. I again framed it to the students as an experiment worth trying that would also change based on how they all behaved.

Syllabus Setup

My full syllabus is on the class's website. For some background, the content is organized around modules that are about one week's worth of work. Each module has four pieces. Prepare is where students consume videos and reading materials and includes knowledge check quizzes. In class, they do a Group Worksheet that I create using just-in-time teaching where it only has concepts the students did poorly on in the knowledge check quizzes (Note: I've turned these into peer instructions after this semester). After the Group Worksheet, they start working on a Practice during and after class, which is basically homework that does not require a perfect score to receive full credit. They are also encouraged to work together on it. Finally, there is the Perform which must be done independently and confirms the students understand the material for that module. The Perform for a module was released one week into the following sprint. This was so that the Practice (hard due at the end of its learning sprint) could be graded and returned before they started working on the Perform.

I released two modules at a time and they ran mostly in parallel during a two-week Learning Sprint. The modules were usually either coding heavy or statistics heavy. For each sprint, I either had one coding and one statistics module or two coding modules. And three sprints had a single module and a project workshop week. My intention was to accommodate the wide variety of student backgrounds. The common denominator is at least 1.5 semesters worth of coding. The computer-science-focused students usually had 2+ semesters and little/no statistics, while the more statistics-focused students had a CS1 and some statistics class that had coding.

Since the students in the class had a wide variety of backgrounds (most were from computer science, but there was a noticeable number with some coding background but from statistics), my intention behind the sprints was to release two coding heavy or one coding heavy and one statistics heavy module. That way students could spend more time on the module they were weaker at without being constrained by the one week per module restriction. Yes, this course is more geared towards those with a coding background, but after all, it is a computer science course.

Deadlines

I had three kinds of deadlines, though each modules pieces only had two of the three:

• Soft deadline - If you didn't make this deadline, it's okay you are not late. These are for Prepares and Group Worksheets. The main natural consequence of not doing these on time is the in-class activities will not benefit the student as much.

• Hard deadline - All module pieces had this deadline. This deadline needs to be met or the student is behind the intended cadence of the class.

• Late deadline - This is the last date we would accept work on the Prepare, Practice, and Perform module pieces.

I did not take off points for late work. I want a student's grade to represent their learning, not their learning + being able to turn stuff in on time.

Due dates went like the following with the two modules labeled A and B:

• Prepare

• Soft due - Prepare A the day before the first day of class that week and Prepare B was the day before the second day.

• Hard due - Originally, at the end of the sprint. After a few sprints, at the end of the first week of the sprint.

• Late due - 24 hours after the hard due date.

• Group worksheet

• Soft due - The day we used it in class.

• Hard due - At the end of the first week of the sprint.

• Late due - No late due date.

• Practice

• Soft due - Not applicable, though I strongly encouraged them to get at least one module's Practice done by the end of the first week. Since there would be Performs from the prior learning sprint released during the learning sprint's second week.

• Hard due - At the end of the learning sprint.

• Late due - One week after the hard due date.

• Perform

• Released - One week into the following learning sprint. 

• Soft due - Not applicable

• Hard due - At the end of the learning sprint it was released.

• Late due - One week after the hard due date.

Here is the graphic I made at the beginning of the semester to help them understand things.

Picture showing my students how the deadline system worked

The main consequence of not meeting the hard deadline is they now have more stuff to do at once than if they are trying to stay on pace with the course. Moreover, we generally promised to get graded feedback returned within 1 week. So for the Practices, they would receive the graded feedback up to a week later than if they submitted it by the hard deadline, making it not available when working on the Perform during the intended window. Leading to a potential cycle of always being late.

Just writing out that bulleted list makes me wonder more how I didn't see that this might not be a good idea. Though on the other hand, I did say this story included a naive teacher (points at self).

Two Versions of Everything and NULLing Out Submissions

On top of all that, I added an extra complication. I created two versions of basically everything except the Group Worksheets. For the Prepares, there were versions for the soft and hard deadlines. For the Practices and Performs, there were versions for the hard and late deadlines. The titles clearly labeled them as one or the other and I thought this would "tap into" the student's desire not to be late by clearly labeling when they were not on track.

However, a problem with this system is a student could submit the Practices and Performs to both versions and potentially cause duplicate work for the TAs when they did the hand grading part of the assignments. Rather than trying to keep the two versions aligned on the teaching staff end, I required the students to fill out a "NULLing" form within two days of the hard deadline. This would tell us to mark their submission to the on-time version of the assignment as NULL so we would not grade it.

What Happened in the End?

I'm sure anyone that has taught students can make some guesses on what happened. As I reflect, it is definitely the case I created a system that if I was a student would have had no problem following, but the system was excellent at differentiating students on their time management skills. Not in terms of their grades, but mainly in terms of when they submitted things.

To start on a positive note, there were students that really liked and appreciated the system. I think quite a few noticed I was struggling with whether the system was a good experiment and so they came to me after class or emailed me how much they appreciated the system's flexibility in helping them get through rough patches in the semester. Others talked about how they always strove to make the soft/hard deadlines but valued they had wiggle room.

I also had quite a few students that hated it. This appeared in my mid-semester survey and end-of-semester evaluations. Unsurprisingly, though, in the mid-semester survey very few answered the multiple-choice question that they wanted to change it completely. They more chose the option to keep it or only have a slight tweak. Some didn't even want me to change the tweak of having the Prepares due the first week of the sprint.

About 60% on any given module treated the late due date as the actual due date. Some even procrastinated an entire two-week sprint's worth of work to the last day it was due. Such behavior led to quite a few extension requests despite having literally 3 weeks to do much of the work.

Finally, needing to NULL out their submission if they submitted to both versions was confusing. We were strict about this requirement and would NULL a student's late submission if they did not request for the on-time submission to be NULLed because by that point the TAs had graded that submission. I did not want to do it, but I also did not want to create a situation where students could keep forgetting and I was putting more work on my TAs. After a student forgot once and had their on-time graded instead of late submission grade, they never forgot.

Costs and Benefits

There were definitely costs that I thought were reasonable, but on reflection cost more than I thought:

• Student confusion - This was definitely something I think I always underestimate because in my mind it's "clearly obvious" because I came up with the system. But students were definitely confused. They submitted to the NULL form when they shouldn't or didn't submit when they should. They also asked questions about why they had a score in both versions of the assignment, even when they did the right thing (Gradescope doesn't let us "retract" a submission so the score was a 0) plus the gradebook needed to have a score in both columns regardless of what a student did.

• Extra work - I did extra work creating both versions. My TAs did extra creating both versions. I tried to compensate for the confusion by giving extensions, which sometimes resulted in me doing the grading since I didn't want to keep asking my TAs to grade late stuff.

• Tracking NULLing - This had a much higher cost than expected. Mainly because I didn't expect so many students to submit to this form. On reflection, I should have known better.

There were benefits, though honestly, this is where the title comes more into play. In some ways, these are great benefits but they really don't outweigh the cost. Or in other words "this is why students can't have nice things" or "a bad apple spoils the barrel."

• Student flexibility - Many students appreciated it. I got emails from and had conversations with more students than ever before about how much they really appreciated that I was trying to be flexible.

• But the caveat here is this system really only benefited the students that had time management skills. Those with poor/undeveloped time management skills were actually worse off in this system.

• The late version was a good backup - Every assignment always had at least one student, and usually more, that made some mistake in their submission. It wasn't fully run, it was the wrong file, etc. For those that submitted on time, this wasn't an issue since we just told them we'll NULL this on-time version out and they can just submit to the late version.

Conclusions

First and foremost, I'm never doing this kind of system again. I said never again after the semester was over and after writing this reflection I'm underlining that thought with multiple strokes of red ink. And if I even consider any of the elements I'm going to force myself to reread this reflection.

I'm teaching the same class this semester and I got rid of almost everything. We are doing one module a week, with posting early when we can since the material is a lot more stable. There is only a single hard deadline. That deadline does have a one-week late window and this time I'm experimenting with slip day tokens that so far (as of week 6 of the semester) seems to be going well. Maybe I'll write a blog post about that when the semester is over. I also have two undergrad TAs keeping track of it for me. The knowledge check quizzes only have one version with a hard deadline and a 1-day late window.

Finally, my goodness, I'm impressed you made it this far. Or one way to spin it is you got here because I made this trainwreck of a story entertaining enough you couldn't look away 🙃 Anyway if you are here, thank you for reading. If you have any thoughts, questions, or concerns, feel free to leave a comment.