What I Learned Today

I’ve been experimenting with some of the new massively multiplayer online course offerings from Coursera. In the spring, I took Cryptography, and I am now taking Fantasy and Science Fiction: The Human Mind, Our Modern World. These courses are offered for free, for anyone who wants to take them. I’ve been curious about the (eventual) business model, since there will have to be some way to recoup the investment in web site architecture and content. The lectures do seem to be “record once, replay forever”, but it’s still a big effort to do up front.

One way they’ve kept the ongoing costs reasonable, though, is by offloading one of the biggest time consumers in traditional education: grading. The Cryptography course was conducted in an entirely auto-grading mode. The homeworks each week were a series of multiple-choice or fill-in-the-blank questions. The feedback was actually quite good from these exercises — if you got something wrong, there might be a clue as to where to look, and if you got it right, there was usually an explanation, which you could learn from if you’d just gotten lucky with your choice. Further, you could attempt each homework 4 times, a process designed to encourage “mastery” (progressive learning). I know what you’re thinking. Four tries on a multiple-choice test should basically ensure you get 100%, since you could explore all possible options. Not so! They’ve made the process more sophisticated, producing a new mixture of answers for each question each time you attempt it. You really do have to think through the problems each time. I approve.

The F&SF class is different. Our assignments consist of 300-word essays, which can’t be auto-graded (with any real reliability). First I must note that I found WRITING a 300-word essay to be particularly challenging. How can you say anything of substance in 300 words? How can you call out something of interest in a 400-page book using only 300 words? But, as in haiku, the limitations of the medium are themselves a spur to inspiration. So then, how to grade them? Coursera has adopted a peer grading strategy, in which you are assigned to grade a random set of your classmates’ essays, and your essay correspondingly is sent to a random set of peers to be graded. In this class, we’re required to grade four peers, but allowed to grade more. The grading itself is very coarse: you assign one score for Form (grammar, organization, etc.) and one for Content. Each can be given a score of 1 (poor), 2 (average), or 3 (exceptional). You are also required to provide some text feedback.

So far, I found the grades I received from my peers to be fair, but I don’t think I’ve learned much from them. Most of the feedback was compliments, with a few rather surface-level critiques, rather than the kind of feedback you’d get from a professor or TA. But one reason for this is the bizarre organization of this particular class. You are required to do the reading, write an essay blind (on no suggested topic, simply something that “will enrich the reading of an intelligent, attentive fellow student”), and only THEN are you permitted to view the professor’s videos with his analysis of the readings. Perhaps this is intended to reduce “bias” from the instructor, but ultimately all it does is set you up to be evaluated tabula rasa (with respect to the course content), so I don’t see how the assessment has anything to do with what you have learned. These should be pre-tests rather than the sum of the grade. With the current scheme, the lectures themselves unfortunately become less of a priority, because by the time they’re available, you should already be moving on to read and plan an essay on the next reading. That’s a shame, because Dr. Rabkin is clearly a thoughtful and knowledgeable source. I’ve found most of his lectures to be interesting and thought-provoking (even though I disagree violently with some of his analysis of Grimm’s Fairy Tales! Ugh!). So, two weeks in, I’m not very enamored of this kind of peer grading. I hope Coursera continues to experiment with new strategies.

You can check out Coursera’s statement of pedagogy in which they explain their design choices and include references to some external work on their efficacy. It’s mostly reasonable arguments. I’m on board with the mastery learning comment, for example. However, I found the argument for peer grading to be weaker. The main motivation (never articulated) has got to be the challenge of providing feedback for thousands (or hundreds of thousands) of students, which is a scaling issue. Instead they cite research on the benefits of peer review, which are valid, but I think never intended to be the SOLE source of feedback for students, and the strengths of crowd-sourcing, which depends on large numbers for reliability, which four random grades from others in the class don’t provide. I’m not asserting that this is an invalid method of instruction, but I’m not convinced by the evidence they’ve offered.

In working through these courses, I’ve already gotten ideas for how I would experiment creatively with this new teaching medium. Watching slides is boring. Watching a talking head is boring. I love, however, the occasional pauses that require you to answer a question (pop quiz!) to proceed. It’s great for capturing attention that may have been wandering. The Cryptography class made good use of these. The F&SF class doesn’t use them at all. If I were teaching, I’d also bring in props or direct students to relevant websites or otherwise increase the level of activity and interactivity as much as possible. Right now, the only interaction in the F&SF course is through the essays (anonymized) and the discussion forums (which no one can keep up with). I’d like to foster more interaction with the professor, without inundating that person. I think well crafted video lectures can improve on this front.