Remembering My Hat

21st September 2009

Learning to assess: Assessing to learn

Filed under: Uncategorized — rememberingmyhat @ 11:25
Tags: , , , , , ,

(Image nicked from studenthacks.org/2008/01/29/conclude-essay/)

Last week the K101 course team had an all-day meeting to discuss the assessment questions for the next couple of presentations of the course. Each person who is responsible for a Block (about a month’s work, a sixth of the 60 point course) had produced suggested questions and guidance for students and tutors for their material. On this occasion we each produced 4 continuous assessment (TMA) essay questions and 1 exam question. We pretty much spent all day discussing the draft questions and their guidance in fine detail. It’s very much like the process of constant critcism and redrafting we used to produce the materials in the first place, although with considerably fewer iterations.

Exam and assessment questions are much harder to write than you might think – it’s very easy to set something misleading, or unsuited to students of varying ability, or insufficiently related to the course materials. And because this is basically the same material presented over many years, you quickly run into the danger of repeating yourself. It’s very important to avoid that when you know people sell past essays for your course on eBay!

I value very highly the co-operative and collegial atmosphere in which these sort of discussions (mostly!) happen. It’s not easy having your work torn to shreds, but people mostly seem to accept that it is done with good intentions and that the end result is immeasurably better for it. I was particularly impressed with the good grace with which a senior and very experienced colleague accepted having to go away and rewrite almost all of their questions. I like the way it seemed we were all focused on the end product and not on our egos or pet interests.

Assessment is of course vitally important in any educational context since, famously, assessment defines the de facto curriculum (Rowntree, 1977). But it’s particularly important in an OU context because feedback on essays is one of the main vehicles of individualised teaching. Students (usually!) get very substantial comment on their work and suggestions for how to improve it. And tutors get regular feedback on their marking and how to improve the ways they teach through marking. The feedback chain does come to a stop with the people who give the feedback to the tutors (Monitors), since we are not assessed, but we do get trained specifically in being a monitor.

One of the course co-Chairs, Anthea Wilson, had produced a paper to help us design the questions (with, I think, some input from Andy Northedge, who was the course Chair while we were producing the course). With her permission, I’m reproducing parts of it here, both so I can find it next time I come to write assessment questions and in case it’s useful to any of my readers (some of it is specific to the pedagogy of K101, e.g. the centrality of case studies, but other points are more widely applicable). I’ve added some notes of my own at the bottom.

TMA Essays

  • Does it lead students into exploring and thinking through a key aspect of a Unit of the Block?
  • Does it give the student enough scope to draw on case material?
  • Does it give students enough opportunity to engage with key ideas and debates?
  • Is the scope of the question focused enough to help students to understand how to limit their answer?
  • Does the task/guidance fit with the writing skills development strategy of the course?
  • Does the guidance guide the writing just enough to still leave a significant challenge for level 1 students?
  • Is there scope for high-end students to extend themselves?

Exam questions

  • Does it give students enough opportunity to engage with key ideas and debates?
  • Does it give the student enough scope to draw on case material from the course?
  • Is there enough contextual information in the question to enable students to think their way quickly into a productive line of answering?
  • Is there enough scope in the question to enable a Level 1 student to write a substantial answer? (E.g. the addition of such wording as ‘and how may these issues be addressed’, or ‘and what are the arguments against’ will often legitimise and guide students towards providing a fuller, more rounded answer.)
  • Is there enough relevant material in the related unit(s) to enable a Level 1 student to write a substantial answer? (E.g. A short subsection of a unit is unlikely to offer enough – and there needs to be clearly associated case material.)
  • Is there scope for high-end students to extend themselves?

My additions

If you’re not sure whether you’ve got the question right, try writing the student and tutor guidance for it. If it’s not going to work, that quickly becomes apparent when you try to tell the students how to approach it and the ALs how to mark it. In K101 the tutors are given a list of key concepts which come from the course material. For most of my questions I had about a dozen concepts, of greater and lesser relevance. I discovered that for one question I had set I could only come up with 3 key concepts. That made me realise that the question wasn’t really one that could be answered from the course materials, even though it was otherwise a good question.

Slightly journalistic questions can work well – ‘Is this the case?’, ‘Does this work?’ – but they usually need an additional sentence like ‘Discuss in relation to the course materials’ or ‘Illustrate your answer with examples from…’. This helps to ensure that students don’t get seduced into ranting on the basis of their own opinions or poorly theorised personal experiences.

Try not to get distracted onto your own pet peeves or interests.

Questions of the form ‘General issue question. Illustrate your answer with reference to 2 case studies from the course material’ often work well. The general questions helps to keep a focus on the theoretical issues and the case study bit helps to keep it focused and grounded on practice and the complexities of real situations. 

And finally, remember to allow plenty of time to draft the questions! It always takes longer than you think.

Rowntree, D. (1977) Assessing students: How shall we know them? London: Kogan. 
Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: