Remembering My Hat

28th March 2013

More principles of assessment

I’m thinking a lot about assessment at the moment, especially in relation to K118 (Title TBC) the new module I’m chairing. So this seminar last week was extremely timely and useful. The notes that follow are, as ever, not full sentences and not a full representation of what was said, just the things I was particularly interested in.

(cc) rileyroxx

Releasing creativity in assessment – New models of assessment and tuition project

First of all we were asked to write down two challenges in designing assessment that were particularly occupying our minds at the moment. Mine were:

  • The desire to be creative v. the danger of doing new stuff which is more likely to fail catastrophically and is generally much harder for staff (especically at the OU where everything is done at scale)
  • Getting the minutiae of weightings, substitutions, thresholds, formative/summative, exam v. EMA etc. right, and being aware of the knock-on unintended consequences on everything else of tweaking one variable.

How transparent is your guidance? All visible to students or separate tutor guidance notes to make marking easier for ALs (I’ve worked on modules with both. I’m not sure which is better).

Common problems:

  • Hard for students to see how the assessment links to the Learning Outcomes
  • Narrow range of assessment tasks and repetition across qualifications – students feel they are being asked to do the same thing again and again.
  • Gap between the students and the Module Team / AL understanding of task
  • Not much use of formative assessment
  • Students don’t pay attention to the feedback – they only care about the grade.
  • Limited use of student self-assessment
  • Delays in feedback mean they sometimes get it too late to be useful for the next TMA
  • No feedback on EMA sometimes
  • Difficult to find critical friend reviewers of assessment and difficult to find good practice examples
  • Tensions between the costs and rewards of innovation
  • Progression in assessment within a qualification

Need to reward the student for taking notice of feedback on previous TMA [could do this either with reflective Part B like K101, or with formative then summative on same essay, and probably other ways too]

Assessment should be [fairly uncontentious, hard to achieve]

  • Central to teaching
  • Part of an ongoing conversation between student and AL
  • Support student’s progress
  • Authentic, engaging, exciting
  • Be linked to specific learning outcomes and other outcome criteria

[Interesting argument between participants in the seminar about whether it is all about testing the learning outcomes, or whether it’s about supporting student learning, and whether those are in fact the same thing]

Life crises argument against formative assessment [if you have too few summative assessments, students drop too many marks if they miss a TMA due to life-crisis kind of factors (bereavement, illnesses, divorce etc.)]

Could maybe do optionality of TMAs? Set 6 but they can chose 4? But would have to think AL workload implications though very carefully [Although is that any different from TMAs that give students two options for questions, in terms of AL impact? Possibly not]

Make all of OCAS formative? Then an EMA that tests the whole module. (Science has one S141)

When assessment is content driven and Blocks are quite different, students can find it hard to build on the feedback they are given on TMAs, especially if it’s an end of Block assessment. [This is a danger for K118 but I think it can be avoided with careful planning]

Lot of people keen on peer-assessment and self-evaluation. Start at level 1 but gently – contribute to a wiki for example. [But all the practical advice I get is against]

Advertisements

21st September 2009

Learning to assess: Assessing to learn

Filed under: Uncategorized — rememberingmyhat @ 11:25
Tags: , , , , , ,

(Image nicked from studenthacks.org/2008/01/29/conclude-essay/)

Last week the K101 course team had an all-day meeting to discuss the assessment questions for the next couple of presentations of the course. Each person who is responsible for a Block (about a month’s work, a sixth of the 60 point course) had produced suggested questions and guidance for students and tutors for their material. On this occasion we each produced 4 continuous assessment (TMA) essay questions and 1 exam question. We pretty much spent all day discussing the draft questions and their guidance in fine detail. It’s very much like the process of constant critcism and redrafting we used to produce the materials in the first place, although with considerably fewer iterations.

Exam and assessment questions are much harder to write than you might think – it’s very easy to set something misleading, or unsuited to students of varying ability, or insufficiently related to the course materials. And because this is basically the same material presented over many years, you quickly run into the danger of repeating yourself. It’s very important to avoid that when you know people sell past essays for your course on eBay!

I value very highly the co-operative and collegial atmosphere in which these sort of discussions (mostly!) happen. It’s not easy having your work torn to shreds, but people mostly seem to accept that it is done with good intentions and that the end result is immeasurably better for it. I was particularly impressed with the good grace with which a senior and very experienced colleague accepted having to go away and rewrite almost all of their questions. I like the way it seemed we were all focused on the end product and not on our egos or pet interests.

Assessment is of course vitally important in any educational context since, famously, assessment defines the de facto curriculum (Rowntree, 1977). But it’s particularly important in an OU context because feedback on essays is one of the main vehicles of individualised teaching. Students (usually!) get very substantial comment on their work and suggestions for how to improve it. And tutors get regular feedback on their marking and how to improve the ways they teach through marking. The feedback chain does come to a stop with the people who give the feedback to the tutors (Monitors), since we are not assessed, but we do get trained specifically in being a monitor.

One of the course co-Chairs, Anthea Wilson, had produced a paper to help us design the questions (with, I think, some input from Andy Northedge, who was the course Chair while we were producing the course). With her permission, I’m reproducing parts of it here, both so I can find it next time I come to write assessment questions and in case it’s useful to any of my readers (some of it is specific to the pedagogy of K101, e.g. the centrality of case studies, but other points are more widely applicable). I’ve added some notes of my own at the bottom.

TMA Essays

  • Does it lead students into exploring and thinking through a key aspect of a Unit of the Block?
  • Does it give the student enough scope to draw on case material?
  • Does it give students enough opportunity to engage with key ideas and debates?
  • Is the scope of the question focused enough to help students to understand how to limit their answer?
  • Does the task/guidance fit with the writing skills development strategy of the course?
  • Does the guidance guide the writing just enough to still leave a significant challenge for level 1 students?
  • Is there scope for high-end students to extend themselves?

Exam questions

  • Does it give students enough opportunity to engage with key ideas and debates?
  • Does it give the student enough scope to draw on case material from the course?
  • Is there enough contextual information in the question to enable students to think their way quickly into a productive line of answering?
  • Is there enough scope in the question to enable a Level 1 student to write a substantial answer? (E.g. the addition of such wording as ‘and how may these issues be addressed’, or ‘and what are the arguments against’ will often legitimise and guide students towards providing a fuller, more rounded answer.)
  • Is there enough relevant material in the related unit(s) to enable a Level 1 student to write a substantial answer? (E.g. A short subsection of a unit is unlikely to offer enough – and there needs to be clearly associated case material.)
  • Is there scope for high-end students to extend themselves?

My additions

If you’re not sure whether you’ve got the question right, try writing the student and tutor guidance for it. If it’s not going to work, that quickly becomes apparent when you try to tell the students how to approach it and the ALs how to mark it. In K101 the tutors are given a list of key concepts which come from the course material. For most of my questions I had about a dozen concepts, of greater and lesser relevance. I discovered that for one question I had set I could only come up with 3 key concepts. That made me realise that the question wasn’t really one that could be answered from the course materials, even though it was otherwise a good question.

Slightly journalistic questions can work well – ‘Is this the case?’, ‘Does this work?’ – but they usually need an additional sentence like ‘Discuss in relation to the course materials’ or ‘Illustrate your answer with examples from…’. This helps to ensure that students don’t get seduced into ranting on the basis of their own opinions or poorly theorised personal experiences.

Try not to get distracted onto your own pet peeves or interests.

Questions of the form ‘General issue question. Illustrate your answer with reference to 2 case studies from the course material’ often work well. The general questions helps to keep a focus on the theoretical issues and the case study bit helps to keep it focused and grounded on practice and the complexities of real situations. 

And finally, remember to allow plenty of time to draft the questions! It always takes longer than you think.

Rowntree, D. (1977) Assessing students: How shall we know them? London: Kogan. 

Create a free website or blog at WordPress.com.