Every year, we deliver more than 160 online assessments to a total of over 17K participants. Over 50 of these are formal University exams
...great expertise in designing and delivering online assessment, enhancing the quality and reliability of assessment, and saving considerable amounts of academic time in marking in the process.
- University of Oxford Digital Education Strategy
As far as we’re aware, we are the only Division routinely running online assessment. Assessments include: self-test quizzes; tests whose completion is a requirement for entry into practical classes; formative assessments sat under exam conditions; and open book assessments available to students over a period of a few days. However, over 50 of these are formal University exams, run with the help of Examination Schools’ invigilators.
Why Online Assessment?
- Marking is instantaneous
- Easy availability of extensive question performance data:
- Examiners can correct results for problem questions immediately
- Problem questions can be identified and omitted from, or rewritten for, future assessments
- Self-test and revision quizzes, with immediate feedback, are available 24/7 from anywhere with an internet connection
- Computer-marking is fairer, removing inter-/intra-marker variability
- Question banks can be used for both summative and formative assessment
- High quality images can be included at lower cost than on paper. Audio, video, and even simulations and complex visualisations are also available to question authors.
Security & Reliability
Our agreement with the Proctors, allowing online University examinations, specifies the availability of a number of contingency measures, including, the provision of paper copies of an exam in the event of a power failure or similar worst case scenarios.
We run QuestionMark Perception, which is widely used in UK HE, on high-availability Oxford servers and, for reasons of security, access is only possible from machines on the Oxford network or those connected to it virtually using the University’s VPN service.
Perception offers a huge variety of question types. These allow the incorporation of images, equations and even audio and video. However, in practice, the majority of questions used in examinations, and assessments preparing students for those examinations, are variations of standard five-part single best answer/multiple choice questions.
I have just been playing with it properly for the first time and ITS GREAT! - Professor in Biochemistry, 2013
If the question types offered by Perception don’t meet your needs, we will be able to help you with something that does. In the past we have successfully explored the marking of free-test answers (using Intelligent Assessment software). We also run a pre-arrival maths test for Biochemistry and Biomedical Sciences students which uses QTIWorks. This allows us to write questions to which students give algebraic answers and which can be made to understand: answers in different units; significant figures and decimal places. Numerical questions can also be made which will randomly generate values according to rules set by the author. These allow a problem of the same type to be attempted multiple times or the problem to be given to different students but with different values and hence answers.
I wanted to let you know what a difference [the Question Banks have] made to the examiners' job of setting questions this year.- Course Administrator, 2012
QuestionBanks are databases of all the available online assessment questions for a subject. Questions can be filtered by syllabus coverage, question type and the assessment(s) in which they appeared. Detailed question analysis data are also available, enabling examiners to consider past performance when choosing exam questions. The banks are secure and only accessible to examiners.
- Question Bank User Guide (Oxford SSO required)
- Creating questions for a Question Bank (Oxford SSO required)
MSDLT will work with you to ensure that your assessment is delivered successfully but a typical process is shown below:
Exams - What you can expect from us and what we expect from you
|WE WILL||YOU WILL|
AT LEAST EIGHT WEEKS BEFORE
Provide advice on creating questions and exams
Provide training on using MSDLT tools to create questions and exams yourself
ON REQUEST Give examiners access to a bank of all the questions that are already in the exam system, filterable by usage in previous exams and optionally by tags, e.g. syllabus references. The bank will include analysis data showing question performance in previous exams, where available.
Book the necessary rooms for the exam:
Book Exam Schools invigilators (if required) or arrange for your own invigilators to be present at the exam. You will need to ensure that each room has at least two invigilators (one or more of whom may be provided by Exam Schools). This is to cover situations (toilet breaks, fetching technical help, etc.) in which one invigilator may need to leave the room.
Check with us
Send us the following details for the exam:
|AT LEAST FOUR WEEKS BEFORE||
Confirm details with you
Send you an appointment email asking you to confirm that we have your exam details correct
Prepare for paper upload
Create an folder in a WebLearn secure site for your exam and provide instructions to enable you to access the secure site
Check we have correct details
Check that we have your exam details correct (both time and date and the information in the SharePoint site) and confirm by accepting the appointment or let us know what is wrong.
Upload your paper
EITHER upload your paper to WebLearn secure site – Word, PowerPoint or another agreed format. When using questions from the Question Bank, please include the question ID, e.g. x_xyz_q101, and clearly indicate any changes that have been made to the question.
OR enter your questions into the exam system, as agreed
|AT LEAST TWO WEEKS BEFORE||
Create your online exam paper
Add questions and agreed settings
Make any adjustments you need until you are happy with the exam
Getting paper to Exam Schools
If requested to by you, upload the paper version to the secure WebLearn site, ensure that Exam Schools staff have access, and email them to let them know it is there.
Check your paper online and (if necessary, on paper)
Communicate with Exam Schools
Getting paper to Exam Schools
EITHER deposit 2-3 copies of the paper version of your exam (including images and answer sheet) with Exam Schools to be used in the event of a student given permission (either in advance or because of a last minute change in circumstances such as an accident) to sit the exam on paper in college. Attach a cover sheet stating the SITS assessment number and title.
OR Remind us to upload the exam to the secure site so it can be accessed by Exam Schools staff and tell us who the contact person is.
|AT LEAST ONE WEEK BEFORE||
Create paper versions
Create paper formats for examiners and students with permission to sit on paper (with answer sheets and image files).
Update the generic 'Invigilator Announcement Sheet' (on your SharePoint site), ready to bring on the day
List of candidates
Send us a list of candidates, including Oxford Single Sign-on username (e.g. abcd1234) and advise us if any have special requirements, indicating which sitting they will be taking (if applicable) and where (same room, alternative venue)
Communicating with candidates
Remind candidates that they:
|ON THE DAY||
At least one member of MSDLT will arrive at least 15 minutes before the start of the exam to set things up, be present for the beginning of the exam, and be on call throughout, in the building where possible.
Bring with us
Unless otherwise agreed, we will bring with us:
Attend on the day (or send a representative) and remain for the whole of the exam to help setup, communicate with examiners, and clear up after the exam.
Ideally, an examiner should be present for at least the first 30 minutes of the exam (at each sitting, if multiple sittings) and should be contactable throughout.
Bring with you
Unless otherwise agreed, you will bring with you on the day:
|AFTER THE EXAM||
Provide an Excel report of the total score, score on each question and answers given by each student. Unless otherwise agreed, we will upload these to our secure WebLearn within 24hrs.
Provide analysis reports - basic statistics on question performance, and breakdown of how many candidates chose each answer for each question. Unless otherwise agreed, we will upload these to our secure WebLearn within 24hrs.
ON REQUEST Create a subject Question Bank for you to facilitate examiners putting together future exams
Get your results
Unless otherwise agreed, log into our secure WebLearn site to download your results and analysis.
This table is also available as a downloadable Word document (Oxford SSO required).
For those already involved in online assessment at Oxford, you can find our guides to the various aspects here:
- What we and you can expect (Oxford SSO required)
- Creating standard assessment questions (Oxford SSO required)
- Creating an assessment (Oxford SSO required)
- Creating additional question types for assessments (Oxford SSO required)
- Adjusting questions within the Authoring Manager (Oxford SSO required)
- Creating paper copies of your assessment (Oxford SSO required)
- Revising Questions (Oxford SSO required)
- Quick checklist for creating questions (Oxford SSO required)
- Adding special characters in Perception (Oxford SSO required)
- Checking exam is set up correctly – an introductory video in less than a minute.