Online Assessment

Every year, we deliver more than 160 online assessments to a total of over 17K participants. Over 50 of these are formal University exams

...great expertise in designing and delivering online assessment, enhancing the quality and reliability of assessment, and saving considerable amounts of academic time in marking in the process.
- University of Oxford Digital Education Strategy

As far as we’re aware, we are the only Division routinely running online assessment. Assessments include: self-test quizzes; tests whose completion is a requirement for entry into practical classes; formative assessments sat under exam conditions; and open book assessments available to students over a period of a few days. However, over 50 of these are formal University exams, run with the help of Examination Schools’ invigilators.

Why Online Assessment?

Students taking online exams

Marking is instantaneous
Easy availability of extensive question performance data:
- Examiners can correct results for problem questions immediately
- Problem questions can be identified and omitted from, or rewritten for, future assessments
Self-test and revision quizzes, with immediate feedback, are available 24/7 from anywhere with an internet connection
Computer-marking is fairer, removing inter-/intra-marker variability
Question banks can be used for both summative and formative assessment
High quality images can be included at lower cost than on paper. Audio, video, and even simulations and complex visualisations are also available to question authors.

Security & Reliability

Our agreement with the Proctors, allowing online University examinations, specifies the availability of a number of contingency measures, including, the provision of paper copies of an exam in the event of a power failure or similar worst case scenarios.

We run QuestionMark Perception, which is widely used in UK HE, on high-availability Oxford servers and, for reasons of security, access is only possible from machines on the Oxford network or those connected to it virtually using the University’s VPN service.

Question Types

Perception offers a huge variety of question types. These allow the incorporation of images, equations and even audio and video. However, in practice, the majority of questions used in examinations, and assessments preparing students for those examinations, are variations of standard five-part single best answer/multiple choice questions.

I have just been playing with it properly for the first time and ITS GREAT! - Professor in Biochemistry, 2013

If the question types offered by Perception don’t meet your needs, we will be able to help you with something that does. In the past we have successfully explored the marking of free-test answers (using Intelligent Assessment software). We also run a pre-arrival maths test for Biochemistry and Biomedical Sciences students which uses QTIWorks. This allows us to write questions to which students give algebraic answers and which can be made to understand: answers in different units; significant figures and decimal places. Numerical questions can also be made which will randomly generate values according to rules set by the author. These allow a problem of the same type to be attempted multiple times or the problem to be given to different students but with different values and hence answers.

Question Banks

I wanted to let you know what a difference [the Question Banks have] made to the examiners' job of setting questions this year.- Course Administrator, 2012

QuestionBanks are databases of all the available online assessment questions for a subject. Questions can be filtered by syllabus coverage, question type and the assessment(s) in which they appeared. Detailed question analysis data are also available, enabling examiners to consider past performance when choosing exam questions. The banks are secure and only accessible to examiners.

Question Bank User Guide (Oxford SSO required)
Creating questions for a Question Bank (Oxford SSO required)

The Process

MSDLT will work with you to ensure that your assessment is delivered successfully but a typical process is shown below:

Online assessment - process

Exams - What you can expect from us and what we expect from you

	WE WILL	YOU WILL
AT LEAST EIGHT WEEKS BEFORE	Advice Provide advice on creating questions and exams Training Provide training on using MSDLT tools to create questions and exams yourself Question Bank ON REQUEST Give examiners access to a bank of all the questions that are already in the exam system, filterable by usage in previous exams and optionally by tags, e.g. syllabus references. The bank will include analysis data showing question performance in previous exams, where available.	Room booking Book the necessary rooms for the exam: IT suite itself (if using any room other than the CAL Lab or Classroom 1 in the MSTC, you will need to agree this with us first) Room(s) for students' bags Room(s) for holding candidates between sittings Room(s) and request computer(s) (with external keyboards, mice and monitors if these are laptops) for students with permission to sit in a separate room. Broadbent only - check that Experimental Psychology IT Support (currently Daniel Walters) is aware that you are running an online exam and will ensure that the computers are logged in at least 1/2 hour before the exam is due to begin Invigilator booking Book Exam Schools invigilators (if required) or arrange for your own invigilators to be present at the exam. You will need to ensure that each room has at least two invigilators (one or more of whom may be provided by Exam Schools). This is to cover situations (toilet breaks, fetching technical help, etc.) in which one invigilator may need to leave the room. Check with us Send us the following details for the exam: Date and time Location Examiner(s), i.e. who will check the paper Administrator(s), i.e. who we should contact about arrangements We will then: Check that we can cover the exam Add it to our exam diary
AT LEAST FOUR WEEKS BEFORE	Confirm details with you Send you an appointment email asking you to confirm that we have your exam details correct Prepare for paper upload Create an folder in a WebLearn secure site for your exam and provide instructions to enable you to access the secure site	Check we have correct details Check that we have your exam details correct (both time and date and the information in the SharePoint site) and confirm by accepting the appointment or let us know what is wrong. Upload your paper EITHER upload your paper to WebLearn secure site – Word, PowerPoint or another agreed format. When using questions from the Question Bank, please include the question ID, e.g. x_xyz_q101, and clearly indicate any changes that have been made to the question. OR enter your questions into the exam system, as agreed
AT LEAST TWO WEEKS BEFORE	Create your online exam paper Add questions and agreed settings Iterate Make any adjustments you need until you are happy with the exam Getting paper to Exam Schools If requested to by you, upload the paper version to the secure WebLearn site, ensure that Exam Schools staff have access, and email them to let them know it is there.	Check paper Check your paper online and (if necessary, on paper) Communicate with Exam Schools provide Exam Schools with list of candidates indicating which room/sitting they are in (include details of candidates with extra time and/or alternative arrangements) ask Exam Schools for invigilators' Bod Card numbers and pass them to the appropriate person (Colin Cook at MSTC) to allow swipe card access to the IT suites(s). Getting paper to Exam Schools EITHER deposit 2-3 copies of the paper version of your exam (including images and answer sheet) with Exam Schools to be used in the event of a student given permission (either in advance or because of a last minute change in circumstances such as an accident) to sit the exam on paper in college. Attach a cover sheet stating the SITS assessment number and title. OR Remind us to upload the exam to the secure site so it can be accessed by Exam Schools staff and tell us who the contact person is.
AT LEAST ONE WEEK BEFORE	Create paper versions Create paper formats for examiners and students with permission to sit on paper (with answer sheets and image files).	Invigilator instructions Update the generic 'Invigilator Announcement Sheet' (on your SharePoint site), ready to bring on the day List of candidates Send us a list of candidates, including Oxford Single Sign-on username (e.g. abcd1234) and advise us if any have special requirements, indicating which sitting they will be taking (if applicable) and where (same room, alternative venue) Communicating with candidates Remind candidates that they: (if necessary) need to wear subfusc should only bring water in a spill-proof bottle should know their single sign on username and password should bring writing implements and calculator, as appropriate
ON THE DAY	Technical support At least one member of MSDLT will arrive at least 15 minutes before the start of the exam to set things up, be present for the beginning of the exam, and be on call throughout, in the building where possible. Bring with us Unless otherwise agreed, we will bring with us: Examiner's version (showing 100% achieved and correct answers) Paper version (in case of computer failure, special circumstances, etc) An encrypted USB memory stick containing electronic versions of the above A list of alternative login credentials for use should any student have forgotten their SSO credentials	Administrative support Attend on the day (or send a representative) and remain for the whole of the exam to help setup, communicate with examiners, and clear up after the exam. Examiner attendance Ideally, an examiner should be present for at least the first 30 minutes of the exam (at each sitting, if multiple sittings) and should be contactable throughout. Bring with you Unless otherwise agreed, you will bring with you on the day: invigilator announcements (downloaded from SharePoint site for your exam, amended and printed, as necessary) log book (downloaded from SharePoint site for your exam, amended and printed, as necessary) list of candidates, with details of alternative arrangements, extra time etc scrap paper and pencils (if required) calculators (if you are providing these) desk labels (either names, or numbers with sheet showing who is sitting at which number)
AFTER THE EXAM	Results Provide an Excel report of the total score, score on each question and answers given by each student. Unless otherwise agreed, we will upload these to our secure WebLearn within 24hrs. Analysis Provide analysis reports - basic statistics on question performance, and breakdown of how many candidates chose each answer for each question. Unless otherwise agreed, we will upload these to our secure WebLearn within 24hrs. Question Bank ON REQUEST Create a subject Question Bank for you to facilitate examiners putting together future exams	Get your results Unless otherwise agreed, log into our secure WebLearn site to download your results and analysis.