Cookies on this website

We use cookies to ensure that we give you the best experience on our website. If you click 'Accept all cookies' we'll assume that you are happy to receive all cookies and you won't see this message again. If you click 'Reject all non-essential cookies' only necessary cookies providing core functionality such as security, network management, and accessibility will be enabled. Click 'Find out more' for information on how to change your cookie settings.

Every year, we deliver more than 160 online assessments to a total of over 17K participants. Over 50 of these are formal University exams

...great expertise in designing and delivering online assessment, enhancing the quality and reliability of assessment, and saving considerable amounts of academic time in marking in the process.
- University of Oxford Digital Education Strategy

As far as we’re aware, we are the only Division routinely running online assessment. Assessments include: self-test quizzes; tests whose completion is a requirement for entry into practical classes; formative assessments sat under exam conditions; and open book assessments available to students over a period of a few days. However, over 50 of these are formal University exams, run with the help of Examination Schools’ invigilators.




Why Online Assessment?

Students taking online exams

  • Marking is instantaneous
  • Easy availability of extensive question performance data:
    • Examiners can correct results for problem questions immediately
    • Problem questions can be identified and omitted from, or rewritten for, future assessments
  • Self-test and revision quizzes, with immediate feedback, are available 24/7 from anywhere with an internet connection
  • Computer-marking is fairer, removing inter-/intra-marker variability
  • Question banks can be used for both summative and formative assessment
  • High quality images can be included at lower cost than on paper. Audio, video, and even simulations and complex visualisations are also available to question authors.

Security & Reliability

Our agreement with the Proctors, allowing online University examinations, specifies the availability of a number of contingency measures, including, the provision of paper copies of an exam in the event of a power failure or similar worst case scenarios.

We run QuestionMark Perception, which is widely used in UK HE, on high-availability Oxford servers and, for reasons of security, access is only possible from machines on the Oxford network or those connected to it virtually using the University’s VPN service.

Question Types

Perception offers a huge variety of question types. These allow the incorporation of images, equations and even audio and video. However, in practice, the majority of questions used in examinations, and assessments preparing students for those examinations, are variations of standard five-part single best answer/multiple choice questions.

I have just been playing with it properly for the first time and ITS GREAT! - Professor in Biochemistry, 2013

If the question types offered by Perception don’t meet your needs, we will be able to help you with something that does. In the past we have successfully explored the marking of free-test answers (using Intelligent Assessment software). We also run a pre-arrival maths test for Biochemistry and Biomedical Sciences students which uses QTIWorks. This allows us to write questions to which students give algebraic answers and which can be made to understand: answers in different units; significant figures and decimal places. Numerical questions can also be made which will randomly generate values according to rules set by the author. These allow a problem of the same type to be attempted multiple times or the problem to be given to different students but with different values and hence answers.

Question Banks

I wanted to let you know what a difference [the Question Banks have] made to the examiners' job of setting questions this year.- Course Administrator, 2012



QuestionBanks are databases of all the available online assessment questions for a subject. Questions can be filtered by syllabus coverage, question type and the assessment(s) in which they appeared. Detailed question analysis data are also available, enabling examiners to consider past performance when choosing exam questions. The banks are secure and only accessible to examiners.

 

 

 

 

The Process

MSDLT will work with you to ensure that your assessment is delivered successfully but a typical process is shown below:

Online assessment - process

 Exams - What you can expect from us and what we expect from you

  WE WILL YOU WILL

AT LEAST EIGHT WEEKS BEFORE

Advice
Provide advice on creating questions and exams

Training
Provide training on using MSDLT tools to create questions and exams yourself

Question Bank
ON REQUEST Give examiners access to a bank of all the questions that are already in the exam system, filterable by usage in previous exams and optionally by tags, e.g. syllabus references. The bank will include analysis data showing question performance in previous exams, where available.

Room booking
Book the necessary rooms for the exam:

  • IT suite itself (if using any room other than the CAL Lab or Classroom 1 in the MSTC, you will need to agree this with us first)
  • Room(s) for students' bags
  • Room(s) for holding candidates between sittings
  • Room(s) and request computer(s) (with external keyboards, mice and monitors if these are laptops) for students with permission to sit in a separate room.
  • Broadbent only - check that Experimental Psychology IT Support (currently Daniel Walters) is aware that you are running an online exam and will ensure that the computers are logged in at least 1/2 hour before the exam is due to begin

Invigilator booking
Book Exam Schools invigilators (if required) or arrange for your own invigilators to be present at the exam. You will need to ensure that each room has at least two invigilators (one or more of whom may be provided by Exam Schools). This is to cover situations (toilet breaks, fetching technical help, etc.) in which one invigilator may need to leave the room.

Check with us
Send us the following details for the exam:

  • Date and time
  • Location
  • Examiner(s), i.e. who will check the paper
  • Administrator(s), i.e. who we should contact about arrangements

We will then:

  • Check that we can cover the exam
  • Add it to our exam diary

AT LEAST FOUR WEEKS BEFORE

Confirm details with you
Send you an appointment email asking you to confirm that we have your exam details correct

Prepare for paper upload
Create an folder in a WebLearn secure site for your exam and provide instructions to enable you to access the secure site

Check we have correct details
Check that we have your exam details correct (both time and date and the information in the SharePoint site) and confirm by accepting the appointment or let us know what is wrong.

Upload your paper
EITHER upload your paper to WebLearn secure site – Word, PowerPoint or another agreed format. When using questions from the Question Bank, please include the question ID, e.g. x_xyz_q101, and clearly indicate any changes that have been made to the question.

OR enter your questions into the exam system, as agreed

AT LEAST TWO WEEKS BEFORE

Create your online exam paper
Add questions and agreed settings

Iterate
Make any adjustments you need until you are happy with the exam

Getting paper to Exam Schools
If requested to by you, upload the paper version to the secure WebLearn site, ensure that Exam Schools staff have access, and email them to let them know it is there.

Check paper
Check your paper online and (if necessary, on paper)

Communicate with Exam Schools

  • provide Exam Schools with list of candidates indicating which room/sitting they are in (include details of candidates with extra time and/or alternative arrangements)
  • ask Exam Schools for invigilators' Bod Card numbers and pass them to the appropriate person (Colin Cook at MSTC) to allow swipe card access to the IT suites(s).

Getting paper to Exam Schools
EITHER deposit 2-3 copies of the paper version of your exam (including images and answer sheet) with Exam Schools to be used in the event of a student given permission (either in advance or because of a last minute change in circumstances such as an accident) to sit the exam on paper in college. Attach a cover sheet stating the SITS assessment number and title.

OR Remind us to upload the exam to the secure site so it can be accessed by Exam Schools staff and tell us who the contact person is.

AT LEAST ONE WEEK BEFORE

Create paper versions
Create paper formats for examiners and students with permission to sit on paper (with answer sheets and image files).

Invigilator instructions
Update the generic 'Invigilator Announcement Sheet' (on your SharePoint site), ready to bring on the day

List of candidates
Send us a list of candidates, including Oxford Single Sign-on username (e.g. abcd1234) and advise us if any have special requirements, indicating which sitting they will be taking (if applicable) and where (same room, alternative venue)

Communicating with candidates
Remind candidates that they:

  • (if necessary) need to wear subfusc
  • should only bring water in a spill-proof bottle
  • should know their single sign on username and password
  • should bring writing implements and calculator, as appropriate

ON THE DAY

Technical support
At least one member of MSDLT will arrive at least 15 minutes before the start of the exam to set things up, be present for the beginning of the exam, and be on call throughout, in the building where possible.

Bring with us
Unless otherwise agreed, we will bring with us:

  • Examiner's version (showing 100% achieved and correct answers)
  • Paper version (in case of computer failure, special circumstances, etc)
  • An encrypted USB memory stick containing electronic versions of the above
  • A list of alternative login credentials for use should any student have forgotten their SSO credentials

Administrative support
Attend on the day (or send a representative) and remain for the whole of the exam to help setup, communicate with examiners, and clear up after the exam.

Examiner attendance
Ideally, an examiner should be present for at least the first 30 minutes of the exam (at each sitting, if multiple sittings) and should be contactable throughout.

Bring with you
Unless otherwise agreed, you will bring with you on the day:

  • invigilator announcements (downloaded from SharePoint site for your exam, amended and printed, as necessary)
  • log book (downloaded from SharePoint site for your exam, amended and printed, as necessary)
  • list of candidates, with details of alternative arrangements, extra time etc
  • scrap paper and pencils (if required)
  • calculators (if you are providing these)
  • desk labels (either names, or numbers with sheet showing who is sitting at which number)

AFTER THE EXAM

Results
Provide an Excel report of the total score, score on each question and answers given by each student. Unless otherwise agreed, we will upload these to our secure WebLearn within 24hrs.

Analysis
Provide analysis reports - basic statistics on question performance, and breakdown of how many candidates chose each answer for each question. Unless otherwise agreed, we will upload these to our secure WebLearn within 24hrs.

Question Bank
ON REQUEST Create a subject Question Bank for you to facilitate examiners putting together future exams

Get your results
Unless otherwise agreed, log into our secure WebLearn site to download your results and analysis.

This table is also available as a downloadable Word document (Oxford SSO required).

 

MSDLT Guides

For those already involved in online assessment at Oxford, you can find our guides to the various aspects here:

FAQs

Online Assessment: What is the best question type? How many options should students choose from?

Online Assessment: Why can’t I access outside quizzes from my home computer? Why am I getting a ‘connection timed out’ message?

Online Assessment: Standard setting - helping to create a defensible assessment

Online Assessment: What results and analysis can I expect from my online assessment?

Online Assessment: What do difficulty, correlation, discrimination, etc. in the question analysis mean?