Technology for teaching, learning and assessment: assistance and software development
We can offer:
- Help with using technology for teaching, learning and assessment for courses taught at Oxford & at a distance
- Multiple educational technologies, from delivering an online exam and help with WebLearn, to creating bespoke, interactive simulations, online experiments and websites
Recent projects that we’ve worked on include:
- CSlide - virtual microscopy system which allows users to dynamically view slides on the web, at up to 40x magnification. Find out how CSlide is being used for teaching and learning and to preserve the history of medicine
- iCases – a tool which allows academics to present engaging and realistic decision-making scenarios to their students, online
- Simulation modelling and online virtual experiments – websites that allow students to virtually experiment with parameters to aid their understanding of complex systems, or undertake virtual lab work that would be too time-consuming, costly or dangerous in the 'real' world.
Further details of some of our many projects and services below and/or get in touch with us – we’re always happy to help with new ideas to improve teaching and learning at Oxford.
Teaching, learning & assessment
Every year, we deliver more than 160 online assessments to a total of over 17K participants. Over 50 of these are formal University exams
Easy-to-use content management system which delivers text, images, video, questions and more
Playing with parameters in complex systems and simulating experiments to allow students to concentrate on the science
This is very much a moot point. The extended matching questions (EMQs) originally espoused by Case and Swanson, are gradually being replaced within the MSD, and more widely in the medical sciences, by standard multiple choice questions (MCQs). This is in response to various papers suggesting that, while increasing the number of answers from which students choose can increase the difficulty of questions, there is very little loss in discrimination with significantly fewer options e.g Swanson et al., 2006.
The effect of decreasing the number of choices per item while lengthening the test proportionately is to increase the efficiency of the test for high-level examinees and to decrease its efficiency for low-level examinees
- Lord, F. M. (1977). Optimal number of choices per item—A comparison of four approaches. Journal of Educational Measurement, 14, 33– 38.
In fact, in a meta-analysis of previous research, Rodriguez suggests that in many cases, choosing between three options is most efficient as it minimises time taken to write questions, and for students to answer them, allowing a better coverage of the subject matter. The argument is that, in many questions, only two of the incorrect answer options are really distractors anyway, any additional options are eliminated immediately, effectively leaving a three-option MCQ. The same argument suggests that, although a five option MCQ might appear to give only a 20% likelihood of getting the right answer by chance, as long as students have sufficient ability and aren’t under time pressure - so are not guessing completely randomly, if there are only three plausible answers, the likelihood is actually 33%.
This may be an over-simplification, with its assumptions about the ability of examiners to write plausible distractors and the abilities of students to spot and discount the implausible, but it does suggest that the number of answer options should be chosen to match the question/subject matter/ability of the author/ability of the students, rather than arbitrarily deciding on a fixed value.
In practice in the MSD, five option MCQs now form the great majority of summative assessment questions while EMQs, multiple response questions (tick boxes), and MCQs with a different number of options make up the remainder.
If you only have 5 minutes: Table 1 in Haladyna et al. 2010 provides some succinct guidelines
At Oxford, there is also a regular seminar run by the Medical School on question writing.
Two of the methods used to determine pass marks for online assessment within MSD are:
- The ‘Cohen method’ sets the pass mark relative to the performance of the best-performing students (typically the 95th centile) on the basis that these students will not vary significantly from year to year. This method requires a fairly large cohort of students for reliability. Cohen-Schotanus, J., & van der Vleuten, C. P. (2010). A standard setting method with the best performing students as point of reference: Practical and affordable. Medical teacher, 32(2), 154–160.
- Variations on the Angoff method (see e.g. QuestionMark handout) in which subject matter experts determine how likely a minimally competent or borderline candidate is to answer correctly. This can be applied question by question and then summed. Alternatively, variations on the Ebel method can be used to classify questions according to difficulty (e.g. easy, medium, hard) and relevance (e.g essential, important, acceptable and questionable). The Angoff likelihoods can then be applied to each class of question. This method requires a group of subject matter experts to go through an assessment question by question. We have a spreadsheet which can be used to collate Ebel-Angoff ratings.
We can normally produce results immediately after the last participant has submitted their answers, although if any remarking is required (to account for a problem question), this will take a little longer.
Analysis for MCQs, with one answer per question, can also be produced fairly instantly using the tools in Perception. Where the a question requires more than one answer from a student, we have to run the results through our own analysis software which can take significantly longer, particularly when the questions haven’t been created using our question making tool.
Online Assessment: What do difficulty, correlation, discrimination, etc. in the question analysis mean?
Difficulty (or p value) should probably be called easiness as a value of 1 indicates that all students got the question correct and a value of 0 indicated that no participant gave the correct answer. It is calculated by dividing the mean score on a question by the maximum possible. A good rule of thumb is that difficulty should be around the pass mark of your assessment.
Item Discrimination, where available, is calculated by taking the top and bottom 27% of students, based on overall score in the assessment, and subtracting the fraction of bottom group who gave the answer from the fraction of the top group who gave the answer, giving a range of -1 to 1. So:
- A positive item discrimination means a higher proportion of people in the top group chose the answer than in the bottom group. A high positive value for the correct answer generally means the question is a good discriminator, which is what we want (but is difficult to achieve!). A positive discrimination for an incorrect answer may indicate a problem, but could just mean that it is a good distractor.
- An item discrimination of 0 means the same number of people from each group gave the answer, so the answer doesn’t discriminate at all. Questions where everyone got the correct answer will always have a discrimination of 0.
- A negative item discrimination means a higher proportion of people in the bottom group chose the answer. This would be expected for an incorrect answer. A negative discrimination on a correct answer may indicate something is wrong, as the ‘good’ students are not choosing the correct answer.
Item-total correlation discrimination uses a Pearson product moment coefficient to give the correlation between the question score and the assessment score. Higher positive correlation values indicate that participants who obtain high question scores also obtain high assessment scores and that participants who obtain low question scores also obtain low assessment scores. This is what we want. Values below one indicate potentially worrying low scoring participants doing well on this question and/or high-scoring students doing badly. So, low values for questions here could indicate unhelpful questions that are worth looking at in more detail.
- Use the Preview Site button in the top right-hand corner to switch to see how an ‘access’ user would see your site. Click Exit preview to return to normal. However, this will not let you check things such as whether users in a particular group can see something.
- For a more reliable test of your site, particularly if it concerns something that is sensitive in some way, you are far better off adding yourself as an external user to your site and any necessary group(s) and then logging in as that user.
- In Site Info, select Add Participants
- In the Email Address of Non-Oxford Participant box, paste in a personal email address that you can access the click Continue.
- From the Roles list, choose access then Continue.
- Choose whether an email is sent to that account which contains a link to the site. If the account hasn’t already been created for this address before, an automatic email will be sent linking to a page where the user can choose a password. Click Continue.
- Check the details on the confirmation page then click Finish.
- Now check your email account for an email from email@example.com – check your ‘junk’ folder if it doesn’t appear almost immediately.
- In the email, click on the link which begins:
Accept this invitation https://weblearn.ox.ac.uk/accountvalidator...
and complete the form, choosing a password you can remember.
- If you have another browser on your machine (other than the one you have been using so far), open this up now and visit the page you are working on. If not, click the Logout button (or close your browser) and reopen the page.
- Now click the Other Users Login button and login using your personal email address and the password you have just set – you are now looking at the site as an access user.