Case study: creating authentic online simulations using branching scenarios

Overview

Kirsten’s team received a grant to create an 'Authentic Work-Based Assessment for Online Students’. The team wanted to focus on using case-based learning to embed a realistic, applied context for the unit structure and assessment using a sustainable model that could be modified for future iterations of the unit.

Rationale

In the past students analysed an industrial accident from history based on written materials e.g. the Esso Gas Plant Explosion at Longford. The plan was to improve this approach by creating an online mimetic simulation that unfolds over 9-weeks of the unit, with related assessments.

Here is a short 2 min interview exploring Kirsten’s plans before she embarked on the project….

ACU Case study: Creating authentic online simulations using branching scenarios I
Length: 2 mins.

Intended project outcomes

• Build higher quality student interaction through discussion forums
• Develop critical thinking skills
• Improve learning behaviours through inclusion of emotionally-charged content
• Prepare students for the realities of the workplace

Approach

The team developed a 9-week simulation around a workplace industrial accident. The simulation was built in LEO lesson, to allow a ‘choose your own adventure’ style structure, in which students are posed realistic questions and then participate in activities about next steps in unpacking the incident and responding to unexpected events as they occur throughout the period.

Development

The team split development into three parts:

1. Develop the 'big picture' storyboard for the LEO lesson, including where to place key decision points and where to link to resources and activities external to the lesson
2. Develop the resources (e.g. authentic materials relevant to investigating the incident, video and audio stimuli, online lecture-based resources) and activities (e.g. forums for discussing issues related to decision-making) that students could link to from within the lesson
3. Build the simulation in LEO

Structure of the simulation

There are three stages to the simulation, each one lasting 3 weeks. Each week a different step in the stage is played out, and students choose how they will respond. There is an authentic assessment associated with each stage.

Stage	1 - Out of the blue	2 - Mind the gap	3 - What's really driving behaviour
Themes	The incident occurs, and students are in the role of first responders and have to undertake a range of actions upon finding out about the incident.	Students do an analysis, and have some interpersonal issues that they have to navigate within the organisation, with the CEO, HR and the grieving family members.	The final stage is around the leadership of the board, and safety culture. Students learn about the different safety leadership styles of board members and attempt to influence their safety behaviours.
Authentic assessment	Business report about the interim findings	Report to the board about the failures of the safety management system that led to the incident	Video pitch to the senior managers and board to implement Safety II measures to improve the safety culture

Planning the scenario

Videographers and actors

Actual project outcomes

While data collection is still ongoing, initial findings indicate:

• Higher quality student interaction through discussion forums
  • There was more than double the number of posts on the discussion forum
  • The quality of posts increased and students shared personal perspectives in how they’ve dealt with these kind of problems in workplaces.
• Most of the lecture videos have been viewed twice as often as in the non-simulation version of the unit. Indicating a more critical engagement with the content and topics covered.

Lessons learned

This is what was learnt from the experience.

• Expect to need to iron out LEO issues in the first iteration. For example, in the first iteration we discovered there is a limit of a maximum of 10 attempts students can make at questions in a LEO Lesson before they automatically get progressed to the next page and can no longer return to the question page. As students tended to go down different paths, they would reach the maximum. The solution was to not use the question pages in this way.
• Getting the complexity and challenge right to cater for students with experience and with no experience
• Even with rigorous planning, the first time the unit runs, with many students going through it and talking about it, errors will surface. Have a strategy in place to deal with these issues as they arise.
• Kirsten plans on using templates for assignments to communicate clear expectations on assessment requirements as some students created assessment responses that were too extensive
• In the first iteration, the team discovered students were perturbed by the intentional inconsistencies in the scenario. The team had embedded gaps and inconsistencies purposely, as they were trying to create a realistic scenario because in real life when you’re doing workplace investigation, people tell you different things and there is information you don’t yet know.To meet the students need for reassurance whilst keeping pedagogic intent, the team added to the second iteration:
  • Introduction modules to give students an overview of the pedagogical approach (e.g. transparent pedagogy)
  • Within the introduction "design warnings" about purposeful inconsistencies and the reason they exist

ACU Case study: Creating authentic online simulations using branching scenarios IV
Length: 2 minutes.

Evaluation

Student feedback:

Positive

...the practical simulation was one of the best features of the unit

...what was more important than that was the whole process filled me with a confidence. Do you know what I mean? I now have a greater degree of confidence if I’m to approach a real life situation that was complex …...

There is a lot more confidence in my approach mentally, I mean I haven’t physically approached a similar problem to the simulation but I do feel confident if that came up I could handle it quite adequately

Negative 

• Some students perceived the workload associated with the simulation as high, as it involved sustained involvement across the semester
• Some students were perturbed by the ‘authenticity’ and unpredictability of case-based simulation

Academic colleague feedback:

I was very, very impressed with the scope of tools that were able to be deployed in order to simulate a real-life experience for an OHS practitioner. In fact, I after about an hour and a half to two hours engaging with the material I found myself almost stressed [laughter] because it took me back to a work role and all these competing things coming at you and trying to prioritize them and dealing with so many different people. Maybe because I was dealing with it in a compressed timeframe and trying to simulate as much as I could within two hours. But I did. I didn't have to go and get a valium or anything but I had to walk away and get a coffee [laughter]. That's a [tick]. I think it's a tick in terms of simulating the real-life activity.

.... the time sequencing is real-life and time spacing is real-time if I can put it that way. So I think it does create a sense of urgency, which you can relate to a work experience rather than just an academic experience. So if I don't do this, then the boss will be onto me, or if I don't do this, well, then I can't do that. So I think there's a lot of time management sequencing and prioritizing attributes in built, which may be incidental, but they're coming to the fore the whole way through.

Awards:

Kirsten won the ACU 2018 'Citations for Outstanding Contributions to Student Learning (Early Career)' for this project. For curriculum redesign and innovative online simulation using technology enhanced learning in Post-Graduate Occupational Health and Safety and Environmental Management Degrees.

Ideas for the future

Collecting further data to further examine:

• improvements in student engagement and learning outcomes
• measureable ‘authenticity’ (Ashford-Rowe et al., 2013) of the simulation and assessment
• mechanisms for efficacy, focusing on students’ ability to recognise emotion in the online simulation, as well as their ability to recognise their own emotional response.

Acknowledgements

Kirsten was also supported in the design and development by:

• FLB Educational Designer – Lisa Burrell.
• FLB eLearning Technical Support Officer – Meghan Brennan
• Academic collaboration – Kevin Ashford Rowe
• Ashford-Rowe, K., Herrington, J. & Brown, C. 2013. Establishing the critical elements that determine authentic assessment. Assessment & Evaluation in Higher Education, 39, 205-222.