Voice-controlled Interactive Audio Narratives for Training

VIANT lets you learn new skills through interactive audio narratives, using your voice to make decisions in fictional training scenarios. You can try it out now, using the speech recognition technology built into your web browser.

Instructions

1. Browser: You must use the desktop version of Google Chrome as your browser for these experiments. Only Google Chrome currently supports open-vocabulary web-based speech recognition. If you are currently using Safari, Firefox, Opera, or something on your mobile phone, then return to this site after you have launched the desktop version of Google Chrome.
2. Speakers/Headphones: Make sure the volume is turned up on your computer and your speakers are on. Even better, wear good headphones.
3. Enable your browser's microphone: After you read and understand these instructions, click on one of the links to our experiments below. When the page is loaded properly, you will see a white page with the title of the story. You may be prompted to allow your browser to use your microphone. Make sure your microphone is enabled. This is how you will interact with the story and say what you want to happen next.
4. Listen: Listen to the audio story, which contains a fictional situation where you are the hero of the story. You will continue to tell the story, from your perspective, and speak your response.
5. Speak: When you hear the bell sound, it is your time to speak. Say what you would do in the same situation, as if you were the narrator. Speak clearly so that the computer can understand your words.
6. Play: After each input you provide, an outcome will occur. Keep playing until you reach the end. You’ve reached the end when you hear the narrator say "The End."

Call For Fires

Learn how to transmit a call for fire using the US Army procedures described in FM 6-30.

9-Line MEDEVAC

Learn how to transmit a 9-line medical evacuation request using the US Army procedures described in ATP 4-02.2.

Submerged Vehicle Escape

Learn how to escape a car that has fallen into water.

Vehicle on Fire

Learn how to extinguish an engine fire.

Stalker Survival

Learn how to avoid and survive an abduction attempt.

Active Shooter Survival

Learn how to survive an active shooter on a school campus.

Mobile Platform

In 2019, our research group at USC-ICT prototyped a hardware/software device for using VIANT scenarios in offline, mobile settings. This device, based on the Raspberry Pi 3 Model B+ equipped with a 32GB microSD card, ran a customized version of the Kaldi open-source speech recognition engine to enable offline, large-vocabulary speech recognition. Coupled with our own software for matching user input to scenario outcomes, this device was loaded with all of our VIANT scenarios. Selecting scenarios and adjusting volume was done via a USB 6-button clicker (PC Sensor HK6), and audio/voice interaction was enabled via a USB headset microphone (Koss CS-100). The Raspberry Pi was encased in an aluminum case and UPS HAT power supply (Geekworm X720) powered by two 18650 Lithium Rechargeable Batteries. For easy portability, the device was wrapped in a tactical pouch (LBX Tactical model LBX-1015 in Coyote Brown) and secured with D-Ring (Boosteady). All inclusive, the hardware costs of this prototype was roughly $100 per unit, and we assembled five such units for use in our evaluations.

About VIANT

Voice-controlled Interactive Audio Narratives for Training (VIANT) was an applied research effort at the University of Southern California's Institute for Creative Technologies, executed between November 2017 and October 2019. The project team included Andrew Leeds, David Cobbins, Rob Fuchs, Volodymyr Yanov, Obiageli Odimegwu, and Andrew Gordon (project leader).

The project explored new designs and technologies for audio-only interactive training, where fictional training scenarios were presented to learners as produced audio clips. Learners interacted with VIANT scenarios via voice, expressing what they would do in the fictional situations in natural language speech. Using large-vocabulary speech recognition technologies and statistical narrative language models, VIANT analyzed the learners' statements to decide how the fictional scenario would proceed.

During the course of the VIANT project, the team produced training scenarios that targeted US Army radio communication skills (Call for fires and Medical evacuation), as well as four scenarios designed for civilian audiences (Submerged vehicle escape, Vehicle fire response, Stalker survival, and Active shooter survival). All scenarios are playable online via the links on this page, using the large-vocabulary speech recognition technology built into recent versions of the Google Chrome web browser.

In addition to scenario generation, the VIANT project explored technologies for the deployment of interactive voice-controlled training on low-cost Raspberry Pi devices, for use outside of traditional classrooms as the point of need. The VIANT Mobile Platform is a battery-powered device in a ruggedized case that is used with a hands-free headset microphone, and serves as a prototype for future soldier training systems that utilize on-board large vocabulary speech recognition technologies.

Are you looking instead for interactive fictional scenarios that are not about training? Click here.

Questions or bug reports? Email Andrew S. Gordon at gordon@ict.usc.edu