Investigating Machine-Learning Interaction with Wizard-of-Oz Experiments
Proceedings of the NeurIPS 2018 Workshop on Learning by Instruction
2018
Abstract
Machine-learning research has historically focused on improving learning algorithms but often neglects the context within which a learning technology is situated, and only minimally evaluates how natural and efficient these systems are for users to interact with. To better investigate these aspects of machine-learning, we introduce a novel experimental methodology, which we refer to as dual-sided, restricted-perception Wizard-of-Oz experiments, and describe a web-based framework for executing them. In classic Wizard-of-Oz experiments, a researcher rapidly tests a hypothetical Artificial Intelligence system with real users by manually simulating its capabilities. We extend this approach to support the prototyping and evaluation of interactive learning systems by instead simulating the learning technology using a naïve experimental participant, who learns the task by receiving the same inputs and outputs as the hypothetical learning system. We present proof-of-concept efforts to use this paradigm to evaluate hypothetical learning systems in the domain of service robotics.
BibTeX
@inproceedings{sheline-neurips-2018,
title = {Investigating Machine-Learning Interaction with Wizard-of-Oz Experiments},
author = {Sheline, Rob and MacLellan, Christopher J.},
booktitle = {Proceedings of the NeurIPS 2018 Workshop on Learning by Instruction},
year = {2018},
}