Conversational Agents and Personal Privacy Harms Case Study

Abstract

This fictional case study examines the question of whether a personal conversational agent/advisor called iSoph, encoding extensive personal information and having a fluent natural language interface, may raise privacy harms that are normally thought to attend to personal relationships, as opposed to harms associated with institutional databases and big data analytics.  The case stipulates that (i) iSoph collects extensive personal data about its users from multiple, multimodal sources, (ii) can make inferences from this data in combination with its models, but (ii) cannot share information with its developer or any third party. It is shown that despite condition (iii), iSoph raises privacy risks. These privacy risks are of the type associated with personal relationships and direct observation. iSoph raises these risks because, as an advanced conversational agent, it is able to evoke anthropomorphizing responses from its users in ways that they are not fully conscious of or able to control.