A few days ago, during the keynote at Google I/O, Sundar Pichai demoed a project called Duplex, a system whereby its voice assistant technology is applied to tasks like booking appointments, making restaurant reservations, or discovering opening hours of businesses.

It’s a striking demo, if you haven’t watched it. It contains a couple of recorded calls wherein Duplex, posing as a human, conducts conversations leading to a haircut appointment and discovering that a restaurant doesn’t take reservations for parties of fewer than 5.

There’s a post on the Google AI blog about Duplex that goes into more detail about the implementation, and mentions that difficult situations may be passed off to a human operator, though it claims that Duplex is able to handle most interactions completely autonomously.

It also makes some claims about transparency:

The Google Duplex technology is built to sound natural, to make the conversation experience comfortable. It’s important to us that users and businesses have a good experience with this service, and transparency is a key part of that. We want to be clear about the intent of the call so businesses understand the context. We’ll be experimenting with the right approach over the coming months.

…which sounds reassuring, but the sample call audio raises some pretty serious questions.

1. Will the voice assistant (Duplex) announce that it is a digital assistant?
2. Will Duplex state that the call is being recorded and analyzed, and will there be a public data retention policy available? In other words, where are all the calls stored, for how long, etc.?
3. Does Google get (or did they get) permission for the demos at I/O from the salon and the restaurant?
4. What tools and assurances will Google provide to make sure manipulation of humans who answer phones is not abused?
5. If someone is called by Duplex and does not wish to be called, will there be a “do-not-duplex me” service?
6. Will people be able to discover their own history of interacting with Duplex? Will third parties? What are the implications for the personal privacy of individual employees at businesses that receive Duplex calls?
7. Does a voice assistant which retains data from calls violate wiretapping prohibitions?

There are complicated questions about whether technologies like Duplex (or any of Siri, Alexa, Google Search, etc.) constitute true “artificial intelligence”, but it’s clear that they’re having profound effects on human cognition, capabilities, and freedoms. Effects which will only increase in magnitude as the technology is refined and becomes ever-more pervasive in day to day life and the economy.

Google is already a corporation with almost unfathomable power to shape our society’s understanding of itself and the flow of our attention and knowledge about the world.  This is a power that’s only going to grow as it gets harder for people to tell the difference between talking to another human mind and talking to massive databases engineered to maximize corporate profits.  It’s past time that we articulate a clear expectation of the rights that human minds should retain in the face of this reality, and the responsibilities that builders of systems have to respect human consent and human freedom from cognitive manipulation.

Update: 5/10/18 – “Google says it’s designing Duplex with ‘disclosure built-in’”

“We understand and value the discussion around Google Duplex — as we’ve said from the beginning, transparency in the technology is important,” a Google spokeswoman said in a statement. “We are designing this feature with disclosure built-in, and we’ll make sure the system is appropriately identified. What we showed at I/O was an early technology demo, and we look forward to incorporating feedback as we develop this into a product.”

Update: 5/17/18 – “What Google isn’t telling us about its AI demo”

Axios asked Google for the name of the hair salon or restaurant, in order to verify both that the businesses exist and that the calls were not pre-planned. We also said that we’d guarantee, in writing, not to publicly identify either establishment (so as to prevent them from receiving unwanted attention).

A longtime Google spokeswoman declined to provide either name.

We also asked if either call was edited, even perhaps just cutting the second or two when the business identifies itself. And, if so, were there other edits? The spokeswoman declined comment, but said she’d check and get back to us. She didn’t.