Moto X is Always Listening, What Could Possibly Go Wrong?

Yesterday, Motorola announced the Moto X; the company's first from-scratch Android phone since being acquired by Google. Among its many features is close integration with Google Now's Glass-esque voice commands. That's nothing to worry about—right?
In her hands-on with the device, PC Mag's Chloe Albanesius described the feature thusly: "Once activated, you can talk to your Moto X from up to 15 feet away, asking it to call certain people, look up things online, or get directions. Say "OK Google Now" to wake it up and command away."
Chloe also pointed out that, unfortunately, there's no way to change your command phrase—so just about anyone could walk up and give your phone a command. Like most voice recognition software, the Moto X can be trained to your voice, making it at least less likely that someone could give your phone a command for you.
That's not what really worries me, though. I've already heard stories about pranksters shouting commands at Google Glass wearers and the devices responding. What bugs me is that this is a phone—a connected network device—that is always waiting for commands. It is, in short, always listening.
What's the Problem?
Most phones or headsets don't have physical switches for microphones, meaning that they could theoretically be activated without your knowledge. This is an issue that the government is already worried about. The difference, to my mind, is that the Moto X's mic is already running all the time. Someone aiming to exploit it doesn't need to figure out how to turn the mic on, just how to get access to the information it gathers.
In the era of NSA PRISM and Xkeyscore surveillance, that's pretty worrisome.
Now before you assume Google wouldn't do something really stupid, remember that Google Glass originally read any QR code that it happened to photograph and would execute it without first informing the user. The issue was caught by Lookout and Google quickly and quietly issued a fix, but it shows that the company doesn't always get it right the first time around.
Of course, Google Glass is an experiment made by Google and in the hands of a select few while the Moto X will be in wide release and made by the venerable phone manufacturer Motorola. Hopefully, the bone-headedness will be kept to a minimum.
But even if there isn't an obvious flaw out-of-the-box, the opportunity to take advantage of an "always listening feature" is sure to attract some attention among security researchers. Perhaps we'll be seeing this at Black Hat 2014.