An artificial intelligence agent researching a medical condition can learn something about an entire family from a conversation with just one member.
Catherine Tucker, professor of management at MIT Sloan, used that example to describe a weakness in privacy rules built around individual disclosure. A person asking about macular degeneration might volunteer a family history to obtain a useful answer, revealing information about relatives who never agreed to share it.
“Do they have the same degree of economic security, which leaves me to reveal this very personal and potentially problematic information? No,” Tucker told Competition Policy International (CPI), a PYMNTS company, in an interview.
The hypothetical illustrates an economic issue that runs through her research, conducted in collaboration with Rozhina Ghanavi, a PhD candidate at MIT Sloan. The person receiving the benefit of disclosure does not necessarily bear all its costs.
Agentic AI delegates more of these choices to software. An agent given access to email, calendars, customer records or payment systems makes decisions about retrieving and using information while completing a task. Privacy arrangements premised on a person approving each disclosure become difficult to apply when the person has delegated that judgment.
Ghanavi described the object of that consent, saying that with agentic AI, “consent doesn’t go directly to data, it goes to some sort of permission.” Ending a connection, limiting its duration and deciding what information an agent retains require answers beyond permission to process a particular record.
For banks and payment providers deploying agents, questions continue to swirl around the authority granted across systems and the information available during an assignment. Access to a calendar or customer database also reveals relationships and routines beyond the immediate transaction.
Generative AI changes the value of the information being supplied, the MIT researchers said. Prompts, documents and corrections support personalization, evaluation and model improvement. The commercial asset can reside in model weights, a specialized model or the ability to generate useful responses. These capabilities continue to have value after the original exchange ends.
Ghanavi used artwork submitted to a foundation model to explain the limits of deletion. Removing the original material after training or fine-tuning does not necessarily remove its contribution to the model. A right attached to a file therefore leaves further questions about the capabilities developed from it.
Data Rules and Acquisitions
Tucker traced existing privacy approaches to a period, decades ago, when policymakers expected to identify a piece of data, secure consent and specify its use. Inference makes that sequence unreliable because ordinary information acquires significance through its association with other details.
“The current privacy regime is focused on trying to highlight data that by itself may be considered sensitive,” Tucker said.
A preference or behavior that appears harmless in isolation can still support conclusions about personal characteristics.
“This has long been a question of how we deal with property rights and ownership in a world where data is shared,” Tucker said, adding that it’s not just an AI issue. Rules assigning control to an individual confront information whose value and consequences extend across a household, workplace or group.
Tucker and Ghanavi’s research, and the resulting paper presenting that research, said efforts to address these problems will also determine access to the information needed to build competing AI services. Restrictions on training, inference and sharing affect developers seeking realistic interactions to improve their products, while compliance costs can favor companies with established customer relationships and broader operations.
Ghanavi’s separate research on advertising technology examines how the California Consumer Privacy Act influenced merger activity. She described firms without first-party data seeking combinations with companies that had it, bringing data exchanges inside the resulting business.
For foundation-model developers, owning applications offers direct access to user interactions. Ghanavi said she expects privacy restrictions to add to the attraction of bringing those applications into the same firm. As she put it, “they’re more willing to internalize those functions and have a more vertically integrated AI stack.”
The trends raise questions for authorities examining horizontal consolidation among competitors. What happens when companies acquire businesses at other stages of AI development and distribution?
Tucker said regulators should scrutinize the treatment of first-party and third-party information. Buying an application changes the corporate relationship between businesses exchanging data, even when the underlying information remains the same.
Privacy and competition authorities need to examine that distinction, including whether restrictions on external sharing encourage acquisitions that bring the same activity inside one company, Tucker said.
Watch the full interview with Catherine Tucker and Rozhina Ghanavi to learn more about:
- Why curly fries appear in an economist’s explanation of sensitive information.
- How personalized answers change consumers’ willingness to disclose personal details.
- Why the researchers raise synthetic data as an option for protecting competition.
For all PYMNTS AI coverage, subscribe to the daily AI newsletter.
