Pepper gets one phone call
The first real question I asked Pepper’s new web tool was the price of a GPU I already own. Testing software that is nondeterministic by definition is a challenge. The closest thing I’ve found to a test suite is fact-finding questions with known answers. I already knew the card and its approximate price, so I could judge whether the answer was useful and check the sources.
It mattered because giving Pepper, the chief-of-staff agent fronting my fleet, any web reach at all is textbook risk. Pepper reads six private toolsets and holds write access behind an approval gate. Add a tool that returns strangers’ text and you have assembled what Simon Willison calls the lethal trifecta: private data, exposure to untrusted content, and a channel that can carry things out. I wanted the capability anyway, on hardware and guardrails I could shape myself.
The first version kept the web at arm’s length. Searches went through a small relay that can reach the public internet but nothing inside my house, and that only knows how to call one search service. Pepper didn’t even see its web tools unless my message asked for the web.
In practice, that meant starting a question with “search the internet for” to get an internet search. It was unintuitive and cumbersome, even for me, and I knew how it worked. Follow-ups were worse: “I specifically mean the 48GB version” was refused, because that message didn’t ask for the web on its own. Part of my job as a product manager is keeping the software’s limitations from reaching the user, and here I was the user. Nobody else in my household would ever have gotten it right. Now, once I’ve asked for the web, the next few follow-ups in the same conversation carry that permission with them.
The safety side had its own problem. Once a stranger’s web page is in Pepper’s context, that text can try to talk Pepper into reading my private data, and in testing, Pepper followed a planted instruction to search my private documents on 2 of 24 attempts. My first fix took Pepper’s private tools away the moment web results arrived, which meant asking for the private half of a question before the web half, or losing it.
So the web moved out of Pepper. Pepper now has a single web tool: it writes one question and hands it to a separate research helper. The helper does the searching and reading. It has no access to my private data and no memory of our conversation, and Pepper treats whatever it brings back as untrusted.
Pepper gets one research hand-off per turn. It writes that request before receiving the current turn’s web results, so those results cannot alter the request already sent. A page can still try to steer what Pepper does next with my private data, and sometimes it may work. I accepted that. The one-call limit blocks a second research request in that turn. It does not, by itself, prevent an injected instruction from influencing a later turn. A turn that can reach the web is shown only my earlier messages and none of Pepper’s earlier answers, which is where a page’s text would otherwise carry into the next request.
What I traded away is context. The helper that runs the searches knows almost nothing about me, by design. The pricing question didn’t need any context, and it came back with real, sourced figures for a card whose price I already knew. Questions that quietly assume my situation are another matter: they can come back thin, and that’s where I expect to hit the edges of the sandbox I’ve built.