When Search Stopped Being the Product

Raffle’s chat launched in 2024 as a conversational layer over the search engine: every message ran a search, and a language model summarised the results. It did what it was built to do, and customers bought it. But a year of real conversations showed that many of the people using it were not searching at all — they were asking to reach a human, or to cancel, return, and get refunds. They were asking to do things, and the product could only answer questions. I proposed inverting the architecture: the chat becomes an agent with a set of tools, and search becomes one of them.

I ran the discovery, set the product direction, and built the proof of concept. The production implementation was built by the data science and development teams — a data science colleague contributed through ideation, worked through the technical questions with me, and led the production build.

Context and responsibility

Raffle began as B2B AI vector search, sold both for customers’ websites and for internal use, then added a generative layer on top of it. That layer became a chat product sold for customer websites, and over time the focus narrowed to websites alone. Today the company is about website experience in two directions at once: helping visitors find what they came for, and helping the customer move them toward the pages that matter commercially.

I own product direction, and the chat was the part of the product I had put my attention on. Search was mature when I arrived and has stayed with data science and the CTO — answer quality is their work, not mine. The chat was the less mature side, so it is where I spent my time.

Problem and evidence

The 2024 architecture had one shape: a message came in, a search ran, and a language model turned the results into a readable answer. That is a coherent design for a product whose job is retrieval, and it worked.

Two things stopped fitting. The first is visible in a single behaviour — the system ran a search on every message, including when someone typed nothing more than “hi”, because it had no way to decide what kind of response a message deserved. The second is that the market moved underneath it. Users increasingly expected a chat to understand intent, take actions, and call functions rather than only retrieve. Competing products had moved in that direction and we had not, which is where discovery started.

So I began with a competitor analysis, and then went looking for what our own users were doing. I exported roughly a year of real chat conversations across several customers and clustered them to find recurring needs and recurring failures.

The clusters are the reason this project exists. Many of the interactions were not searches at all. People were asking to talk to a human. People were asking about refunds, returns, and cancellations. Those are requests to act, and no improvement to a search-summarising architecture reaches them — a better summary of a help article is not a live-chat handover, and it is not a cancellation.

Options and decision

Option	What it optimised for	Decision
Improve the search-and-summarise chat	The cheapest path, with no architectural risk	Rejected. It makes the answers better for people who were asking questions, and does nothing at all for the people asking to do something.
Build the customer-facing tool platform	The full long-term product in one move	Rejected for now. It is the right destination, but it was too large to deliver, and nothing yet told us which tools were worth generalising.
An agent with tools, and custom tools in the codebaseChosen	Serving the clusters now while learning what to generalise later	Chosen. Search becomes one tool among several, and the first custom tools are implemented directly in the codebase for specific customers.

Three ways to respond to what the conversations showed.

The inversion is the whole decision. Instead of a chat built on top of search, the chat became an agent with access to a collection of tools, and search became one of those tools. A question that needs information calls search, reviews the results, and summarises them. A greeting gets answered without a search running behind it.

What matters more than either behaviour is what the shape allows. The same structure admits tools that have nothing to do with retrieval: querying a database, fetching a specific record, guiding someone through a form, reaching data only an authenticated user can see. Which tools exist can differ per customer, because what a visitor needs to do differs per customer.

What I deliberately did not build

The long-term version of this is a platform where customers build their own tools and functions for their website experience. That is the destination, and it was too large to deliver as the first step.

So we took the pragmatic route: build the architectural foundation so that custom tools can be implemented directly in the codebase for a specific customer’s use case, and treat each of those implementations as an experiment. Each one tells us whether that capability is valuable enough to generalise and offer to everyone.

That is the part of the decision I would defend hardest. It replaces a guess about which tools customers want with a mechanism that produces the answer, and it costs one architectural choice rather than a platform build.

Trade-offs

One-off tools carry per-customer engineering cost, and every one of them is a small commitment to maintain something that may never generalise. That is the price of learning this way rather than deciding up front, and it is real.

The larger trade-off is that this was a rebuild of the core of a shipped product rather than an improvement to it. The 2024 chat was doing its job for the customers who had bought it, and choosing to re-architect underneath them meant taking on that risk deliberately.

Execution

The prototype, and why it was built the way it was

I built the proof of concept myself, and two choices about how mattered more than the code.

First, I built it inside the constraints of the existing product and technical setup rather than in a clean room. A greenfield prototype would have been faster and would have proved the wrong thing — that the idea works somewhere, rather than that it works here, against the systems we actually had.

Second, I rebuilt the original chat’s existing functionality in simpler form on the new tool-based architecture before adding anything new. That gave a like-for-like comparison: the same job, done both ways, with the difference attributable to the architecture rather than to a longer feature list.

The result was strong enough to act on. The prototype avoided unnecessary searches, responded appropriately to different kinds of input, and established a foundation for capabilities beyond search. That judgement is not only mine — QA tested it against the old chat, and the tech team reviewed it internally before the company committed to it.

From prototype to production

The research and the proof of concept were finished in Q1 2026, and production development followed in Q2 2026. The concept and the architecture from the prototype became the foundation of the chat, and the new version shipped in August 2026.

I did not build the production version. The data science and development teams did, and a data science colleague led that work after contributing through ideation and helping me think through the technical questions. My contribution was the discovery, the product direction, and the initial technical prototype that made the direction concrete enough to commit to.

Result and limits

The new chat shipped in August 2026 on the architecture the prototype established. Search is one tool among several, the chat can decide how to respond to what is actually being asked, and there is a foundation for tools that do things rather than find things.

There is no outcome measure, and it would be too early to have one. The product shipped days before this was written. There is no adoption figure, no usage comparison, no support-deflection number, and no commercial result — and there is no earlier measurement to compare against either, because the case for the change came from conversation analysis rather than instrumentation.

What exists instead is validation before the decision rather than after it: a prototype tested by QA against the product it would replace, and reviewed by the engineering team who would have to build it. That is the honest claim, and it is a weaker one than a number. I would rather publish it that way than dress a shipped architecture up as a result.

Learning

The useful part of this was not choosing an agent architecture. That direction was in the air, and adopting it because the market had adopted it would have been a fashion decision.

The part I would repeat is looking at a year of what people had actually typed before deciding anything. The conversations did not say “we want an agent”. They said people were trying to reach a human and trying to cancel things, inside a product that could only look things up. The architecture followed from that, which is why I could argue for it — and it is also why the scope stayed honest, because the same evidence that justified the rebuild said nothing about which tools to build next.

The other lesson is about prototyping as a non-engineer. A prototype in a clean room proves an idea. A prototype built inside the real stack, doing a job the current product already does, is something a team can compare, argue with, and decide on. The constraint made it useful.

Evidence note

This case study uses verified interview evidence covering the original architecture, the discovery method, the architectural decision, the deferred platform scope, the prototype and its validation, the delivery dates, and the contribution boundary. The conversation analysis is described in aggregate only: no customer is named or characterised, and no conversation is quoted or paraphrased. No colleague is named. It contains no internal document, ticket, screenshot, or commercial figure, and it makes no claim about model behaviour or answer quality, which are not my work.

Read Owning Product at Raffle AI for the company context and my wider Product Manager scope, or Making a Rebuild Worth Funding for a focused example of getting an architectural rebuild approved and what its schedule miss cost.