The Iceberg of Building an AI Agent Inside Your Product

Updated February 05, 2026

TL;DR: Getting a prototype AI agent running inside your product takes a weekend. Getting it production-ready takes 3-6 months and 2-4 engineers. According to MIT's 2025 research, 95% of enterprise AI pilots fail to reach production, and purchased solutions succeed at roughly twice the rate of internal builds. The hidden work — front-end interactors, session context, monitoring, multi-use-case scaling, prompt regression testing — is the iceberg that sinks most projects. You can build from scratch, use frameworks like the OpenAI Agents SDK or Vercel AI SDK, or deploy a full platform like Tandem that handles the entire stack in under an hour. The right choice depends on whether you want to build infrastructure or ship outcomes.

You got the demo working in a day. The real work hasn't started.

Every engineering team has had the same experience. Someone on the team wires up an LLM API call, drops a chat widget into the product, connects it to some documentation, and the demo is genuinely impressive. The agent answers questions. It even sounds smart. Leadership sees it and asks: "How soon can we ship this?"

That question is where the trouble begins. According to LangChain's 2025 State of Agent Engineering survey of over 1,300 professionals, 57% of respondents now have agents in production — but quality remains the top barrier, cited by 32% of teams. The gap between "it works on my machine" and "it works for 100,000 users reliably" is enormous, and it's almost entirely invisible from the demo stage.

This article maps the full iceberg. Every layer of hidden work you'll encounter when building an AI agent that operates inside your product's user interface — not as a standalone chatbot, but as a genuine assistant that can explain, guide, and execute actions within your application. We'll cover every architectural option, from pure DIY to open-source frameworks to full platforms, so you can make an informed decision about where to invest your engineering time.

Why the MVP deceives you

The prototype feels like 80% of the work. It is roughly 10%.

A basic in-app AI agent needs just a few components to produce a convincing demo: an LLM API call (OpenAI, Anthropic, or any provider), a prompt with some system instructions, a basic RAG pipeline pulling from your docs or help center, and a chat UI component. Modern tooling makes this genuinely fast. The Vercel AI SDK lets you build a streaming chat interface in under an hour. The OpenAI Agents SDK gives you tool calling, handoffs, and tracing in a few dozen lines of Python. You can have something working by lunch.

The problem is that this agent is fundamentally blind and powerless. It can talk about your product. It cannot see what the user is doing, understand where they are in the app, or take any action on their behalf. It's a chatbot wearing a product's skin, and your users will figure that out within three messages.

To build an agent that genuinely operates inside your UI — one that can walk a user through a complex configuration, pre-fill a form, navigate to the right screen, or complete a multi-step workflow — you need to solve an entirely different set of problems. And that's the iceberg.

Layer 1: Context — what does the agent actually know?

The first and deepest challenge is context. A useful in-app agent needs to know far more than what's in your documentation.

User identity and account state

The agent needs to know who it's talking to. Not just their name, but their role, their permissions, their plan tier, their account age, their feature flags, their onboarding status. Without this, every response is generic. "Click the Export button" is useless if that user's plan doesn't include export functionality.

Where does this data come from? Typically from your backend — through direct API calls from the agent's server-side logic, or by passing user context through the front-end when the chat session initializes. Either way, you're building and maintaining an integration layer between your agent and your user data model.

Session context — what's happening right now

The agent needs to understand the user's current session. What page are they on? What have they already done? Did they just encounter an error? Are they midway through a three-step workflow? This is the difference between an agent that can say "I see you're on the billing settings page" and one that responds with "Could you tell me what page you're on?"

Capturing session context requires front-end instrumentation. You need to track the current URL/route, the state of forms and inputs on the current page, recent user actions (clicks, navigations, errors), and the visible state of the UI (which panels are open, which tabs are selected). Building this instrumentation is a significant front-end engineering project. Every page in your product may have different components, different states, and different data structures. You're essentially building a real-time observation layer on top of your entire application.

Front-end awareness — reading the DOM

This is where things get genuinely hard. If you want the agent to understand what the user is looking at — the actual content rendered in the browser — you need to parse the DOM (Document Object Model) and translate it into something the LLM can reason about.

This means identifying interactive elements (buttons, inputs, selects, toggles) and their current states, reading visible text and labels, understanding the visual hierarchy and layout, and tracking which elements are enabled or disabled and why. The challenge is that modern front-end frameworks (React, Vue, Angular) generate complex, deeply nested DOM trees that change constantly. Components mount and unmount. State changes cascade. Building a reliable DOM reader that works across your entire product — and keeps working as your team ships new features — is a substantial, ongoing engineering effort. This is one of the hardest pieces of the entire stack to build and maintain.

Knowledge sources — documentation, APIs, internal data

Beyond the user's immediate context, the agent needs access to your product knowledge. This typically involves a RAG pipeline pulling from help center articles, product documentation, release notes, and internal knowledge bases. You may also expose backend APIs so the agent can look up specific data (e.g., "What's the status of invoice #4521?") or reference internal data about feature configurations, limits, and edge cases.

Each knowledge source requires its own integration, its own chunking strategy, its own retrieval tuning, and its own maintenance process when content changes. This isn't a one-time build — it's an ongoing content operations workflow.

Layer 2: Action — what can the agent actually do?

Once the agent has context, the next question is execution. Can it just tell the user what to do, or can it actually do things?

The explain-guide-execute spectrum

Most in-app agents start at the "explain" level: they answer questions in natural language. Some advance to "guide": they can highlight UI elements, walk users through steps, or point them to the right place. Very few reach "execute": they can take actions directly within the UI on the user's behalf — filling forms, clicking buttons, navigating between screens, completing workflows.

Each level requires dramatically more engineering. Explaining is mostly a prompt engineering and RAG problem. Guiding requires deep front-end integration — you need to programmatically identify, highlight, and annotate specific UI elements. Executing requires the ability to programmatically interact with your application's UI, which means simulating user actions in a way that's safe, reversible, and correctly triggers all the downstream effects (state updates, API calls, analytics events).

Building front-end interactors

If you want the agent to execute actions, you need to build interactors — code that can programmatically interact with your UI components. This means clicking buttons, toggling switches, selecting dropdown options, filling text fields, navigating between pages, and submitting forms.

These interactors need to be resilient to UI changes. If a developer renames a CSS class, moves a button, or refactors a component, the interactor breaks. You can use stable selectors (data-testid attributes, ARIA labels), but this requires discipline across your entire front-end team and adds a dependency between your agent infrastructure and every UI change.

APIs and MCP servers

For backend actions (creating records, changing settings, triggering workflows), you have several architectural options.

Direct API calls from the agent's backend to your product's API are the simplest approach. The agent decides what to do, calls the appropriate endpoint, and returns the result. This works well for straightforward operations but requires building, maintaining, and securing an agent-specific API layer.

Model Context Protocol (MCP) servers provide a standardized way to expose your product's capabilities to an LLM. Instead of custom API integrations, you build MCP servers that describe available tools, and the LLM can discover and invoke them through a consistent protocol. MCP was introduced by Anthropic in late 2024 and has since been adopted by OpenAI, Google DeepMind, and major tooling companies. It reduces the "N x M" integration problem but introduces its own operational overhead — you're now running and maintaining MCP server instances for each capability you want to expose.

Function/tool calling through the LLM provider's native API (OpenAI function calling, Anthropic tool use) is the most common pattern. You define the available functions, the LLM decides when and how to call them, and your backend executes them. This is powerful but requires careful schema design, input validation, and error handling.

Layer 3: Orchestration — making it all work together

With context and action capabilities in place, you need an orchestration layer that ties everything together.

Agent frameworks: your main options

If you're building custom, you'll likely use one of these frameworks:

Each framework solves the orchestration problem — defining how the agent reasons, which tools it calls, and when to hand off between specialized sub-agents. None of them solve the front-end integration problem, the session context problem, or the monitoring problem. Those remain your responsibility.

Conversation and state management

An in-app agent isn't a one-shot Q&A system. Users have multi-turn conversations, leave and come back, switch between tasks, and expect the agent to remember context. You need to manage conversation history (how many messages to keep, when to summarize or truncate), session persistence (what happens when the user refreshes the page or comes back tomorrow), state synchronization between the agent's understanding and the actual application state, and context window management to avoid hitting token limits on long conversations.

The OpenAI Agents SDK provides built-in session memory, and LangGraph offers durable state persistence. But these handle the LLM side — you still need to build the application-state side, ensuring the agent knows when the user has taken actions outside the chat, when the application state has changed, or when another system has modified relevant data.

Multi-step workflow handling

Real product workflows aren't one-step. Configuring an integration might involve navigating to settings, selecting an integration type, authenticating with a third-party service, mapping fields, testing the connection, and enabling it. If your agent handles this, it needs to track where the user is in the workflow, handle errors at each step, recover gracefully when something fails midway, and know when the workflow is complete versus when the user abandoned it.

This is effectively building a workflow engine on top of your agent, which is a significant architectural investment.

Layer 4: Reliability and safety — keeping it from going wrong

An agent that can take actions in your product can also take the wrong actions. Reliability engineering is where production costs escalate.

Guardrails and validation

Every action the agent takes needs validation. Before executing, you should verify that the action is permitted for this user's role and permissions, that the inputs are valid and won't corrupt data, that the action won't have unintended side effects, and that the user has confirmed high-stakes operations (deleting data, changing billing, modifying access controls).

Framework-level guardrails (available in OpenAI Agents SDK and LangGraph) handle input/output validation at the LLM level. But application-level guardrails — ensuring the agent respects your product's business logic, access controls, and data integrity rules — are entirely your problem.

Prompt regression testing

This is the hidden maintenance burden that catches most teams off guard. Your agent's behavior is defined by its prompts. Every time you update a prompt — to fix one edge case, to add a new capability, to adjust the tone — you risk breaking behavior in other scenarios. Unlike traditional code, where you can write deterministic unit tests, prompt changes produce non-deterministic outputs.

You need a prompt testing framework that can run your updated prompts against a comprehensive set of scenarios covering every major workflow in your product, evaluate whether the responses are correct and appropriate, detect regressions (cases that worked before the change but don't now), and run automatically before any prompt change ships to production.

Building and maintaining this test suite is an ongoing investment. Every time your product adds a feature, you need new test scenarios. Every time a customer reports a bad agent response, you need a new regression test. According to LangChain's survey, only 52% of teams with production agents have implemented evaluations at all — meaning almost half are flying blind on quality.

Latency and performance

LLM calls are slow compared to traditional API responses. A single agent turn might involve multiple LLM calls (reasoning, tool selection, response generation), tool execution (API calls, database queries), and context retrieval (RAG lookups, session data loading). End-to-end latency of 3-8 seconds per response is common, and complex multi-step operations can take much longer. You need streaming responses to keep the user engaged, optimized context windows to minimize token usage, caching strategies for repeated queries, and graceful handling of timeouts and rate limits from your LLM provider.

Layer 5: Monitoring and analytics — understanding what's happening

You've shipped the agent. Now how do you know if it's working?

Conversation-level monitoring

You need to monitor every agent session: what the user asked, what the agent did, whether the task was completed successfully, where conversations broke down, and what error states occurred. This isn't standard application monitoring — it's a new category of observability that combines traditional logging with LLM-specific metrics like token usage, response quality, and tool call success rates.

Tools like LangSmith, Helicone, or Logfire provide LLM-specific tracing. But you'll likely need custom monitoring that ties agent actions to your product's business metrics: did the agent successfully help the user activate a feature? Did ticket volume decrease? Did the user complete the workflow they started?

Clustering and pattern analysis

At scale, you'll have thousands of conversations to analyze. You need automated clustering to identify the most common user questions and intents, recurring failure patterns (where the agent consistently struggles), emerging topics (new questions that indicate product confusion), and quality trends over time.

Building this analysis pipeline requires additional infrastructure — either custom-built or through a dedicated tool — and someone on your team needs to review the insights regularly and act on them.

Success metrics and attribution

The hardest monitoring challenge is measuring whether the agent is actually delivering value. Defining "success" for an in-app agent is context-dependent: was the user's question answered (satisfaction)? Did the user complete the action they were trying to do (task completion)? Did the user avoid contacting support (ticket deflection)? Did the user discover and adopt a feature they weren't using (activation)?

Each metric requires different instrumentation, different baselines, and different attribution logic. Building this measurement infrastructure is itself a project.

Layer 6: Scaling to multiple use cases — the multiplication problem

Here's where the iceberg gets truly daunting. Everything described above gets you an agent that handles one use case well. Your product has dozens.

The single-use-case trap

Most teams build their first agent for one workflow — typically onboarding or a specific feature configuration. It works well for that scenario because they've hand-tuned the prompts, built the specific interactors, and tested thoroughly for that flow.

Then someone asks: "Can we use this for the billing section too? And the integrations page? And the admin console?"

Each new use case requires new front-end interactors for different UI components, new prompt engineering and context specific to that area, new test scenarios and regression coverage, new knowledge source integrations, and often new guardrails specific to the sensitivity of that area (billing guardrails are different from onboarding guardrails).

You haven't built a product — you've built a template. And every new use case is a mini-project.

Chat versus embedded UI

A chat interface is the default, but it's not always the right modality. Some actions are better served by the agent proactively highlighting a button, pre-filling a form, or showing a contextual tooltip. Supporting multiple interaction modes (chat, guided walkthroughs, proactive nudges, in-line assistance) requires building different front-end components for each mode, orchestrating when each mode is appropriate, and maintaining consistency in the agent's behavior across modes.

This is no longer an AI project — it's a full product within your product.

What this actually costs

The numbers vary widely, but the patterns are consistent.

According to industry data, building a mid-complexity AI agent in-house costs $50,000-$150,000 for initial development, with ongoing costs of $5,000-$25,000/month for LLM API bills at scale, plus $1,000-$2,500/month for prompt maintenance and testing, and $200-$1,000/month for monitoring tools. In engineering time, most projects take 10-12 weeks from concept to deployment — and that's for the initial version, not the ongoing maintenance.

For an in-app agent that genuinely operates inside your UI (not just a chatbot overlay), plan for the higher end of these ranges. The front-end integration, session context, and interactor layers add significant complexity that most cost estimates don't account for.

And here's the sobering context: MIT's 2025 research found that internally built AI tools succeed only about a third as often as purchased solutions. S&P Global's survey found that 42% of companies abandoned most of their AI initiatives in 2025, up from 17% the previous year. The average organization scrapped 46% of proof-of-concepts before they reached production.

The option spectrum: build, framework, or platform

Given the full iceberg, you have three broad approaches.

Option 1: Build from scratch

You pick an LLM provider, build your own orchestration, and engineer every layer yourself.

When this makes sense: You have unique requirements that no framework or platform can accommodate — perhaps a highly specialized UI technology, extreme security constraints, or a need for deep integration with proprietary infrastructure. You also need a team of 2-4 engineers who can dedicate 3-6 months to the initial build and maintain it indefinitely.

What you own: Everything. Context layer, action layer, orchestration, monitoring, testing, and ongoing maintenance. You also own every bug, every edge case, and every weekend on-call rotation when the agent starts hallucinating in production.

Option 2: Use an agent framework

You adopt the OpenAI Agents SDK, Vercel AI SDK, LangGraph, or CrewAI to handle orchestration, and build the rest yourself.

When this makes sense: You have strong engineering capacity and want to move faster than building from scratch, but you still need custom control over the front-end experience and business logic. This is the most popular path — LangChain's survey shows the majority of production agents are built on frameworks.

What the framework gives you: Orchestration, tool calling, some tracing, and (in some cases) session management. What you still build yourself: Front-end integration, DOM awareness, interactors, session context, prompt testing, monitoring dashboards, multi-use-case scaling, and the entire content layer (help docs, prompts, flows).

Option 3: Deploy a purpose-built platform

You use a platform like Tandem that provides the complete stack: context awareness, front-end integration, action execution, monitoring, and analytics — pre-built and ready to deploy.

When this makes sense: Your goal is outcomes (activation lift, ticket deflection, feature adoption), not building infrastructure. You want to deploy across multiple use cases quickly and iterate on the experience without rewriting code. Tandem specifically implements the explain/guide/execute framework — the agent can explain features in context, guide users through workflows step by step, and execute actions directly within the UI.

What the platform gives you: The entire stack. Tandem deploys via a JavaScript snippet in under an hour, requires no backend changes, includes a no-code configuration interface, and self-heals when your UI changes. Monitoring, session analytics, conversation clustering, and success metrics are built in.

What you still own: Content strategy. Like every digital adoption solution, you still need to define the flows, write the messaging, choose the targeting rules, and iterate on the experience based on data. No tool — built or bought — eliminates this work.

Proof points: Qonto used Tandem to activate 100,000+ users on paid features and guided 375,000 through an interface redesign, with account aggregation adoption jumping from 8% to 16%. Aircall achieved a 20% activation lift for self-serve accounts. Sellsy saw an 18% activation lift across their CRM onboarding.

Honest limitations: Tandem is web-only (mobile support is coming), does not include deep product analytics (pair it with Amplitude, Mixpanel, or your existing analytics stack), uses custom pricing (no public tiers), and is an early-stage company founded in 2024. These are real trade-offs to evaluate against your requirements.

The decision framework

The right choice depends on your situation, not on the technology.

Build from scratch when your product has a highly unusual UI stack (not standard web), you have 3+ engineers who can dedicate months to this and maintain it long-term, your security or compliance requirements preclude any third-party tools, and you want to treat in-app AI as a core competitive differentiator you'll invest in for years.

Use a framework when you have a strong engineering team that wants control over the experience, you're willing to invest 10-12 weeks for the initial build plus ongoing maintenance, you need deep customization in how the agent reasons and acts, and you're comfortable owning the front-end integration and monitoring stack.

Deploy a platform when you want to ship across multiple use cases in weeks, not months, your primary metrics are business outcomes (activation, deflection, adoption), you don't want to hire and maintain a dedicated agent engineering team, and you value a self-healing architecture that adapts to UI changes automatically.

ROI calculation framework

Here's a simple model to evaluate the build-vs-buy economics. Fill in your own numbers.

Your costs (build in-house):

• Engineering time: [number of engineers] x [hourly rate] x [hours over 3-6 months]

• LLM API costs: [estimated monthly token usage] x [per-token rate]

• Monitoring infrastructure: $500-$2,000/month

• Ongoing maintenance: [engineers] x [10-20 hours/month]

Worked example (mid-market SaaS):

• 2 engineers x $150/hour x 500 hours = $150,000 initial build

• $3,000/month LLM costs + $1,000/month monitoring = $48,000/year ongoing

• 1 engineer x 15 hours/month x $150/hour = $27,000/year maintenance

• Total first year: ~$225,000

Potential value (using Tandem case study benchmarks):

• 20% activation lift (Aircall) on X thousand users = Y revenue impact

• Ticket deflection at industry average of 23% on Z tickets/month at $15-35 per ticket = W savings/month

• Feature adoption increase (Qonto: 8% to 16%) on key paid features = V revenue impact

Compare total first-year build cost against platform pricing and expected value. For most teams, the math favors buying unless the in-app agent is a core strategic investment you plan to differentiate on for years.

CTA

If you want to see what the full stack looks like without building it, book a 20-minute demo with Tandem. Bring your most complex workflow — the one that generates the most support tickets or has the steepest activation curve — and watch Tandem walk a user through it end-to-end. You can also request reference calls with product leaders at Aircall and Qonto to hear how deployment went at scale.

FAQ

How long does it take to build an in-app AI agent from scratch versus deploying a platform?

Building from scratch typically takes 10-12 weeks for a single use case, assuming 2-3 dedicated engineers. Deploying a platform like Tandem takes under an hour for technical setup (a JavaScript snippet), with additional time for content configuration — typically days to a few weeks depending on how many flows you want to launch. The difference is even more pronounced when scaling to multiple use cases: each new DIY use case is a mini-project, while platforms handle multiple flows through configuration.

Can I start with a framework and migrate to a platform later (or vice versa)?

Yes, but be aware of the switching cost. If you build on a framework like LangGraph or the Vercel AI SDK, you're investing in custom front-end integration, monitoring, and testing infrastructure that won't transfer to a platform. Moving from a platform to custom is easier — you'll already have validated which flows matter, what users ask, and where the agent adds value. Some teams start with a platform to prove ROI and only build custom for highly specific use cases where the platform doesn't reach.

How do I monitor AI agent quality in production?

You need three layers: conversation-level tracing (what the agent said and did), task-completion tracking (did the user achieve their goal), and business metric attribution (did activation, ticket deflection, or adoption actually improve). Frameworks provide the first layer through built-in tracing (LangSmith, OpenAI's tracing). The second and third layers require custom instrumentation or a platform that includes them natively.

Does Tandem support mobile apps?

Not yet. Tandem currently supports web applications. Mobile support is on the roadmap. If mobile is a critical requirement today, you'll need a custom build or a framework approach for mobile surfaces, potentially paired with Tandem for web.

What's the typical ROI timeline for an in-app AI agent?

Teams using platforms like Tandem typically see measurable results within 2-4 weeks of launch, since deployment is fast and the analytics are built in. Custom-built agents have a longer feedback loop: 10-12 weeks to build, then additional time to instrument monitoring and gather enough data to measure impact. Plan for 3-6 months before you have confident ROI data on a custom build.

Can frameworks like the OpenAI Agents SDK handle front-end UI interaction?

Not natively. Agent frameworks handle LLM orchestration — reasoning, tool calling, handoffs, and tracing. Front-end integration (DOM reading, element interaction, session context, UI highlighting) is entirely your responsibility. This is the single biggest gap between what frameworks provide and what an in-app agent requires.

What happens when my product's UI changes — does the agent break?

If you've built custom interactors using CSS selectors, element IDs, or hard-coded paths, yes — UI changes will break the agent until someone manually updates the interactors. This is one of the most painful maintenance burdens of the DIY approach. Tandem's self-healing architecture adapts to UI changes automatically, which eliminates this maintenance category but trades off some control over exactly how the agent identifies elements.

Glossary

In-app AI agent: An AI system embedded directly within a software product's user interface that can understand user context, answer questions, guide workflows, and execute actions — as opposed to a standalone chatbot or external assistant.

DOM (Document Object Model): The structured representation of a web page's content and elements that browsers use to render UI. AI agents that interact with product UIs need to parse the DOM to understand what's visible to the user and to programmatically interact with interface elements.

Model Context Protocol (MCP): An open standard introduced by Anthropic in November 2024 that standardizes how AI systems integrate with external tools and data sources. MCP provides a universal interface so developers can build one integration that works across multiple AI providers, rather than custom connectors for each.

Explain/guide/execute framework: A three-tier model for in-app AI assistance. "Explain" means answering questions in natural language. "Guide" means walking users through workflows step by step with visual indicators. "Execute" means the agent takes actions directly within the UI on the user's behalf. Each level requires progressively more engineering investment.

Self-healing architecture: A design pattern where an in-app agent automatically adapts to changes in the host product's UI — such as renamed elements, moved buttons, or restructured pages — without requiring manual updates to the agent's configuration.

Prompt regression testing: The practice of running updated AI prompts against a comprehensive set of test scenarios to ensure changes haven't broken previously working behaviors. Analogous to unit testing in traditional software, but more challenging because LLM outputs are non-deterministic.

Agent framework: An open-source library (such as OpenAI Agents SDK, Vercel AI SDK, or LangGraph) that provides orchestration primitives for building AI agents — including tool calling, multi-agent handoffs, guardrails, and tracing. Frameworks handle the LLM interaction layer but do not provide front-end integration, UI interaction, or business-specific monitoring.

Activation rate: The percentage of new users who complete a key action that correlates with long-term retention and value. B2B SaaS median activation rate is approximately 36-37.5%, with top performers exceeding 50%.