LLM Integration for Mobile and Web Apps: The Production Architecture

Integrating a large language model into an application is architecturally simple and operationally demanding. The request is an HTTP call. What makes it a real engineering project is everything required to make that call fast enough, cheap enough, reliable enough, and safe enough to put in front of paying users.

‍

‍

This is a description of the architecture that works, in the order you should build it.

‍

‍

Rule one: the model call never happens in the client

Every LLM request from a mobile or web application should route through your own backend. There are four reasons and all of them are decisive.

Your provider API key cannot ship in a mobile binary or a JavaScript bundle. Both can be extracted trivially, and an extracted key is someone else's token bill charged to you.

Prompts change constantly in the first months of a feature's life. Prompts held on the server change with a deployment. Prompts compiled into a mobile app change with an app store release and a user update cycle, which means a bad prompt lives for weeks.

Spend control requires a chokepoint. Rate limiting, token caps, caching, and per-user quotas all need a place to live, and that place is your server.

Observability requires the same chokepoint. You need to know what was sent, what came back, how many tokens it consumed, and how long it took, per user and per feature.

A proposal that calls a provider directly from a Flutter, React Native, or browser client is not a shortcut. It is a defect.

The reference architecture

The client sends a normal authenticated request to your API describing the user's intent, never a raw prompt. Your service builds the prompt from a versioned template plus whatever context is needed, checks the cache, checks the user's quota, then calls the provider. The response streams back through your service to the client, and the interaction is logged with token counts and latency attached.

Note the shape of the contract: the client asks for an outcome, such as summarise this document, and the server decides how. Clients that send prompts are clients that will need updating every time the prompt strategy changes.

Bolder Apps builds mobile clients in Flutter and native Swift and Kotlin, web front ends in React, and backends in Node.js and Laravel, which is worth mentioning here only because this architecture is stack-agnostic. The client framework has almost no bearing on LLM integration quality. The server discipline has all of it.

Streaming, and why it is not optional

A model generating two hundred tokens takes several seconds. A user watching a spinner for several seconds concludes the product is broken. The same wait with text appearing progressively reads as fast.

Implementing this means server-sent events or a comparable streaming transport from provider to your server to your client, with the client rendering tokens as they arrive. On mobile this needs care around connection interruption, backgrounding, and partial responses, because a user switching apps mid-stream is a normal event rather than an edge case. Decide what happens to a partial response: discard, persist, or resume.

Cost control, designed in rather than added

Conversation history deserves particular attention, because it is where costs grow without anyone deciding to grow them. Sending an entire conversation with every turn means cost rises with the square of the conversation length. Summarising older turns, or keeping a rolling window, converts that curve into a line.

In practice, the two changes that most often cut an LLM feature's running cost by more than half are routing easy tasks to a cheaper model and capping conversation context. Neither requires touching the interface.

Retrieval, when the model needs your data

If the model must answer using your content, you need a retrieval layer, and its quality determines the quality of the feature far more than the model does.

The pipeline: split source documents into chunks, generate embeddings, store them in a vector index, and at query time embed the question, retrieve the most relevant chunks, and pass them to the model with instructions to answer only from the provided context.

Three decisions carry most of the outcome. Chunk size and overlap, because chunks that split a sentence in half retrieve poorly and chunks that are too large dilute relevance. Whether to combine vector search with keyword search, which reliably improves results when users search for exact terms, product codes, or names. And whether to re-rank retrieved candidates before sending them, which improves precision at a modest cost.

For the vector store, a Postgres database with the pgvector extension is sufficient for most products and avoids adding infrastructure. Dedicated vector databases earn their place at large scale or with demanding filtering requirements.

Structured output, when you need data rather than prose

Many features need the model to return something a program consumes: a category, extracted fields, a decision. Ask for prose and parse it and you will spend months fixing edge cases.

Use the provider's structured output or function calling facilities to constrain responses to a schema, validate every response against that schema on receipt, and define behaviour for validation failure, usually one retry then a fallback. Treat model output as untrusted input, because that is what it is.

Failure handling

Providers degrade. Plan for four states explicitly.

Timeout: define a limit, and decide whether to retry once with backoff or fail. Rate limiting: queue and retry, or return a clear message rather than a generic error. Provider outage: fall back to a secondary provider if you abstracted the interface, degrade to a non-AI path, or disable the feature with an honest notice. Bad output: validation failure handling as above.

Products without defined answers here fail in front of users in ways that are hard to diagnose, because the underlying call succeeded and returned something unusable.

Latency budgets, which decide whether the feature feels good

Users judge an LLM feature on perceived speed more than on output quality, within limits. Build a latency budget explicitly rather than measuring it after complaints.

Account for each stage: the client request, your own authentication and authorisation, retrieval if present, the provider's time to first token, and streaming duration. Retrieval is the stage most often ignored and frequently the slowest, because an embedding call plus a vector query plus optional re-ranking happens before generation even begins.

Three practices help materially. Show something immediately, even an acknowledgement of what is being done, so the interface is never blank. Run retrieval and any other independent preparation concurrently rather than sequentially. And cache aggressively on normalised input, because in most products a meaningful share of queries are near-duplicates of earlier ones.

On mobile, add the reality of network variability. A feature that performs well on office wifi and poorly on a congested cellular connection needs timeouts and messaging tuned for the worse case, and it needs testing under throttled conditions rather than only at a desk.

What to ask a development partner about LLM work

Model fluency is easy to claim and easy to fake. These five questions separate teams that have run this in production from teams that have built a demo.

• How do you control token spend, and what happened to the bill on your last feature when usage grew? A specific story here is the strongest signal available.
• Where do prompts live, and how do you know which version produced a given response?
• What does your product do when the provider is slow or returns an error? Four defined states or a shrug.
• How do you evaluate output quality when you change a prompt? If the answer is that they read some outputs, there is no evaluation.
• How do you keep provider keys out of the client, and how is retrieval scoped to the authenticated user? The second half of that question is where cross-account data exposure hides.

Bolder Apps is an official OpenAI partner with API credits available for qualifying projects and prices project work fixed-scope rather than hourly, and the partnership is worth considerably less as a credential than concrete answers to those five questions are as a filter. Partner listings are obtainable. Production experience is not.

Evaluation and prompt versioning

Once a feature is live, you will change prompts. Without evaluation, each change is a guess validated by however many outputs someone happened to look at.

Build a test set of thirty to a hundred representative inputs with expected characteristics, and run it whenever a prompt or model changes. Store prompts as versioned artefacts and log which version produced each response, so a quality regression can be traced to a change rather than debated.

This is unglamorous infrastructure and it is what separates a feature that improves over time from one that oscillates.

Safety and abuse