One Server or Two? How to Place an AI Application and Its Inference Service

Start with the two deployment patterns

In a colocated design, the user-facing AI application and the service that runs model inference share a server. They may still be separate software processes or containers, but use the same host.

In a separated design, the application sends inference requests to a service on another server or managed endpoint. Microsoft describes a pattern in which remote clients share a centrally operated model endpoint ([Microsoft documentation](https://learn.microsoft.com/en-us/windows-server/ai/local-ai-inference)). This supports that specific pattern; it does not compare the two layouts or establish which is preferable.

The questions below are a way to investigate your own requirements, not universal guidance about which topology performs better or is easier to operate.

• Colocated: what would sharing a host mean for your resources, maintenance and recovery arrangements?
• Separated: what network, endpoint and ownership arrangements would the application depend on?

Map the components and questions to investigate

Draw the system as components and connections, not just as a list of servers. Include the user-facing application, inference runtime, model files, database, file or object storage, external APIs, identity systems and background jobs. Mark which components a request requires and which are optional.

For each component, note where it runs, who maintains it, what resources it uses and what your users would experience if it were unavailable. These are items to verify against your application’s supported configuration and your own operating requirements—not conclusions that follow from choosing one layout.

Write down the tasks users perform, expected concurrency and whether requests arrive in bursts. If you do not know these values, treat them as questions for a pilot rather than filling gaps with generic performance estimates.

• List the application, inference service, database, storage, scheduled work and external APIs.
• For each connection, record its purpose, direction, credentials and expected behavior when the dependency is unavailable.
• Identify possible resource overlap to investigate, such as application activity and model execution using the same CPU, memory, accelerator, disk or network capacity.
• Record where model files, prompts, uploaded documents, generated responses and application records are stored.

Questions for a colocated design

Colocation may be a candidate if you are considering keeping both services on one host. Before deciding, investigate whether your application supports the intended connection, whether model files fit your storage plan, and whether your team can observe and recover both components.

Ask what would happen if application activity and inference competed for resources, or if a host maintenance event or problem affected both services. The answer depends on your configuration and acceptable user experience; the layout alone does not establish the impact.

• Is it acceptable for application and inference activity to share a host and its maintenance arrangements?
• How will you check application behavior while representative inference tasks run?
• What should users experience if inference is busy or unavailable, and can they still use non-AI parts of the application?

Questions for a separated design

A separate inference service creates a distinct service and network boundary. Investigate whether that boundary addresses a specific requirement, such as different maintenance arrangements, resource management or administration. Separation alone does not establish isolation, security or improved performance.

Microsoft documents a centrally operated inference endpoint shared by remote clients. If you are considering that pattern, verify who may call the endpoint, how access is granted and revoked, and whether the applications may send the same categories of data.

Consider what the application should do if it cannot reach the endpoint, credentials are invalid, or the endpoint returns an error. Assign responsibility for endpoint monitoring, maintenance and recovery as part of evaluating the design.

• What requirement would a separate service boundary address?
• Who owns the endpoint, access control, maintenance and incident response?
• How will network and endpoint errors be handled and communicated to users?

Trace data and responsibilities

Draw the data path for a real task. It may include user input, application instructions, retrieved documents or other context, a request to the inference service and a response. The actual path depends on the application and its configuration, so verify what is sent rather than assuming.

For each data item, ask whether it crosses a service boundary, where it is processed and whether it is stored in application records, service logs, backups or another system. Do not infer logging or retention behavior from where the services run.

Record who can access data and credentials, what retention settings apply and how backups are handled. Review external API calls separately; a two-server design does not imply that all processing stays within those servers.

• Trace inputs, retrieved context, attachments, outputs, identifiers and credentials separately.
• For each component, verify what may be logged, what is retained, who can access it and how long it is kept.
• Confirm what application data and configuration are included in backups; separately document how model files or other large assets are managed.
• Assign owners for data, access and retention decisions.

Use a pilot to answer your open questions

A pilot can help you assess your application, model configuration, data and operating conditions. Generic benchmark claims cannot establish whether your own tasks, concurrency pattern or failure expectations will be acceptable.

Run representative tasks against the layout or layouts you are considering. Include normal use and the busiest realistic period you can reproduce. Decide in advance what you will observe—such as application behavior, inference completion, resource use and suitability of results—and define acceptance criteria for your use case.

If appropriate for your environment, test relevant failures in a controlled setting: for example, an unavailable endpoint, interrupted network path or invalid credentials. Record the observed behavior and recovery steps rather than assuming how the system will respond.

• Choose representative tasks and inputs, including relevant document retrieval or external API steps.
• Test the expected usage pattern and any realistic concurrent activity you can reproduce.
• Observe application and inference behavior while requests run.
• Define acceptance criteria in advance, including task completion, user experience, data handling and recoverability.
• Record the configuration, assumptions and results so you can compare them after changes.

Document the decision and revisit it when requirements change

Record the selected topology, the requirements it is intended to meet, and the assumptions behind the decision. Include component owners, data paths, relevant failure behavior and pilot findings. Revisit the decision if workload, maintenance arrangements, access requirements or recovery expectations change; assess the new requirement rather than assuming that a larger design is automatically better.

Airbip provides managed deployment of catalog applications as Docker workloads on Airbip cloud servers, with automated routing and TLS certificates through Traefik and Let’s Encrypt, DNS checks, service lifecycle management, and configurable daily, weekly and monthly backups. These infrastructure capabilities do not determine where a particular inference service should run or remove customer responsibility for data, access and governance choices. Confirm that the desired application configuration and inference arrangement fit your requirements before selecting a deployment model.

• Selected layout and the requirement it is intended to address.
• Assumptions about workload, concurrency, data location and external dependencies.
• Data that crosses service boundaries, retention choices and access owners.
• Responsibilities, relevant failure behavior, recovery procedure and pilot findings.
• Conditions that would prompt you to review the decision.