What Happens When a Dependency Fails? A Resilience Test for Self-Hosted Applications

An application can be healthy while its work is impossible

A running container, a successful login-page request, or a green reverse-proxy check answers only a narrow question: is that component responding under the condition being checked? It does not prove that users can sign in through the organisation’s identity provider, send a notification, retrieve a file, complete a payment, call a connected API, or obtain a response from a model service.

Docker Compose can start services in dependency order, but startup order alone does not mean a dependency is ready for the application’s work. Docker documents using a configured health check and the `service_healthy` condition where a dependent service must wait for readiness ([Docker documentation](https://docs.docker.com/compose/how-tos/startup-order/)). Even then, health checks need precise interpretation: Traefik’s documented `/ping` endpoint is evidence that the Traefik process is live, not proof that every upstream application path is usable ([Traefik documentation](https://doc.traefik.io/traefik/reference/install-configuration/observability/healthcheck/)).

Treat resilience as an outcome-based question: which user tasks must still succeed, which may be delayed, and which must be blocked safely when a dependency is impaired? This produces a far more useful plan than a list of integrations.

• Liveness: a process answers a simple request.
• Readiness: a service can accept the specific type of work it is about to receive.
• Business completion: the intended outcome occurred once, with the correct data and any required audit trail.
• Recovery: the team can restore normal operation, reconcile uncertain work, and explain the impact.

Map dependencies by the function they perform

Start with the user journeys, not the configuration file. For each important journey—such as signing in, creating a customer record, publishing content, sending an invoice, accepting a form submission, or answering an AI query—trace every service required from user action to confirmed outcome.

Include managed, external, and human-operated dependencies. A Docker deployment diagram is useful but incomplete: DNS, a custom-domain registrar or provider, identity, SMTP or email APIs, SaaS APIs, webhook senders, model endpoints, and monitoring may all sit outside the application stack. Docker Compose also supports externally delegated provider services, reinforcing that a workload can depend on resources whose lifecycle is elsewhere ([Docker Compose reference](https://docs.docker.com/reference/compose-file/services/)).

Use the following functional groups so gaps are visible early.

• Identity and access: identity provider, federation, multi-factor authentication, role directory, administrator access path.
• Network and domain: DNS, custom-domain configuration, reverse proxy, TLS certificate validation and renewal path.
• Communication: SMTP relay, transactional email API, inbound email handling, messaging or notification services.
• Data and files: primary database, object or file storage, backup destination, import or export locations.
• Commercial and connected services: payment provider, CRM, accounting, analytics, maps, search, partner APIs and API credentials.
• Asynchronous processing: queues, schedulers, workers, webhooks and callback endpoints.
• AI services: model endpoint, embedding service, vector store, document retrieval pipeline and model credentials.
• Operations: logs, monitoring, alert delivery, incident contacts, secrets management and administrative documentation.

Classify dependencies by the consequence of losing them

Do not assign the same resilience target to every dependency. NIST supply-chain guidance supports varying requirements according to criticality, considering mission or business impact, the data being processed, and the product or service supplied ([NIST SP 1305](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1305.pdf)). Apply the same reasoning to each dependency in your application architecture.

A simple four-part classification forces a decision about acceptable degraded operation. A service can appear in more than one category for different workflows; for example, email may be optional for viewing records but required for password recovery or legally important delivery.

• Required to sign in: without it, ordinary users cannot establish a session. Define a secure administrator emergency-access route separately.
• Required to work: without it, a core transaction cannot be completed correctly. Examples may include the primary data store or a mandatory external API.
• Required for optional features: users can continue their primary work, but a nonessential action is unavailable or delayed.
• Required for recovery: daily operation may continue temporarily, but restoration, certificate renewal, incident notification, or data recovery depends on it.
• For every classification, set an acceptable interruption window, a maximum tolerable backlog, the permitted fallback, and the person authorised to invoke that fallback.

Ask for evidence, not reassuring descriptions

Vendor documentation and a test environment should answer operational questions that marketing pages often do not. Capture a link, test result, configuration reference, or support contact for each answer. When documentation is silent, mark the behaviour as unknown rather than assuming a retry, queue, or backup exists.

Check the precise boundary being tested. Traefik notes that, when health checks are enabled and all servers in a service are unhealthy, it can return HTTP 503. That is useful evidence about upstream availability at the reverse-proxy boundary, but it is not evidence that a user transaction completed ([Traefik FAQ](https://doc.traefik.io/traefik/getting-started/faq/)).

Certificate automation deserves its own dependency entry. Let’s Encrypt HTTP-01 validation requires the certificate authority to retrieve a challenge file from the web server, while DNS-01 requires it to query a TXT record in DNS. The relevant route, DNS control, credential scope, and propagation behaviour should therefore be documented before renewal becomes urgent ([Let’s Encrypt challenge documentation](https://letsencrypt.org/docs/challenge-types/)).

• What exact operation proves the dependency is usable, rather than merely reachable?
• What timeout, error code, retry policy, retry limit, and backoff behaviour apply?
• Can a request be accepted but completed later, and where can its final status be verified?
• Can events arrive late, more than once, or out of order?
• Does the application queue work during an outage? What is the queue limit, visibility, and reconciliation method?
• Which credentials are needed for recovery, where are they stored, and who can use them?
• Which DNS or network changes can prevent certificate renewal or custom-domain access?
• Is there a non-production environment or provider staging environment for safe validation?

Run controlled failure tests that resemble production incidents

A resilience test should have a defined scope, a safety condition, a rollback action, and an observer. Begin in a representative non-production environment. Move to limited production exercises only when impact, authorisation, monitoring, and reversal are understood. Do not run an experiment that is expected to cause uncontrolled workload failure.

Test an actual business workflow, not only a connection check. For example, if email delivery is unavailable, test account invitation, password reset, and the action that depends on the recipient receiving the message. Record separately whether the application accepted the request, whether the email provider accepted it, and whether the intended recipient can complete the required action. If a model endpoint is unavailable, test what the user sees, whether source documents remain protected, and whether an incomplete interaction is recorded accurately.

For certificate troubleshooting or client development, use the Let’s Encrypt staging environment rather than repeatedly causing failed production authorisations. Let’s Encrypt recommends staging for diagnosing authorisation conditions without consuming production limits ([Let’s Encrypt rate-limit documentation](https://letsencrypt.org/docs/rate-limits/)).

• Unavailable identity provider: attempt ordinary user sign-in, session renewal, administrator access, and logout; verify emergency access only under its documented controls.
• Failed email delivery: initiate messages, inspect application status and delivery evidence, verify the user-facing message, then restore delivery and reconcile delayed or failed actions.
• Unreachable API: interrupt the route or deny the credential in a test environment; check timeouts, error presentation, saved state, and whether the action can be safely repeated.
• Delayed webhook: delay delivery, then release it; test redelivery, duplicate events, and out-of-order events where the provider or application documentation identifies those behaviours.
• Exhausted rate limit: simulate quota errors where possible; verify bounded retries, backoff and a clear stop condition.
• Unavailable model endpoint: test a timeout and an error response; determine whether the application offers a clearly labelled unavailable state, defers work, or must block the workflow.
• Unhealthy upstream: make all servers behind a health-checked reverse-proxy service unavailable in a test setting; verify expected 503 handling, alerting, and recovery.

Record the outcomes that determine whether a failure is safe

A test is incomplete when the only result is “it retried”. A retry can follow an operation whose outcome is unknown. A successful response from an application or external API may show that one processing boundary was reached; it does not, by itself, prove every downstream business outcome. Define the completion evidence required for each workflow.

Record what users experience and what administrators must do, but also determine whether data is correct. For every delayed or retried operation, establish from the relevant application and provider documentation whether duplicate delivery, reordering, partial completion, or uncertain outcomes are possible. Where they are possible, use a duplicate-safe design and maintain a reconciliation plan.

Where an API documents idempotency support, use its documented mechanism for retryable create or update requests. Do not assume such a mechanism exists for an API that does not document it.

• User impact: exact task affected, error message, visible status, and whether users can continue other work.
• Data-integrity risk: could the action be lost, duplicated, partially saved, processed out of order, or left in an unknown state?
• Retry behaviour: trigger, timeout, schedule, backoff, maximum attempts, and final failure state.
• Queue behaviour: where work is held, backlog limit, duplicate handling, ordering assumptions, expiry, and replay method.
• Observability: logs, metrics, alerts, correlation IDs, and the evidence used to decide that processing completed.
• Administrator actions: immediate containment, verification steps, reconciliation procedure, escalation contact, and the rule for declaring recovery complete.

Design fallbacks that are practical and secure

A good fallback preserves security and makes the limits explicit. It is not simply a way to force the system through. Define what can continue, what must pause, who may approve a manual process, where the record is kept, and how deferred work will be reconciled afterwards.

Emergency administrative access is a key example. Microsoft recommends designed emergency accounts for federated identity-provider outage scenarios, using cloud-only accounts that do not depend on federation or on-premises synchronisation. The same guidance emphasises strong authentication that differs from normal administrative accounts, secure credential storage, logging and monitoring, and regular validation drills ([Microsoft emergency-access guidance](https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/security-emergency-access)).

For customer-facing work, a manual procedure may be safer than automated retries when the operation can create financial, legal, or duplicate-record consequences. A deferred-work option is appropriate only when the queue, final status, and reconciliation process are understood.

• Local emergency access: tightly controlled, separately authenticated administrator access for an identity outage; use only according to a documented and tested procedure.
• Manual process: a time-bounded alternative such as recording requests in an approved system for later entry, with an owner and reconciliation check.
• Deferred work: queue eligible work until a dependency returns, while tracking backlog, expiry, duplicate processing and user status.
• Feature isolation: disable only the affected optional function while preserving safe core workflows.
• Clear communications: state what is unavailable, what users should do instead, whether work has been saved, and when the next update will be provided.

Avoid workarounds that turn an outage into a security or data incident

Never make “disable authentication” the default response to an identity outage. That removes the control exactly when administrators have reduced visibility and increased pressure. Instead, prepare a limited emergency-access design with distinct credentials, restricted use, monitoring, and scheduled tests.

Likewise, do not treat retries as proof that nothing was lost or duplicated. Rate limits can make repeated failures worse. Let’s Encrypt applies limits to failed authorisations and recommends staging while troubleshooting ([Let’s Encrypt rate-limit documentation](https://letsencrypt.org/docs/rate-limits/)). Configure bounded retries and a stop condition, then investigate or reconcile work whose outcome remains uncertain.

Avoid unscoped DNS credentials as a convenience for certificate automation. Let’s Encrypt cautions that placing broad DNS API credentials on a web server increases the impact of compromise, and suggests narrowly scoped credentials or validation from a separate server where feasible ([Let’s Encrypt challenge documentation](https://letsencrypt.org/docs/challenge-types/)).

• Do not bypass access controls without a pre-authorised, logged emergency procedure.
• Do not replay unknown transactions until you can determine whether the original completed.
• Do not assume webhook delivery is once-only or ordered without confirming the documented behaviour.
• Do not allow an unbounded retry loop to create load, cost, or provider lockout.
• Do not grant broad DNS permissions merely to automate a single validation task.
• Do not close an incident based solely on process liveness; verify the affected business outcome and reconcile the backlog.