Configuration Drift in Self-Hosted Applications: How to Find It Before It Causes an Outage

Configuration drift is a recovery problem, not just a deployment problem

Configuration drift in self-hosted applications is the difference between the intended setup and the configuration that is actually in use. It may begin with a sensible short-term decision: a DNS record is changed during an incident, an administrator adjusts a setting in a dashboard, a variable is supplied from a local shell, or a third-party credential is added to make an integration work.

The application can continue to work after each of these changes. The operational risk appears later, when you need to reproduce the service, transfer responsibility, migrate it, apply a major change or restore it after a failure. A database backup may be valid while the recovered application still lacks its domain routing, mail configuration, scheduled tasks, identity settings or access to an external service.

Treat the baseline as an agreed description of a working service, not merely a repository of deployment files. NIST describes a baseline configuration as including components, settings or parameters, network topology and the logical placement of components. For a small team, the aim is not bureaucracy. It is to ensure that another authorized person can understand what the service depends on and recover it safely.

A useful test is simple: if the current operator were unavailable, could the responsible team rebuild or restore the application without relying on memory, old chat messages or a search through dashboards? If not, there is drift to investigate.

• A working URL does not prove that the service is recoverable.
• A successful backup does not prove that all recovery dependencies are known.
• The most important drift is usually the change nobody remembers making.
• Documenting intended state makes future changes safer, even when the live state is currently healthy.

Map every configuration layer before you try to control it

Start by mapping the application as a service chain. This prevents a common mistake: documenting the container or application settings while overlooking the components that make it reachable, secure or useful in day-to-day operation.

The exact stack will vary, but the same review areas recur across content systems, CRM platforms, analytics tools, project-management software and AI applications. Record what exists, where it is controlled and who can change it. A simple table with a layer, source of truth, owner, last-review date and recovery note is enough to begin.

Reverse proxies deserve separate attention. In Traefik, startup or installation configuration includes items such as entry points and provider connections, while routing configuration controls how requests are handled. Routing configuration, including HTTPS certificates, can be changed without restarting the proxy. That flexibility is useful, but it means a running service may not match an older deployment note.

• Domain and DNS: registered domain, DNS provider, authoritative zone, records, record owner and access path.
• Proxy and TLS: entry points, routes, certificate settings, renewal or validation method and redirect behavior.
• Container deployment: image references, Compose files, volumes, networks, restart behavior and rendered environment configuration.
• Application administration: administrator accounts, authentication choices, roles, application URL settings, mail settings and in-app configuration.
• Data services: databases, object storage where applicable, persistent volumes, connection references and backup scope.
• Scheduled jobs: cron entries, workers, queues, retention tasks, reports, synchronization jobs and the machine or service that runs them.
• External integrations: email delivery, identity providers, payment services, webhooks, AI-model services, analytics, DNS APIs and other third-party dependencies.

Find the changes most likely to become undocumented

Drift is rarely caused by a formal change that was reviewed and recorded. It usually enters through an exception, a convenience step or an urgent repair. Make these paths explicit during review rather than assuming that the deployment repository is complete.

Emergency fixes are particularly important. A workaround made during an outage may be correct at the time, yet remain after the original cause is forgotten. Manual edits in cloud, DNS, proxy or application dashboards have the same problem: they can be effective immediately but invisible to a deployment file.

For Docker Compose deployments, an unchanged Compose file is not necessarily proof of unchanged behavior. Compose interpolation may obtain values from the shell environment, a project .env file or an explicitly selected environment file. Capture which inputs were used to render the live configuration. The docker compose config command can provide a fully resolved view of merged Compose files and substituted variables, making it useful evidence for comparison.

One-off integration credentials also deserve a deliberate review. The important question is not the secret value; it is whether the service depends on it, who owns the account, what permissions it has and how the team would rotate or replace it.

• Changes made under incident pressure.
• Dashboard edits made directly in DNS, proxy, hosting or application administration interfaces.
• Environment variables copied between servers, terminals, notes or private files.
• Temporary routes, redirects, allowlists or firewall exceptions that became permanent.
• Administrator accounts created for a contractor, former employee or short-lived project.
• Webhook endpoints and API credentials created for a single integration test.
• Schedules added outside the normal deployment definition.

Create a minimum viable configuration baseline

A good baseline is concise enough to maintain and detailed enough to support recovery. Store it in an access-controlled location that the appropriate operators can find. Use links to authoritative locations rather than copying large configurations into several documents; duplicated records create their own drift.

For each application, write a one-page service record first. Then link it to the relevant deployment files, DNS zone records, proxy configuration, backup settings and operating procedures. Give every item an owner, even if the owner is a small internal team rather than an individual.

Baseline updates should be routine. NIST recommends maintaining a current baseline under configuration control and reviewing or updating it on a defined schedule, when circumstances require it and after components are installed or upgraded. Quarterly review is a practical starting point for many small teams, with an additional review before migrations and material changes.

• Service identity: application name, purpose, business owner, technical owner, criticality and user groups.
• Location and entry point: production URL, custom-domain or subdomain choice, DNS provider and the account or team responsible for DNS.
• Architecture: server or hosting location, container workload definition, persistent data locations, network dependencies and supporting services.
• Deployment evidence: repository or controlled file location, deployment command or process, image references and rendered configuration evidence where appropriate.
• Application settings: admin access process, authentication and role model, mail configuration reference, base URL and key operational preferences.
• Data protection: backup frequency and retention configuration, backup owner, restore destination and date of the most recent restoration exercise.
• Automation: scheduled jobs, worker processes, task purpose, schedule, execution location and failure-checking method.
• Integrations: provider, purpose, account owner, callback or webhook location, permission scope and rotation or recovery procedure.

Separate configuration from secrets

Configuration records must help an authorized operator locate and manage secrets without turning ordinary documentation into a secret store. Do not paste passwords, API keys, private keys or recovery codes into tickets, wikis, repositories or baseline spreadsheets unless those systems are expressly designed and controlled for that purpose.

Docker warns that passwords and API keys supplied in environment variables can be unintentionally exposed to processes or logs. Docker Compose also supports granting secrets to individual services as files. The implementation choice depends on your environment, but the core discipline is the same: minimize exposure, grant only the access required and make secret dependencies recoverable.

A secret inventory should contain metadata and procedures, not values. It should tell the team where the secret is managed, who owns the related third-party account, which service consumes it, what permissions it needs and how rotation is performed without breaking the application.

DNS validation credentials warrant particular care. Let’s Encrypt DNS-01 validation uses a TXT record under _acme-challenge. If automated DNS changes are required, access design matters: broad DNS API credentials on a web server increase the consequences of a server compromise. Use the narrowest practical authority and record the recovery path.

• Record the secret reference or secure-storage location, not the secret value.
• Name the consuming application, container or service account.
• Assign a business owner and a technical rotation owner.
• Document required permissions and any domain, project or account scope.
• State the rotation sequence, validation check and rollback option.
• Record expiry, renewal or review dates when relevant.
• Remove access and rotate credentials when ownership changes.

Compare intended state with actual state

A drift review is a structured comparison, not a hunt for perfection. Start with the baseline and ask whether the live service still matches it. Mark each difference as intentional and recorded, intentional but missing from the baseline, temporary, unknown or obsolete. Unknown differences should be investigated before they become part of the next migration or recovery.

Use evidence from more than one layer. A deployment definition shows what you intend to run; a rendered Compose configuration helps show what Compose will apply; live application settings show administrator-level choices; DNS records and proxy routes show how users actually reach the service. Job schedules and restoration results reveal dependencies that are otherwise easy to miss.

For Traefik-based services, compare both static and dynamic concerns. Certificate resolvers are configured statically, and a router or entry point must explicitly reference a resolver for it to be used. A review that checks only whether a certificate currently works can miss the configuration relationship needed to reproduce it later.

• Deployment: compare version-controlled definitions with docker compose config output and the running workload configuration.
• DNS: confirm production records, redirects, validation records and the people or accounts able to update the authoritative zone.
• TLS and routing: verify entry points, routers, service targets, certificate resolver references and expected HTTP-to-HTTPS behavior.
• Application administration: review privileged accounts, authentication choices, role assignments, application URLs and important email or integration settings.
• Data: verify which data stores and volumes are in scope for backup and restoration.
• Schedules: list all recurring jobs and identify whether they run successfully after restoration.
• Integrations: verify current endpoints, callback URLs, owners, scopes and the continued availability of the underlying accounts.
• Recovery evidence: compare the documented recovery path with what was actually required in the latest test.

Use a small-team change process that people will follow

A change process works only when it is proportionate. A small team does not need a heavyweight approval board for every adjustment, but it does need a durable record for changes that affect availability, security, recovery or ownership.

NIST configuration-change guidance covers documenting decisions, implementing approved changes, retaining records and monitoring or reviewing the process. In practice, a lightweight change record can fit into a ticket, issue or controlled operations log. The record should connect the reason for the change to the implementation evidence and the result.

Emergency changes are not an exception to documentation; they are a reason to document after service is stabilized. Make the follow-up review a defined step, with a deadline to update the baseline, remove temporary access or convert a manual adjustment into a repeatable configuration.

• Request: what is changing, why, which service layers are affected and who owns the decision?
• Review: is there a security, DNS, data, integration or recovery consequence? Is a second reviewer needed?
• Record: link the approved change, relevant files or dashboard evidence, owner and planned implementation window.
• Test: define the success check, including user-facing behavior and any necessary background job or integration check.
• Rollback: identify the prior state, who can restore it and what condition triggers reversal.
• Close: update the baseline, note the result and remove temporary access, routes or credentials.
• Review emergency work: record the final state and confirm that the workaround is still necessary.

Use recovery exercises to expose hidden dependencies

Backups are essential, but a successful backup run is not proof that recovery will work. NIST calls for contingency-plan testing, review of test results and corrective action where necessary. The most valuable recovery test is one that demonstrates whether the service can return to a usable state, not merely whether a database archive can be created.

Design exercises around what matters to the organization. NIST recovery guidance emphasizes prioritizing resources and using realistic scenarios. For one application, the meaningful test may be restoration to a clean environment with the production domain deliberately left disconnected. For another, it may be proving that a critical user can sign in, view recent data, send a message or complete a workflow after restoration.

During the exercise, record every item that was needed but absent from the runbook: a DNS permission, an application encryption setting, a mail relay reference, an integration callback change, a scheduled worker or an administrator account. Those discoveries are configuration-drift findings. Correct the baseline and repeat the relevant validation.

• Restore data into an appropriate test environment or other controlled destination.
• Recreate the application and its supporting services from the documented configuration.
• Confirm that the expected URL, proxy routing and certificate approach can be reproduced.
• Test privileged access through the documented access process, not a remembered personal shortcut.
• Verify critical application workflows and selected external integrations.
• Check scheduled jobs and workers where they are required for normal operation.
• Measure and record the actual steps, gaps and corrective actions; do not treat a failed test as wasted effort.