How I Moved My Production Services to a New Server

I knew the migration was finished when I deleted the old VPS and nothing interesting happened.

I checked everything again. The public health endpoints answered, the production workloads were healthy, my IRC bouncer completed a TLS connection, backups had recent verified restores, and monitoring had no open incident. The sites carried on serving traffic from the new host. Nothing tried to call home to the machine I had just removed.

It took several long days to make that moment boring.

The old VPS had served me well for years. Services arrived at different times and reflected the tools and habits I used when I built them. The move gave me a chance to give them one operating model: exact releases, known data ownership, tested recovery, and monitoring that does not disappear with the production host.

I built the replacement beside the live machine. I did not clone the server and tidy it afterward. Each service was staged, checked, moved, and watched on its own.

I started with an inventory

Before installing anything, I wrote down what was real.

For each service I recorded its authoritative data, dependencies, start procedure, health signal, backup method, restore procedure, and public route. The list covered this stateless portfolio, Pushify, temporary file storage, a news service, public transport routing, databases, IRC tools, and years of ZNC configuration.

Some applications were one process and one database. Others had data, caches, generated artifacts, and background work that needed to agree before I could call them healthy. Treating all of that as “copy the Docker volumes” would have hidden the difficult parts.

The old VPS stayed online while I prepared the new host. I could inspect a working service, stage its replacement, and test the new copy before changing public traffic. Until the final cleanup, I still had a known system to compare against.

That overlap took time, but I never had the whole platform in flight at once.

A baseline I can reproduce

I wanted capacity and architecture to remain separate decisions. If the current host needs more CPU, memory, or storage, I can resize it without redesigning the platform. If demand or availability eventually calls for several hosts, the runbooks already separate application processes, authoritative data, ingress, and message delivery. Those pieces can move to shared services and replicated application instances when there is a reason to do it.

The host runs an Ubuntu LTS release with automatic security updates. Reboots are reported and scheduled instead of happening unexpectedly. SSH uses keys, direct root login is disabled, and access is restricted at both the provider and host layers.

Most applications run in containers. Their networks are private, logs are bounded, and health checks have an actual purpose in deployment and monitoring. Containers use non-root users and read-only filesystems where the application permits it. I drop Linux capabilities instead of accepting the default set and hoping none of them matter.

Public web traffic reaches an outbound tunnel. A private reverse proxy then sends the request to the correct application. The host does not need public HTTP or HTTPS ports. Backups and private health checks use an encrypted path to a separate system.

The tunnel, proxy, and application networks have different jobs. The backup path is separate from normal web traffic as well. I did not want one routing problem to explain every failure at once.

Moving one writer at a time

I moved Pushify early. Its first working version was already finished, and the rest of the migration immediately gave it real work. Deployments reported progress to my phone. Monitoring could open and recover incidents. I could test the phone actions against the production path instead of a pretend example.

The portfolio followed because it has no production data. It became the template for stateless releases: build an exact Git revision into an immutable image, run it without root privileges, keep its filesystem read-only, and retain the previous image for rollback. There is no mutable production checkout waiting to drift away from Git.

Stateful services needed a different rule. During the final copy, only one writer could be active. The old writer stopped first. I validated the source, copied the authoritative data into staging, compared the result, and started the new writer only after those checks passed.

set -Eeuo pipefail

stop_old_writer
validate_source_data
copy_authoritative_data "$staging_dir"
compare_counts_and_hashes "$staging_dir"
start_new_writer
verify_public_health

The commands differed for SQLite, PostgreSQL, Redis, files, and application configuration. The ownership rule did not. Once the old writer stopped, it no longer received writes. The new copy did not become authoritative until its data and application checks passed.

One storage migration kept the existing object keys while moving the objects to a filesystem. The migration used SQLite’s online backup mechanism for the metadata, rejected unsafe keys, copied the objects, verified every database reference, and wrote a hash manifest for the result. Preserving the keys meant the database did not need a mass rewrite just because the storage implementation changed.

The public transport planner had a different problem. Its source timetable, database tables, routing graph, cache, and application had to describe the same release. The deployment stages that set together, starts candidates against it, and runs a real journey query before switching production. An unchanged source checksum is a no-op. Rollback restores the previous matched set instead of pairing old timetable data with a new routing graph.

Those two services share almost no application code. They still fit the same migration rule because each one has a clear definition of authoritative state.

I also resisted forcing every service into Docker. ZNC stayed with its packaged systemd service and AppArmor profile. A container would have made the inventory look tidier, but it would not have made ZNC easier to update, restore, or diagnose.

Rehearsing a stateful release

A health endpoint can tell me that a process started. It cannot tell me that a new build understands the current database or that a migration will behave properly.

For stateful applications, the deployment helper can make an isolated copy of the database and start the candidate against that copy first. Outbound networking is disabled. Schedulers, scrapers, and other background writers stay off. The candidate may run schema changes and application queries without sending messages or doing real work.

create_isolated_database_copy
start_candidate_without_network_or_scheduled_jobs
run_schema_and_application_checks
stop_candidate
replace_only_the_live_service

If the rehearsal fails, the live application has not changed. If it passes, the deployment still replaces only that service rather than restarting an unrelated stack around it.

This also helped with compatibility migrations. One application kept its raw source category and added a separate canonical, indexed value for API queries. Startup could add and backfill the new column without rewriting the original input. The rehearsal checked that the application still saw the expected records before production was touched.

Releases come from Git

The portfolio was the first full implementation of the new production release path.

Ordinary branch pushes do not deploy. A push to main does not deploy either. The requested revision first passes hosted verification with locked dependencies, the Astro build, Go tests, a server build, and tests for the deployment scripts.

Production requires a manual workflow or an annotated version tag. The deployment waits for verification and uses the protected production environment:

on:
  workflow_dispatch:
  push:
    tags: ["v*.*.*"]

permissions:
  contents: read

jobs:
  verify:
    uses: ./.github/workflows/verify.yml

  deploy:
    needs: verify
    environment: production
    steps:
      - name: Check out exact revision
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0
          persist-credentials: false

The workflow confirms that the revision belongs to the reviewed main history. It creates a source archive directly from Git and calculates its checksum:

set -Eeuo pipefail

git merge-base --is-ancestor "$GITHUB_SHA" refs/remotes/origin/main

archive="$RUNNER_TEMP/release-${GITHUB_SHA}.tar.gz"
git archive --format=tar.gz --output="$archive" "$GITHUB_SHA"
checksum=$(sha256sum "$archive" | cut -d' ' -f1)

The production runner has one purpose. It has no interactive login, Docker membership, or general sudo. Its only privileged operation is a reviewed deployment helper that accepts the revision, checksum, and archive on standard input.

That helper validates the archive before extracting it. Unsafe member names are rejected. It builds the exact image, checks its runtime user and revision label, takes a fresh backup where needed, and changes only the selected application.

Rollback is armed before the first production change:

rollback() {
  exit_code=$?
  trap - ERR

  if (( deployment_started )); then
    restore_previous_stack
    restore_previous_release_link
    recreate_only_this_service
  fi

  exit "$exit_code"
}

trap rollback ERR

An error during validation leaves the healthy service alone. Once deployment_started is set, a failed container or public acceptance check restores the previous stack and exact image.

The final check goes through the public route. Internal container health is useful when something fails, but a user reaches the service through DNS, TLS, the tunnel, and the reverse proxy. A deployment is not successful until that path works too.

Restoring backups on purpose

Every stateful backup job now proves that it can restore something useful. Uploading encrypted data off the server is necessary, but the existence of an archive says very little about its contents.

The jobs restore snapshots into disposable locations. SQLite files get integrity checks and an application-specific query. File stores get count or hash comparisons. Repository checks look for damage in the backup store itself. Retention runs only after a successful backup.

restore_dir=$(mktemp -d)
trap 'rm -rf "$restore_dir"' EXIT

restic restore latest --target "$restore_dir"
restored_db="$restore_dir/restored/service.db"

test -s "$restored_db"
test "$(sqlite3 "$restored_db" 'PRAGMA quick_check;')" = ok
sqlite3 "$restored_db" 'SELECT COUNT(*) FROM important_records;'

That last query changes by service. It may check a known record, a row count, or a relation between the database and stored files. PRAGMA quick_check tells me whether SQLite can read the file. It cannot tell me whether I restored the correct database.

Pushify’s restore test goes further. It restores devices, events, recipients, responses, deliveries, and deduplication records. A repeated dedupe key must still return the original event after restore. That is the behavior I care about during recovery.

The off-site monitoring system is backed up in the other direction. Production and monitoring sit in separate failure domains, and neither machine is the only copy of the other’s recovery data.

Watching a restore query return the expected data is much more comforting than watching a backup directory get larger.

Monitoring incidents instead of polling failures

The local monitor runs roughly once a minute. It checks container state and restarts, system services, resource pressure, disk space, inodes, public health, TLS expiry, backup freshness, required reboots, and unexpected data growth.

It reports state changes. An unchanged failure belongs to the incident already open, so I do not receive the same phone notification every minute.

previous=$(read_previous_state "$check_name")
current=$(run_check "$check_name")

if [[ $current != "$previous" ]]; then
  correlation_id="${check_name}-${incident_started_at}"
  publish_transition "$previous" "$current" "$correlation_id"
  write_state_atomically "$check_name" "$current"
fi

The full monitor adds severity, deduplication, recovery context, and correlation between checks. If a local container failure explains several public failures, I get one useful incident rather than a notification for every symptom.

Correlation has an escape hatch. Suppose the container recovers but the public route remains down. The next public check immediately opens its own incident. The earlier suppression cannot hide a second problem in DNS, the tunnel, or the proxy.

Planned work is narrow too. A backup job may temporarily suppress the matching service check, but only for that exact pairing. If the service remains unhealthy after the job finishes, the following monitor pass opens an incident.

One growth check exposed an interesting conflict. The monitor’s systemd sandbox could not traverse container-owned host directories. I could have weakened the service permissions. Instead, the check reads only approved paths through the container runtime with an argument array and read-only commands. The monitor kept its existing capability boundary, and the implementation changed to fit it.

The alert service can also be down

Pushify runs on the server it monitors, so the local monitor needs somewhere to put an event when Pushify is unavailable. Failed sends enter an ordered local outbox.

Each event is written to a temporary file, given restrictive permissions, and renamed into the queue:

queue_event() {
  event=$1
  temporary=$(mktemp "$outbox/.incoming.XXXXXXXX")

  printf '%s\n' "$event" > "$temporary"
  chmod 600 "$temporary"
  mv "$temporary" "$outbox/$(next_sequence).json"
}

The consumer removes a file only after Pushify accepts it. A crash leaves a complete queued event or no event at all.

Local monitoring still disappears if the whole VPS disappears. A separate system checks the public services and notification path. It can use email when Pushify is unreachable. Dead-man heartbeats notice when either monitoring location stops reporting.

Error pages have to survive the application

I wanted an application failure to produce something better than a generic edge error. The reverse proxy therefore owns a small set of self-contained upstream pages. Their HTML, CSS, and cat artwork live with the proxy. They use no JavaScript, analytics, remote fonts, or external image host.

The first outage drill found a wrinkle. For some upstream status codes, the edge replaced the proxy’s response with its own error page. The proxy now keeps the internal distinction for logging while returning a public status that the edge passes through unchanged.

I also left proxy error interception disabled for normal application responses. If an application intentionally returns JSON with a 404, 429, or 502, the proxy does not replace that body with a cat.

Then I stopped the stateless portfolio on purpose and checked the result like a browser would. The expected page arrived, its image loaded without the application container, the response had the intended cache and security headers, and monitoring opened one incident. When the container returned, the public route recovered and the same incident closed.

That drill tested a surprising amount of the platform with one harmless outage.

Useful phone actions, kept on a short leash

Selected alerts offer actions for the few operations I genuinely want on a phone. I can request a status summary, ask for disk information, read a short set of sanitised warnings, or acknowledge an incident.

A reboot-required alert may offer a controlled reboot. It needs confirmation and maps to one fixed helper. The helper refuses to proceed while backup or one-shot package work is active.

Acceptance testing caught a subtle case here. A package-management service can remain active as an idle shutdown helper even when no update is running. Waiting for that long-lived service would block the reboot forever. The guard now waits for the jobs that actually make a reboot unsafe. The same confirmed phone action can continue once those jobs finish without asking me to approve an unrelated command.

The phone never supplies a command. The operator fetches the original event and verifies its source, identity, expiry, action definition, and incident correlation. Arbitrary service names, paths, URLs, and values are rejected. A valid action ID maps to one allowlisted operation.

The result comes back as another Pushify event on the same incident. Routine lines are omitted, warnings are cleaned up, and the useful part fits on a phone screen. Android is not my terminal. It handles the small decisions that should not require one.

The strangest failure was not an application

One test left a failed systemd unit behind even though every production service was healthy. The culprit was a generated hotplug rule whose match was broader than intended. A short-lived container network interface happened to match it, then vanished before the hotplug unit finished.

The fix narrowed the rule to the actual provider interface identity. I checked the result with the system’s device tooling and a synthetic container-style identity. That removed the false failure without hiding genuine network events.

I kept this in the runbook because it is exactly the sort of problem that returns six months later looking completely unfamiliar.

Public cutover still found stale state

There was no master switch for the server. I staged each service, tested it privately, moved its traffic, and tested it again through the route its users take.

DNS made some cutovers quick and others slow. At one point, public resolvers had the current answer while a resolver used by the production host still held the retired one. Users reached the healthy application, but an internal monitor followed stale data and reported a TLS failure.

That was useful evidence. A migration can look complete from outside while an internal dependency still points backward.

I kept application rollback separate from data migration throughout the move. An immutable image can roll back quickly. Data rollback needs a decision about which writer is authoritative, especially after the new service has accepted writes. Combining those two problems into one magic rollback button would have made the risky moments harder to reason about.

Deleting the old VPS

Before deleting the old machine, I checked production DNS for references to it, reviewed its remaining connections, removed obsolete monitoring and private-network configuration, and created one final validated archive alongside the service backups and exact source releases.

Then I ran the acceptance pass: public health endpoints, production workloads, reviewed system services, the IRC TLS path, encrypted backup jobs, restore results, off-site monitoring, dead-man heartbeats, and the remaining private link.

Everything passed. I deleted the VPS and ran the checks again.

The sites loaded. Monitoring stayed quiet. Nothing depended on the deleted machine.

I can now point to the exact revision of an application in production, rebuild its image, restore its data somewhere empty, and see when one of those paths stops working. The old VPS did its job. I was still relieved when removing it turned out to be uneventful.