Deployment Troubleshooting

This guide helps resolve common issues when deploying or running TurboPanel in production. For security-related issues, see Security. For setup verification, see Control plane and Daemon setup.

Introduction

Production issues typically fall into TLS/Caddy misconfiguration, instance socket permissions, daemon WebSocket connectivity, Postgres socket access, or Ansible playbook failures. Self-hosted browser entrypoint is HTTPS on port 8443 by default (Caddy). lets_encrypt serves :443 (plus :80 for HTTP-01). On co-located dev hosts only, Caddy also serves a plaintext mirror at http://<host>:8880 when TURBOPANEL_DEV_HTTP_CONTROL_PLANE=1.

Port conflicts (8443 / 8880 / 80 / 443)

Problem: Caddy cannot bind :8443 or (dev) :8880. In lets_encrypt, it cannot bind :80/:443.

Solutions:

1. Change CADDY_PORT / CADDY_HTTP_PORT in instance/Caddy env and update firewall rules
2. Check for existing listeners: ss -tlnp | grep -E '8443|8880|80|443'
3. Stop conflicting services or remap the published port in your reverse proxy

UI or API not loading

Problem: Browser shows errors at https://<host>:8443 by default, https://<hostname>/ in lets_encrypt, or (co-located dev) http://<host>:8880.

Solutions:

1. Check instance: systemctl status turbopanel-instance
2. Check Caddy: systemctl status turbopanel-caddy
3. Test health over HTTPS: curl -k https://localhost:8443/api/health (omit -k when TURBOPANEL_TLS_PUBLIC is set)
4. Co-located dev only: test plaintext mirror: curl http://localhost:8880/api/health (requires TURBOPANEL_DEV_HTTP_CONTROL_PLANE=1; 403 when the flag is off)
5. Logs: journalctl -u turbopanel-instance -u turbopanel-caddy -f
6. Static UI: ensure ui/dist exists if TURBOPANEL_UI_MODE=static
7. Dev UI: ensure turbopanel-ui is running when TURBOPANEL_UI_MODE=dev

Daemon shows offline but service is running

Problem: UI badge is Offline (or Update fails) while systemctl status turbopaneld is active and the Cell panel shows recent activity.

Cause: The server list uses a Postgres presence projection that can drift from live WebSocket state after a failed upgrade or reconnect.

Solution: Re-run the release installer — see Refresh a stuck daemon.

Daemon not connecting

Problem: Remote server daemon fails to register.

Solutions:

1. Verify instance URL: TURBOPANEL_INSTANCE_URL=https://<host>:8443 by default, or https://<hostname>/ in lets_encrypt. On co-located dev only, http://<host>:8880 when TURBOPANEL_DEV_HTTP_CONTROL_PLANE=1 (no CA required)
2. Test from node over HTTPS: curl -k https://<host>:8443/api/daemon/v1/version (omit -k when the origin is publicly trusted)
3. Co-located dev only: test plaintext: curl http://<host>:8880/api/daemon/v1/version
4. Confirm platform CA (HTTPS path, default mode): compare /etc/turbopanel/instance-ca.pem with GET /api/daemon/v1/instance/ca (bundle, current CA first). A 404 means public TLS — use the system trust store and do not install instance-ca.pem.
5. Firewall: allow 8443 by default from remote daemons to instance (443 in lets_encrypt; 8880 is dev-only and not for remote production nodes). The installer removes ufw, firewalld and iptables-persistent from every host it installs on — TurboPanel owns the host firewall — so a distribution firewall cannot be what blocks 8443 after a completed install. If curl https://127.0.0.1:8443/api/health answers locally but not remotely, look upstream: a cloud security group, a provider firewall, or a NAT in front of the host.
6. Logs: journalctl -u turbopaneld -f

Stale platform CA (tls-trust parked)

Problem: Daemon log shows tls-trust parked (dialed host, CA path, fingerprint) and reconnects only every 5 minutes to 1 hour — not a silent 30 s loop.

Cause: The host still trusts an old platform CA after the control plane rotated (or the leaf SAN no longer matches the dialed hostname). Control-plane identity (JWKS JWT) is unchanged; only the transport anchor is stale.

Solutions:

1. Prefer the overlap path: on the control plane, keep the old CA in ca-bundle.pem and enqueue server.tls.trust.reconcile while the existing WSS session is still valid.
2. If the session is already dead, re-run the installer with --instance-ca (or --insecure-tls for bootstrap fetch). run.sh pins --cacert first; on HTTP 000 it retries once unpinned and installs only if the fetched PEM validates the live leaf. Old and new fingerprints are printed when $CA_PATH is replaced.
3. If verification fails, keep the existing CA and fix the instance URL / SAN list rather than blindly replacing the file.

Postgres connection failures (self-hosted)

Problem: Instance cannot connect to Postgres.

Solutions:

1. Check Postgres container: docker ps | grep postgres
2. Verify Unix socket under /var/run/turbopanel/postgres/
3. Confirm TURBOPANEL_DATABASE_URL in turbopanel-instance.service — a full Postgres URL, typically injected by the instance-launch role
4. Socket directory permissions: /run/turbopanel should be 2770 tp:tp

Install wizard / PAM failures (Deno)

Problem: Install bootstrap rejects host credentials.

Solutions:

1. Install pamtester on the host
2. Confirm tpctrl user sudoers allows pamtester login * authenticate
3. Use a host account in sudo / wheel / admin, or root

UI Update stuck on "update already in progress"

Problem: After clicking Update, Ansible runs but the daemon keeps the same PID for a long time. A second update attempt returns update already in progress.

Cause: The UI update path uses run.sh --no-start (does not stop the daemon during reconcile). The daemon then tries to restart itself with plain systemctl enable --now as the tp user — that fails silently (non-root cannot restart systemd units; enable --now also skips restart when already active). An in-memory lock then blocks further updates.

Check logs on the server:

sudo tail -30 /var/log/turbopanel/daemon.err.log
sudo grep -i 'systemctl\|sudo\|update' /var/log/turbopanel/daemon.log | tail -20

Immediate recovery on the server:

sudo systemctl restart turbopaneld

Or reconcile by re-running run.sh — see Refresh a stuck daemon.

Ansible / upgrade failures

Problem: daemon orchestration fails (or, on a contributor dev host only, the console's Upgrade System action — managed installs have no such button; they re-run the installer).

Solutions:

1. Check daemon logs — on managed hosts Ansible runs as the tp user; on co-located dev it runs as the current dev user
2. Dirty git checkouts block upgrade; commit or stash changes in $HOME source repos (~/turbopaneld, ~/turbopanel, ~/ui, ~/website)

What to point a monitor at

/api/health is not an outage signal. It is a static identity payload — licence, version, the commit the build came from — and it never touches the database. It answers 200 with Postgres stopped, which is exactly when an operator most wants it to say something. A game day on a dev stack confirmed it: the control-plane database was killed and /api/health returned 200 throughout.

Monitor /api/daemon/v1/readiness instead. It reads the database, so it is the endpoint that can fail, and its three answers are distinguishable:

Alert on the third. The second is a normal state for a freshly installed instance, and co-located daemons already poll this endpoint and wait it out.

Getting paged when a server goes dark

The control plane watches its fleet on a sweep and marks a daemon offline when it stops answering. Without somewhere to send that, it is a log line nobody reads at 3am. Point it at an incoming webhook:

curl -X PUT https://<host>:8443/api/admin/v1/settings/alert-webhook \
  -H 'content-type: application/json' \
  --cookie "$SESSION" \
  -d '{"url":"https://hooks.slack.com/services/T000/B000/XXXX"}'

{"url": null} clears it. Two alerts go out:

The body carries a text field, which Slack, Mattermost, Rocket.Chat and Discord all render, plus kind and detail for anything that parses JSON.

Two things to know about the URL:

• It is a credential. In every common webhook scheme the path is the secret. It is stored sealed, the same way an SMTP password is, and reading the setting back returns only the origin and whether one is configured — never the path. If you lose it, set a new one; there is no way to read the old one out.
• It has to be https, with no credentials in the URL. That is the whole rule — a receiver on your LAN (an Alertmanager next to the control plane) is fine. The forge URLs keep a stricter, public-only gate because those fetches carry the App's credentials; this one carries nothing and its response goes nowhere.

A delivery that fails is logged and dropped. It never delays or fails the sweep — an unreachable Slack must not be the reason a dead host stays marked online.

What recovers on its own

One trap worth knowing before you test any of this by hand: docker kill <name> is not a crash simulation. Docker records it as a manual stop, and a restart policy does not fire for a manual stop — the container stays down until you run docker start. To simulate a real crash, signal the container's process directly:

sudo kill -9 "$(docker inspect -f '{{.State.Pid}}' turbopanel-database)"

(docker exec <name> kill -9 1 does nothing at all — PID 1 of a PID namespace ignores SIGKILL sent from inside that namespace.)

Contributors can run the whole exercise, with these assertions checked for them, from the dev checkout:

./scripts/game-day.sh --all --include-operator-stop

It refuses to run outside a development guest, bounds every wait, and exits non-zero if the stack does not come back.

General debugging

Related documentation

• Security — TLS and authentication
• Control plane — Service layout
• Daemon setup — Node installer
• Dev console troubleshooting — Local dev issues