Saltar al contenido principal

Operations

Day-2 material for a deployment installed with the Production Installation guide.

Upgrading​

Pre-upgrade checklist​

Check your overlay for a pinned redis.websocketDb

global.required.redis.websocketDb must be 2, the chart default. If your overlay pins it — most do not — remove the pin or set it to 2. Redis pub/sub is scoped per database, so a publisher and subscriber on different indices simply stop seeing each other and live scan progress goes quiet with no error anywhere.

Back up data before every upgrade

Database schema migrations may run automatically during helm upgrade. If the upgrade fails mid-migration, the database may be in a state that the old chart version cannot read. A backup taken immediately before the upgrade is your only guaranteed rollback path for the data layer.

See the Backup and Restore guide for the full procedure.

# 1. Review release notes from the Plexicus team before running the upgrade.
#    Breaking changes or required values migrations are always listed there.

# 2. Take a backup immediately before upgrading.
#    (Full commands in the Backup and Restore guide.)

# 3. Pin the new version and upgrade:
helm upgrade plexicus \
  oci://europe-west3-docker.pkg.dev/plexicus-registry/charts/plexicus \
  --version <new-version> \
  --namespace plexicus \
  --values my-values.yaml

Post-upgrade verification​

After the upgrade, confirm the platform is healthy before considering the upgrade complete:

# All pods must be Running or Completed — no Pending or CrashLoopBackOff
kubectl get pods -n plexicus

# API health endpoint must return 200
curl -s -o /dev/null -w "%{http_code}" https://api.<your-domain>/health
# Expected: 200

# Smoke test: open https://<your-domain> in a browser, log in, and run a scan

The upgrade is successful when all pods reach Running, the /health endpoint returns 200, and the login flow completes without errors.

Rollback​

If the post-upgrade verification fails, roll back to the previous Helm release revision:

# List available revisions
helm history plexicus -n plexicus

# Roll back to the previous revision (or specify a revision number)
helm rollback plexicus -n plexicus
Rollback does not downgrade database schemas

helm rollback restores the previous Helm release state — it does not revert database schema migrations that may have run during the upgrade. If the schema was migrated forward, the rolled-back application pods may fail to connect to the database. In that case, restore the database from the backup you took before the upgrade rather than relying on helm rollback alone.

Troubleshooting​

Indexed by the symptom you actually see, because that is what you have when something goes wrong. Causes that are documented in depth in the Production Installation guide are linked rather than repeated.

Pods stuck in ContainerCreating, with a FailedMount event​

kubectl -n plexicus describe pod <pod> | grep -A5 Events
# MountVolume.SetUp failed for volume "plexicus-license" :
#   secret "plexicus-license" not found

The plexicus-license Secret does not exist, or is in the wrong namespace. Both fastapi and worker mount it with optional: false, so the container never starts and there are no application logs to read. The pods recover on their own once the Secret exists — no rollout needed. Full detail: Troubleshooting license failures.

ImagePullBackOff / ErrImagePull​

kubectl -n plexicus describe pod <pod> | grep -A3 Events

Usually one of:

  • the pull secret is missing from the plexicus namespace, or is named something other than the gar-secret the chart references (step 4);
  • global.imagePullSecrets does not list the name you actually created;
  • the credential has expired, or was created with the wrong --docker-server.

Confirm the Secret is the right type — this is the trap that looks like a credentials problem but is not:

kubectl -n plexicus get secret gar-secret -o jsonpath='{.type}'
# kubernetes.io/dockerconfigjson

kubectl create secret generic produces an Opaque Secret, which Kubernetes silently ignores for image pulls. Use kubectl create secret docker-registry (step 4).

worker CrashLoops with WORKER_CONTROL_PLANE_SECRET must be set​

Not a licence problem — see Troubleshooting license failures. The value must be identical in the plexicus-fastapi and plexicus-worker Secrets: the API presents it as X-Shared-Secret when it calls the worker's control plane, so a mismatch is as bad as an absence — the worker boots, then rejects every control-plane call.

EXPORTER_INTERNAL_TOKEN is not the same shape and will not produce this symptom: the worker never reads it at boot, so its absence does not stop the pod starting. It is read lazily by the daily EPSS-refresh schedule and sent to the exporter as X-Internal-Token. Missing or mismatched, that one daily activity gets a 401 — logged, but the workflow still completes, so nothing pages and EPSS data quietly goes stale. It is boot-blocking for the exporter pod. See the key catalog.

CrashLoopBackOff with a clean container start​

The Secret mounted correctly but the licence itself is invalid or expired. See Troubleshooting license failures.

The install fails before anything is created​

A values.schema.json rejection. Nothing was created, so there is nothing to clean up — fix the overlay and re-run. The error names the offending path; the ones you are most likely to hit are listed under what the schema enforces.

Registration or invitation returns 503​

Outbound email is neither configured nor declared absent. Set EMAIL_HOST, or declare global.required.smtp.deliveryMode: "disabled" — and check the rendered value for a typo first, because an unrecognised mode silently means smtp. See step 10.

A user cannot log in: "not yet verified"​

Expected after any self-registration, in every delivery mode — POST /registrations always writes is_verified: false. Either the user redeems a verification link, or an admin invites them and passes on the verification_link from the invite response. Do not reach for MongoDB for anyone but the very first admin; see step 10.

Passkey registration fails silently​

Nothing appears in any log, because the request never leaves the browser. The browser rejects an RP ID that is not equal to — or a registrable domain suffix of — the page's own domain. Two things to check:

  • A secure context. WebAuthn is unavailable on plain http origins, with localhost as the only exception browsers make. If you set global.scheme: "http", passkeys will not work on any real hostname.
  • A shared parent domain. If you serve the interface from several hostnames and want one passkey across them, override WEBAUTHN_RP_ID with the parent domain. The chart cannot derive that for you — doing it correctly requires the Public Suffix List, and a wrong guess produces an RP ID browsers reject outright.

See the values derived from global.domain.

redirect_uri_mismatch after clicking Authorize​

The callback registered with the provider is not what Plexicus sent. Compare it character by character against the callback URL rules. The commonest cause is an api. prefix that should not be there; the second commonest is bitbucket-cloud written with a hyphen instead of the underscore the route actually uses.

In-chart operator documentation​

The chart artifact bundles operator documentation under docs/. After running helm pull --untar (see step 7), the following guides are available locally:

ls plexicus/docs/
  • getting-started.md — customer install walk-through
  • secrets-management.md — secrets catalog and ESO/Sealed Secrets recipes
  • ingress-tls.md — ingress and TLS configuration
  • image-registry.md — image mirroring and air-gapped deployments
  • upgrading.md — how to upgrade to a new chart version and roll back
  • troubleshooting.md — common install failures and fixes
  • uninstall.md — clean removal of all chart resources

These ship with the chart and stay in sync with the version you installed. For environment-specific guidance, contact engineering@plexicus.ai.

For backup and restore procedures (MongoDB, PostgreSQL, object storage), see the Backup and Restore guide.