Skip to main content

Operations

Day-2 material for a deployment installed with the Production Installation guide.

Upgrading​

Automated upgrade​

update.sh performs the whole checklist below in order — plan, pre-upgrade backup, release-note gates, a pinned helm upgrade with your values file, then post-upgrade verification:

curl -fsSL https://docs.plexicus.ai/update.sh -o update.sh
sudo -E bash update.sh --check --values my-values.yaml    # plan only: version, gates, backup location
sudo -E bash update.sh --values my-values.yaml            # do it

It finds your values file from --values, ./my-values.yaml, ./values.yaml, ./values-override.yaml, ~/plexicus-deploy/my-values.yaml or ~/my-values.yaml; if none is found it falls back to the live helm values captured into the backup and tells you so. ArgoCD-managed installs are detected and left alone — the script prints the exact-pin steps instead of running helm against them. The manual recipe below remains authoritative; use it whenever you would rather drive helm yourself.

Pre-upgrade checklist​

Check your overlay for a pinned redis.websocketDb

global.required.redis.websocketDb must be 2, the chart default. If your overlay pins it — most do not — remove the pin or set it to 2. Redis pub/sub is scoped per database, so a publisher and subscriber on different indices simply stop seeing each other and live scan progress goes quiet with no error anywhere.

Back up data before every upgrade

Database schema migrations may run automatically during helm upgrade. If the upgrade fails mid-migration, the database may be in a state that the old chart version cannot read. A backup taken immediately before the upgrade is your only guaranteed rollback path for the data layer.

See the Backup and Restore guide for the full procedure.

# 1. Review release notes. For chart **1.2.71**: feature and fix release (remediation Activity Center with a batch activity API; the XBOM nav entry is hidden while a tenant's XBOM state is off; finding enrichment and the remedium Job resolve the SMART AI slot / single AI credential resolver; war room package v0.1.13; reconnecting an SCM connector reopens the scans it promised to; the weekly repository-priority job no longer reports healthy runs as failures; the Nexus results webhook is acknowledged only after the ingestion workflow actually starts) - no operator actions required; worker, fastapi and frontend images change. The nexus engine image stays `1.2.66`. For chart **1.2.69**: reliability patch (settled AI pentests no longer leave zombie Nexus Jobs behind - the worker's boot sweep reaps Jobs of terminal runs still proven live, capped at 50/boot with a 30-day window, same `PLEXICUS_PENTEST_ORPHAN_SWEEP` kill switch; repository-creation workflows report real per-repository failure reasons instead of blanket success and no longer persist a live JWT in workflow history; findings on tenants with a remediation automation binding get their `remediation_type` again - run scripts/ops/p102_backfill_remediation_type.py to backfill; completed Nexus pentests with no verified findings no longer publish report section headers as Medium findings and severity casing is canonical; plexalyzer build fix pinning the trivy DB mirror to ghcr.io) - operator action: run the P102 backfill if you use remediation automation bindings; worker and fastapi images change; mirror trivy-db from ghcr.io in air-gapped installs. The nexus engine image stays `1.2.66`. For chart **1.2.67**: reliability patch (a worker restart no longer orphan running AI pentests: on boot the worker recreates a missing monitor for any running pentest, settles runs that finished while orphaned and reaps their Jobs; disable with `PLEXICUS_PENTEST_ORPHAN_SWEEP=0`; also fixes AI connection tests blocked by a busy provider minute) - no operator actions required; the nexus engine image stays `1.2.66`. For chart **1.2.66**: feature and security release (the platform OAuth app secret can no longer be sent to a host a tenant chose - GitLab/GitHub/Bitbucket self-hosted connections need their own OAuth app, 409 with guidance otherwise; finding exports are closed for API tokens end to end - repositories CSV and the PDF lane answer 403, tool/provider facets answer 400, magic-filter is branded; the remediation pull-request claim no longer leaks its lock and transient Mongo errors retry; the Wiz connector code is removed, existing Wiz findings stay hidden as discontinued; worker, fastapi and frontend images change) - operator actions: self-hosted GitLab connections without the tenant's own OAuth app must register one and reconnect; air-gapped installs are unaffected. The nexus engine image stays `1.2.61`. For chart **1.2.65**: feature and security release (API tokens get allowlisted views of findings, repositories, clients and remediations; finding exports answer 403 to API tokens and are sanitised for the web; the bulk **Create pull request** opens one pull request per repository group and the remediation flow opens its pull request again; each finding owns its own remediation state; CI scanner tokens can only be managed by admins from a signed-in web session; API tokens die on deactivation or credential change; the CLI exchange token drops to 90 days; client login v1 (loopback PKCE and device flow) ships for the TUI and MCP; the GitHub connector OAuth callback is fixed; worker, fastapi and frontend images change) — operator actions: API-token scripts must move to the allowlisted endpoints (see the API changelog); CLI tokens older than 90 days keep working until their next exchange. The nexus engine image stays `1.2.61`. For chart **1.2.64**: security patch release (the connector-config routes `PUT/DELETE /clients/self/connectors/{name}` only accept known connector names, so arbitrary Client fields such as `subscription.tier` can no longer be written through them; fastapi image only) — no operator actions required; the nexus engine image stays `1.2.61`. For chart **1.2.63**: security patch release (API-token callers get an allowlisted `GET /pentests` and `GET /pentests/{id}` view and pentest logs answer 403, `GET /credits/summary` counters add `used`, `remaining`, `limit` and `unlimited`, and `GET /users/developers` returns only `_id`, `email`, `username` and `role`; scripts that read other pentest fields with an API token must switch to the allowlisted ones) — no operator actions required; the nexus engine image stays `1.2.61`. For chart **1.2.62**: security patch release (the 2FA-required login response no longer returns the TOTP secret; API clients must not read `otp_data` from it, users enrolled before this release should re-enrol 2FA; fastapi and frontend must be upgraded together) plus connector provider icons, the Findings columns panel and Azure DevOps repository display fixes — the nexus engine image stays `1.2.61`. For chart **1.2.61**: feature release (Deep Code Analysis is the new name and engine id of the AI SAST tool, the legacy id is still accepted; codex-remedium LLM traffic goes through the AI gateway, so the gateway must be enabled with its identity secret, or set `CODEX_REMEDIUM_VIA_GATEWAY=false`; per-provider SCM roles; editable BYOAI embedding model; PlexiFlow editor improvements) — the nexus engine image moves to `1.2.61` (a copy of the previous engine image, same digest); air-gapped installs mirror `nexus:1.2.61`. For chart **1.2.60**: patch release (Pentest list actions in the UI only) — no operator actions required; the nexus engine image stays `1.2.48`. For chart **1.2.59**: feature release (plan change requests and offers replace self-service upgrade; tenant AI connections API; PlexiFlow AI flow generation; AI SAST C# phase 2; the cloud-account, container-registry and legacy nuclei/strix scanning code is removed; stuck-findings sweep payload fix). Operator action: before upgrading the worker, delete any `registry-sync-*` Temporal schedules (the retired Registry workflows are no longer registered); the nexus engine image stays `1.2.48`. For chart **1.2.58**: security and feature release (self-service plan checkout and payment endpoints now answer `410 Gone` and the Stripe webhook no longer applies a plan, so plan changes are made by Plexicus staff only; per-repository xBOM analysis status; AI credits pilot usage page; fixes to the findings report date range, the findings bulk bar and IaC Dockerfile checks) — no operator actions required; the nexus engine image stays `1.2.48`. For chart **1.2.57**: feature release (AI credits usage page and tenant credits API, read-only while the credits engine is off; the Deep Code Analysis tool id `deep-code-analysis` is accepted alongside `plexicus-ai-scan`, no data migration; PlexiFlow fixes, SBOM licence resolution and a dashboard report fix) — no operator actions required; the nexus engine image stays `1.2.48`. For chart **1.2.55**: patch release (findings list no longer flashes empty or shows stale results; the pentest detail API scrubs engine logs; the false-positive mute opens its pull request and un-mute cleans up; AI-SAST retries a language on a provider rate limit instead of dropping it; PlexiFlow canvas shows branch handles for IF, on-failure and SWITCH; failed tools and the failure reason are cleared after a clean scan) — no operator actions required; the nexus engine image stays `1.2.48`. For chart **1.2.53**: patch release (PlexiFlow editor fixes: trigger configuration is saved, run outcomes are explained, AI-generated flows apply without duplicates; the legacy Strix/nuclei DAST scanning is retired, repository scans skip `strix`/`nuclei` and enabling them answers `410` while historical findings stay) — no operator actions required; the nexus engine image stays `1.2.48`. For chart **1.2.52**: feature release (Azure DevOps end to end: Repos, Pipelines setup, Boards work items, service hooks, Teams and Entra ID; finding enrichment no longer stalls on reasoning models; LLM usage is priced and recorded; discontinued cloud and registry scanning is hidden from the UI). Consumer-visible changes: the SARIF export now emits one run per tool (readers of `runs[0]` only must iterate all runs); Microsoft Teams notifications need a Teams Workflows webhook URL (retired Office 365 connector URLs are no longer accepted by Teams); Azure DevOps repositories imported before this release keep their old display name and get no service-hook subscription until re-imported. No values changes required; the nexus engine image stays `1.2.48`. For chart **1.2.51**: patch release (the nightly `fastapi` daily-task jobs run again in release images and the HistoricalGraph snapshot no longer aborts on UUID repository ids; new `fastapi.cronjob.jobs` value selects which nightly jobs run, empty = all as before) — no operator actions required; the nexus engine image stays `1.2.48`. For chart **1.2.50**: PlexiFlow parity release (every remediation, issue, bulk, scheduled-scan and capability action runs as a PlexiFlow flow). API contract changes: `POST /issues/{finding_id}` is now asynchronous and answers `202 Accepted` with the PlexiFlow run (poll `GET /automations/runs`); `POST /remediations/bulk` and `PUT /automations/settings` are removed. Operator actions: `analysis-scheduler` is removed, so drop any `analysis-scheduler:` block from your values file and, after the upgrade, delete the `plexicus-analysis-scheduler` Secret by hand (`kubectl -n plexicus delete secret plexicus-analysis-scheduler`); the nexus engine image stays `1.2.48`. For chart **1.2.49**: hotfix release (pentest logs no longer expose stored target credentials; flow remediations honour subscription and remediation quota; uploads to Google Cloud Storage fixed; the retired built-in purge no longer blocks workers) — no operator actions required; the nexus engine image stays `1.2.48`. For chart **1.2.48**: feature release (PlexiFlow always on and silent by default — the `global.features.plexiflow` value is removed and ignored if still set; retired always-on built-ins are purged on worker boot; emergency brake; post-scan proof pipeline; default nexus engine image `1.2.48`) — no operator actions required; air-gapped installs mirror `nexus:1.2.48`. For chart **1.2.47**: patch (Azure DevOps clone username + TFVC PAT, anima v1.0.2, ai-scan no-retry-empty, finding-detail fail-open for unmapped SCMs) — no operator actions required. For chart **1.2.46**: feature release (operator-supplied pentest target auth, XBOM daily refresh, Azure DevOps end to end, PlexiFlow off by default) — no operator actions required. For chart **1.2.45**: hotfix release (fastapi multi-worker entrypoint + RBAC decision cache) — no operator actions required. For chart **1.2.43** (Nexus era): the Strix subchart is retired; AI Pentest runs as worker-spawned Nexus Jobs pinned via `global.required.scanJobImages.nexus`. Operators may delete leftover `plexicus-strix` resources after upgrade.  from the Plexicus team before running the upgrade.
#    Breaking changes or required values migrations are always listed there.

# 2. Take a backup immediately before upgrading.
#    (Full commands in the Backup and Restore guide.)

# 3. Pin the new version and upgrade:
helm upgrade plexicus \
  oci://europe-west3-docker.pkg.dev/plexicus-registry/charts/plexicus \
  --version <new-version> \
  --namespace plexicus \
  --values my-values.yaml

Post-upgrade verification​

After the upgrade, confirm the platform is healthy before considering the upgrade complete:

# All pods must be Running or Completed — no Pending or CrashLoopBackOff
kubectl get pods -n plexicus

# API health endpoint must return 200
curl -s -o /dev/null -w "%{http_code}" https://api.<your-domain>/health
# Expected: 200

# Smoke test: open https://<your-domain> in a browser, log in, and run a scan

The upgrade is successful when all pods reach Running, the /health endpoint returns 200, and the login flow completes without errors.

Rollback​

If the post-upgrade verification fails, roll back to the previous Helm release revision:

# List available revisions
helm history plexicus -n plexicus

# Roll back to the previous revision (or specify a revision number)
helm rollback plexicus -n plexicus
Rollback does not downgrade database schemas

helm rollback restores the previous Helm release state — it does not revert database schema migrations that may have run during the upgrade. If the schema was migrated forward, the rolled-back application pods may fail to connect to the database. In that case, restore the database from the backup you took before the upgrade rather than relying on helm rollback alone.

Troubleshooting​

Indexed by the symptom you actually see, because that is what you have when something goes wrong. Causes that are documented in depth in the Production Installation guide are linked rather than repeated.

Pods stuck in ContainerCreating, with a FailedMount event​

kubectl -n plexicus describe pod <pod> | grep -A5 Events
# MountVolume.SetUp failed for volume "plexicus-license" :
#   secret "plexicus-license" not found

The plexicus-license Secret does not exist, or is in the wrong namespace. Both fastapi and worker mount it with optional: false, so the container never starts and there are no application logs to read. The pods recover on their own once the Secret exists — no rollout needed. Full detail: Troubleshooting license failures.

ImagePullBackOff / ErrImagePull​

kubectl -n plexicus describe pod <pod> | grep -A3 Events

Usually one of:

  • the pull secret is missing from the plexicus namespace, or is named something other than the gar-secret the chart references (step 4);
  • global.imagePullSecrets does not list the name you actually created;
  • the credential has expired, or was created with the wrong --docker-server.

Confirm the Secret is the right type — this is the trap that looks like a credentials problem but is not:

kubectl -n plexicus get secret gar-secret -o jsonpath='{.type}'
# kubernetes.io/dockerconfigjson

kubectl create secret generic produces an Opaque Secret, which Kubernetes silently ignores for image pulls. Use kubectl create secret docker-registry (step 4).

worker CrashLoops with WORKER_CONTROL_PLANE_SECRET must be set​

Not a licence problem — see Troubleshooting license failures. The value must be identical in the plexicus-fastapi and plexicus-worker Secrets: the API presents it as X-Shared-Secret when it calls the worker's control plane, so a mismatch is as bad as an absence — the worker boots, then rejects every control-plane call.

EXPORTER_INTERNAL_TOKEN is not the same shape and will not produce this symptom: the worker never reads it at boot, so its absence does not stop the pod starting. It is read lazily by the daily EPSS-refresh schedule and sent to the exporter as X-Internal-Token. Missing or mismatched, that one daily activity gets a 401 — logged, but the workflow still completes, so nothing pages and EPSS data quietly goes stale. It is boot-blocking for the exporter pod. See the key catalog.

CrashLoopBackOff with a clean container start​

The Secret mounted correctly but the licence itself is invalid or expired. See Troubleshooting license failures.

The install fails before anything is created​

A values.schema.json rejection. Nothing was created, so there is nothing to clean up — fix the overlay and re-run. The error names the offending path; the ones you are most likely to hit are listed under what the schema enforces.

Registration or invitation returns 503​

Outbound email is neither configured nor declared absent. Set EMAIL_HOST, or declare global.required.smtp.deliveryMode: "disabled" — and check the rendered value for a typo first, because an unrecognised mode silently means smtp. See step 10.

A user cannot log in: "not yet verified"​

Expected after any self-registration, in every delivery mode — POST /registrations always writes is_verified: false. Either the user redeems a verification link, or an admin invites them and passes on the verification_link from the invite response. Do not reach for MongoDB for anyone but the very first admin; see step 10.

Passkey registration fails silently​

Nothing appears in any log, because the request never leaves the browser. The browser rejects an RP ID that is not equal to — or a registrable domain suffix of — the page's own domain. Two things to check:

  • A secure context. WebAuthn is unavailable on plain http origins, with localhost as the only exception browsers make. If you set global.scheme: "http", passkeys will not work on any real hostname.
  • A shared parent domain. If you serve the interface from several hostnames and want one passkey across them, override WEBAUTHN_RP_ID with the parent domain. The chart cannot derive that for you — doing it correctly requires the Public Suffix List, and a wrong guess produces an RP ID browsers reject outright.

See the values derived from global.domain.

redirect_uri_mismatch after clicking Authorize​

The callback registered with the provider is not what Plexicus sent. Compare it character by character against the callback URL rules. The commonest cause is an api. prefix that should not be there; the second commonest is bitbucket-cloud written with a hyphen instead of the underscore the route actually uses.

In-chart operator documentation​

The chart artifact bundles operator documentation under docs/. After running helm pull --untar (see step 7), the following guides are available locally:

ls plexicus/docs/
  • getting-started.md — customer install walk-through
  • secrets-management.md — secrets catalog and ESO/Sealed Secrets recipes
  • ingress-tls.md — ingress and TLS configuration
  • image-registry.md — image mirroring and air-gapped deployments
  • upgrading.md — how to upgrade to a new chart version and roll back
  • troubleshooting.md — common install failures and fixes
  • uninstall.md — clean removal of all chart resources

These ship with the chart and stay in sync with the version you installed. For environment-specific guidance, contact engineering@plexicus.ai.

For backup and restore procedures (MongoDB, PostgreSQL, object storage), see the Backup and Restore guide.