AmpNexus
Engineering

Safe, staged firmware rollouts at scale (how AmpNexus updates EV charge points)

January 4, 2026AmpNexus Engineering
firmwarerolloutsotaobservabilityocppev-charging
Safe, staged firmware rollouts at scale (how AmpNexus updates EV charge points)

Keeping a fleet of chargers secure and reliable means shipping firmware changes regularly—bug fixes, safety improvements, new features, compliance updates—without bricking devices or taking sites offline.

AmpNexus’ firmware rollout mechanism is designed to make updates repeatable, observable, and boring (in the best way): publish an artifact once, target the right audience, roll out in stages, and monitor success in real time—with audit trails and quick rollback.


What this system solves

Firmware updates fail for predictable reasons:

  • Devices are spread across unreliable networks (4G/5G, NAT, captive portals).
  • Fleets are heterogeneous (different models, hardware revisions, feature flags).
  • A “working” update can still be “bad” if it introduces regressions at scale.
  • Manual USB updates don’t scale and don’t provide visibility or auditability.

So our approach focuses on:

  • Staged rollouts (canary → phased → broad release)
  • Policy-based targeting (firmware groups per tenant / model / cohort)
  • Verifiable artifacts (hashes + signing)
  • Strong observability (progress, failures, reasons, per-device history)
  • Safe stop / pause / rollback controls

High-level architecture

At a high level:

  1. CI builds firmware, runs tests, produces an immutable artifact + metadata.
  2. AmpNexus stores the artifact and makes it available for device download.
  3. A rollout defines who gets it, when, and how fast.
  4. Devices update and continuously report status back to the platform.
Rendering diagram…

Screenshot suggestion (S1): Architecture overview graphic

  • Export this diagram to SVG/PNG (or recreate as a branded “boxes & arrows” image).
  • Place it here under “High-level architecture”.

Step 1 — Publishing a firmware artifact (build → sign → store)

A firmware release in AmpNexus starts life in the CI pipeline. The output is an artifact that’s:

  • Immutable (same input = same version)
  • Traceable (commit SHA / build ID / changelog link)
  • Verifiable (SHA-256 + signature)
  • Typed (model / hardware revision / minimum bootloader)

Typical metadata stored with an artifact includes:

{
  "firmwareId": "fw_01J...cuid",
  "version": "2.7.0",
  "model": "JOLT-7kW-S3",
  "hardwareRevision": "v2",
  "sha256": "…",
  "sizeBytes": 1048576,
  "releaseChannel": "stable",
  "releaseNotes": "Fixes contactor sequencing edge case; adds improved Wi-Fi roaming; security patches.",
  "build": {
    "gitSha": "a1b2c3d",
    "pipelineId": "123456",
    "createdAt": "2026-01-04T00:00:00Z"
  }
}

Screenshot suggestion (S2): CI pipeline “Release” stage

  • Capture the pipeline view showing build + test stages passing, artifact upload, and the generated version.

Screenshot suggestion (S3): Portal → Firmware Library list

  • Show the firmware list with columns like Version, Model/HW rev, Channel, Uploaded, Status.

Screenshot suggestion (S4): Firmware detail page

  • Include SHA/hash, signature indicator, compatibility, release notes, and “Used by rollouts”.

Step 2 — Targeting: Firmware Groups (who should receive what)

Rather than manually selecting devices every time, we define Firmware Groups—audiences that can be reused across releases.

Examples:

  • Tenant “ACME CPO” → All JOLT 7kW devices
  • Tenant “ACME CPO” → London region pilot cohort
  • Internal → Dogfood devices

Groups can be built from:

  • Charge point model / hardware revision
  • Site / region / tags
  • Current firmware version (e.g., “< 2.6.x”)
  • Operational state (online/offline history)

Screenshot suggestion (S5): Firmware Groups overview

  • Show groups list + counts (Devices, Online/Offline).

Screenshot suggestion (S6): Create/Edit Firmware Group

  • Capture the rule builder (selectors + preview count) and the Save action.

Step 3 — Rollouts

Staged delivery with guard rails

A Rollout ties together:

  • A target Firmware Group
  • A firmware artifact (version)
  • A schedule / window
  • A staged plan (percentages or fixed counts)
  • Health thresholds (auto-pause conditions)

A typical staged rollout might be:

  • Stage 0: Canary — 5 devices (internal/dogfood)
  • Stage 1: Pilot — 5% of fleet
  • Stage 2: Ramp — 25%
  • Stage 3: Broad — 100%

With rules like:

  • auto-pause if failure rate > 2% in any stage
  • auto-pause if a specific error spikes (e.g., download failures)
  • manual pause anytime

Screenshot suggestion (S7): Create Rollout wizard

  • Show firmware selection, group selection, stage plan controls, and threshold settings.

Screenshot suggestion (S8): Rollout confirmation summary

  • The “review” screen with firmware version, group, stages, and thresholds.

Step 4 — Device update flow

Command → download → install → confirm

Once a rollout begins, devices move through a consistent state machine:

  1. Queued (eligible, waiting for its stage)
  2. Commanded (update instruction issued)
  3. Downloading
  4. Installing
  5. Rebooting
  6. Verified (device reports the new version)
  7. Failed (with a reason code)
Rendering diagram…

Screenshot suggestion (S9): Rollout progress dashboard

  • Overall progress bar, success/failure counts, per-stage breakdown, and pause/resume controls.

Screenshot suggestion (S10): Per-device update timeline

  • One device’s timeline: queued → downloading → installing → verified (with timestamps).

Screenshot suggestion (S11): Logs / events view

  • Filtered logs for the device during update (command issued, download started, hash verified, install complete).

Safety mechanisms (what keeps this from going wrong)

1) Eligibility checks

Before a device is instructed to update, we validate:

  • model + hardware revision compatibility
  • minimum bootloader requirements
  • device is online (or scheduled for next check-in)
  • not already on target version
  • not blocked by policy / incident (optional)

Screenshot suggestion (S12): “Skipped devices” panel

  • Show reasons like incompatible hardware rev, offline, already updated, blocked by policy.

2) Artifact verification

Devices verify what they download:

  • expected SHA-256 hash
  • signature verification (where supported)
  • secure transport (TLS)

Screenshot suggestion (S13): Artifact verification UI

  • “Verified” badge + hash + signature status.

3) Automatic pause and rollback-ready posture

If failure rates spike, we pause to keep the blast radius small.

Rollback is treated as a first-class operation:

  • mark version as blocked
  • stop further deployments
  • optionally create a “rollback rollout” targeting affected devices

Screenshot suggestion (S14): Auto-pause banner + threshold settings

  • Show the rule that triggered the pause and a CTA to inspect failures.

Screenshot suggestion (S15): Rollback action UI

  • Show “Block version” + “Create rollback rollout” actions.

API examples (so you can automate everything)

The portal UI is only one way to drive the system. Everything is exposed via API so operators can integrate rollouts into their own workflows.

List available firmware artifacts

curl -H "Authorization: Bearer $TOKEN" \
  https://api.ampnexus.app/v1/firmware?model=JOLT-7kW-S3&channel=stable

Create a rollout

curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  https://api.ampnexus.app/v1/rollouts \
  -d '{
    "firmwareId": "fw_01J...",
    "firmwareGroupId": "fg_01J...",
    "stages": [
      { "name": "canary", "type": "count", "value": 5 },
      { "name": "pilot",  "type": "percent", "value": 5 },
      { "name": "ramp",   "type": "percent", "value": 25 },
      { "name": "broad",  "type": "percent", "value": 100 }
    ],
    "thresholds": {
      "pauseOnFailureRatePercent": 2,
      "pauseOnConsecutiveFailures": 3
    }
  }'

Fetch rollout status

curl -H "Authorization: Bearer $TOKEN" \
  https://api.ampnexus.app/v1/rollouts/ro_01J.../status

Screenshot suggestion (S16): Developer docs excerpt

  • Capture the API schema for rollouts: request body + sample response.

Operational notes (what we learned building this)

  • Staged rollouts are the difference between “a bad day” and “five minutes of inconvenience”.
  • Failure reasons matter. A single “FAILED” status isn’t useful; “download timeout vs hash mismatch vs install error” is.
  • Fleet visibility is a feature. Operators want to answer: Who is affected? Where? Since when?
  • Make pause easy. People pause early if the UI makes it safe and obvious.

What’s next

We’re continuing to evolve the rollout mechanism with:

  • smarter cohorts (region/site/power profile awareness)
  • tighter integration with incident management and automated mitigations
  • more “fleet analytics” views (update duration distribution, network failure heatmaps)
  • delta updates (where feasible) to reduce bandwidth and time-to-install

If you’re building a fleet of connected devices and want to compare notes, get in touch.


Screenshot checklist (one-page)

  • S1 Architecture diagram
  • S2 CI pipeline release stage
  • S3 Firmware Library list
  • S4 Firmware detail page (hash/signature/compatibility)
  • S5 Firmware Groups overview
  • S6 Group rule builder + preview count
  • S7 Rollout create wizard
  • S8 Rollout summary confirmation
  • S9 Rollout progress dashboard
  • S10 Per-device timeline view
  • S11 Logs/events filter view
  • S12 Skipped devices + reasons
  • S13 Artifact verification UI
  • S14 Auto-pause banner + thresholds
  • S15 Rollback action UI
  • S16 Developer docs excerpt (API schema)