Onboarding Edge Devices to FlightCTL with FIDO Device Onboarding

I've written about what FDO is and why it matters and, more recently, about packaging all three FDO roles into a single bootc image for local testing. Both of those stopped short of the part that actually matters: getting a real device onboarded into a fleet manager without anyone touching it by hand. This post covers that — using the FDO all-in-one server to onboard edge devices into FlightCTL, an open source project for managing fleet of edge devices.

The short version: FDO's owner service isn't just a credential handshake. During the TO2 exchange it can run Service Info Modules (FSIMs) against the device — deliver files, run commands, all inside the same protocol exchange that hands over the device's identity. I'm using two of them, fdo.download and fdo.command, to push FlightCTL's enrollment credentials onto a device and start its agent as part of onboarding, not as a separate provisioning step bolted on afterward. Create a bootc image that layers go-fdo-client onto the FlightCTL agent base image from edge-fleet-management, and a device goes from freshly booted EC2 instance to pending enrollment in FlightCTL with zero manual steps in between.

Standing up the FDO server on AWS

I covered the Containerfile and systemd units for fdo-aio-server in the previous post, so I won't repeat that here — one bootc image running fdo-manufacturing, fdo-rendezvous, and fdo-owner as three systemd services on three ports. What that post didn't cover was actually getting it running somewhere a real device could reach, which is what the fdo-aio-server repo's Makefile and playbooks are for.

Build image, overlay cloud-init, push image to registry and create AMI

Three Makefile targets carry the image from source to something AWS can boot:

• build — compiles the two-stage image (go-fdo-server build stage, rhel-bootc runtime stage) and pushes it to the registry.
• cloud-init — layers cloud-init and open-vm-tools onto that image via a second, small Containerfile and pushes an :aws tag. Bootc images don't ship cloud-init by default, and bootc-image-builder needs it present to process an AMI's user-data on first boot.
• ami — pulls the :aws tag and runs Red Hat's bootc-image-builder container against it, privileged, with the local container storage and ~/.aws credentials bind-mounted in, to produce and upload an AMI.

sudo podman run \
	--rm \
	-it \
	--privileged \
	-v ${HOME}/.aws:/root/.aws:ro \
	-v ./ami/output:/output \
	-v /var/lib/containers/storage:/var/lib/containers/storage \
	--env AWS_PROFILE=default \
	registry.redhat.io/rhel9/bootc-image-builder:latest \
	--type ami \
	--aws-bucket bootc-amis-demo \
	--aws-region ap-south-1 \
	--aws-ami-name fdo-aio-server \
	quay.io/rprakashg/fdo-aio-server:aws

The sudo matters — bootc-image-builder reads the host's container storage directly, which means the rootful podman storage location, not a rootless user's. Run make build && make cloud-init && make ami in order and there's an AMI ID in the account a few minutes later.

Launch the server instance with FlightCTL enrollment credentials injected late bound using cloud-init

Before launching, generate the FlightCTL enrollment certificate the owner is going to hand to every device it onboards, and save it as config.yaml right next to the playbooks:

flightctl certificate \
    request \
    --signer=enrollment \
    --expiration=365d \
    --output=embedded > config.yaml

That file matters more than it looks. playbooks/templates/user-data.j2 writes it into the server's own filesystem at /etc/fdo/downloads/config.yaml via cloud-init on first boot — which is also exactly the file the owner's fdo.download FSIM will later hand to devices during onboarding (more on that below):

write_files:
- path: /etc/fdo/downloads/config.yaml
  permissions: '0755'
  owner: root:root
  content: |
    {{ lookup('file', 'config.yaml') | from_yaml | to_nice_yaml | indent(4) }}

With that in place, fill in vars/params.yaml — the AMI ID from the step above, subnet, and security group (the repo's own README table calls that field ami_id; the variable the playbook actually reads is just ami) — and launch:

ansible-playbook --vault-password-file <(echo "$VAULT_SECRET") launch_instance.yaml -e @vars/params.yaml

launch_instance.yaml loads vars/secrets.yaml (an ansible-vault file with admin_user, admin_user_password, admin_user_ssh_pubkey, key_name) and hands the rendered user-data.j2 to amazon.aws.ec2_instance as the instance's cloud-init payload — the same pattern used for the edge devices later, just serving a different file.

Once the instance is up, tell the manufacturing server where rendezvous lives and tell the owner where devices should reach it after redirection, using its public DNS and IP:

ansible-playbook configure.yaml -e fdo_aio_server_dns=ec2-43-205-93-231.ap-south-1.compute.amazonaws.com -e fdo_aio_server_ip=43.205.93.231

Screen capture below shows server up and running in AWS console

SSH into server or verify health check endpoint to ensure all three roles are running.

FDO Manufacturing

FDO Rendezvous

FDO Owner

The owner service: download fsim and command fsim

This is the part that turns FDO from a one-time identity handshake into an actual provisioning step. Here's the full owner config, etc/fdo/config/owner.yaml:

# FDO Owner Server Configuration
log:
  level: "info"

db:
  type: "sqlite"
  dsn: "file:/etc/fdo/db/owner.db"

http:
  ip: "0.0.0.0"
  port: "8043"
  # cert: /etc/pki/go-fdo-server/owner-https-example.crt
  # key: /etc/pki/go-fdo-server/owner-https-example.key

device_ca:
  cert: "/etc/fdo/pki/device_ca_cert.pem"

owner:
  cert: "/etc/fdo/pki/owner_cert.pem"
  key: "/etc/fdo/pki/owner_key.der"
  reuse_credentials: false
  to0_insecure_tls: false
  service_info:
    defaults:
      - fsim: "fdo.download"
        dir: "/etc/fdo/downloads"
    fsims:
      - fsim: "fdo.download"
        params:
          files:
            - src: "config.yaml"
              dst: "/etc/flightctl/config.yaml"
              may_fail: true

      - fsim: "fdo.command"
        params:
          cmd: "chown"
          args: ["root:root", "/etc/flightctl/config.yaml"]
          may_fail: true
          return_stdout: true
          return_stderr: true

      - fsim: "fdo.command"
        params:
          cmd: "chmod"
          args: ["0755", "/etc/flightctl/config.yaml"]
          may_fail: true
          return_stdout: true
          return_stderr: true

      - fsim: "fdo.command"
        params:
          cmd: "systemctl"
          args: ["enable", "--now", "flightctl-agent.service"]
          may_fail: true
          return_stdout: true
          return_stderr: true

service_info.defaults sets a base dir that fdo.download resolves relative paths against — /etc/fdo/downloads on the server, the same directory user-data.j2 wrote config.yaml into above. Everything under fsims then runs, in order, against the device during the TO2 exchange:

• fdo.download copies config.yaml from that directory to /etc/flightctl/config.yaml on the device — FlightCTL's enrollment cert and CA bundle, generated once on the server side and handed to every device that onboards through this owner.
• fdo.command chown root:root and chmod 0755 on that same path, since the file lands with whatever ownership the download step leaves it with, not necessarily what flightctl-agent expects.
• fdo.command systemctl enable --now flightctl-agent.service — the step that actually starts the agent and kicks off FlightCTL enrollment, triggered by the owner over the authenticated FDO channel rather than by cloud-init or an SSH session.

may_fail: true on every one of these steps is doing a lot of quiet work. There's no dry-run for service info execution — a typo in dst, or a device image where /etc/flightctl doesn't exist yet, fails the systemctl command silently instead of aborting onboarding. return_stdout/return_stderr: true surface the command output in the owner's logs, which is the only way to notice a step failed. Worth checking those logs the first few times a new device image goes through this, before trusting may_fail to mean "this is fine."

The device: layering go-fdo-client on the base image with FlightCTL agent

The device-side image lives in the same edge-fleet-management repo as the FlightCTL base image itself. images/fedora-bootc-base/Containerfile starts from fedora-bootc:43 and installs flightctl-agent from FlightCTL's own repo, plus podman-compose:

FROM quay.io/fedora/fedora-bootc:43

RUN mkdir -p /etc/flightctl

# Add Flight Control agent
ARG FLIGHTCTL_VERSION=1.0.2
RUN dnf -y install dnf5-plugins && \
    dnf -y config-manager addrepo --from-repofile=https://rpm.flightctl.io/flightctl-fedora.repo && \
    dnf -y install flightctl-agent${FLIGHTCTL_VERSION:+-${FLIGHTCTL_VERSION}} \
      --nodocs --setopt=install_weak_deps=False && \
    dnf clean all

# Add podman-compose tool
RUN dnf -y install podman-compose \
      --nodocs --setopt=install_weak_deps=False && \
    dnf clean all && \
    systemctl enable podman.service

images/fido-device/Containerfile takes that image as a build arg and overlays go-fdo-client from the Fedora IoT COPR:

ARG FROM

FROM $FROM

RUN dnf install -y 'dnf-command(copr)' && \
    dnf copr enable -y '@fedora-iot/fedora-iot' && \
    dnf install -y go-fdo-client && \
    dnf copr disable -y @fedora-iot/fedora-iot

RUN mkdir -p /etc/fdo

# Create a systemd unit that onboards the device using the credential blob
# generated by go-fdo-client device-init in cloud-init's runcmd. Ordered
# after cloud-final.service so the blob exists before onboarding runs.
RUN cat > /etc/systemd/system/fido-onboard.service <<'EOF'
[Unit]
Description=FIDO Device Onboarding
After=network-online.target cloud-final.service
ConditionPathExists=!/etc/fdo/onboard.done
Wants=network-online.target
Requires=cloud-final.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/go-fdo-client onboard --default-working-dir /etc/fdo --key ec256 --kex ECDH256 --blob /etc/fdo/cred.bin
ExecStartPost=/bin/touch /etc/fdo/onboard.done
StandardOutput=journal
StandardError=journal

TimeoutStartSec=300
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

The interesting part is the systemd unit it writes, fido-onboard.service. A few things worth reading closely:

• Requires=cloud-final.service / After=network-online.target cloud-final.service — ordering that matters: the device's credential blob (/etc/fdo/cred.bin) doesn't exist until cloud-init's runcmd has run go-fdo-client device-init, so onboarding can't start before cloud-init finishes.
• ConditionPathExists=!/etc/fdo/onboard.done, set by an ExecStartPost that touches that file — makes the unit idempotent across reboots; without it, every boot would retry FDO onboarding of an already-onboarded device.
• Restart=on-failure / RestartSec=10 / TimeoutStartSec=300 — the manufacturing, rendezvous, and owner servers might not be reachable the instant this unit fires, so it's built to retry rather than fail once and stay failed.

One thing worth flagging if you're following along: the Containerfile writes the unit file and gives it an [Install] section, but doesn't call systemctl enable fido-onboard during the build. As the repo stands today that means the unit exists on the image but won't start on its own — add RUN systemctl enable fido-onboard.service to the Containerfile if you want it to actually fire at boot.

Building the image and AMI

edge-fleet-management's Makefile builds several different device images off the same base, and it's worth knowing what each target actually does before reaching for one:

• base — builds images/fedora-bootc-base (the flightctl-agent + podman-compose image above) and pushes ${REGISTRY}/fedora-bootc-base:${BOOTC_BASE_IMAGE_TAG}.
• cloudinit — overlays cloud-init and open-vm-tools onto whatever image BOOTC_BASE_IMAGE points at, and pushes it with an :aws tag. Needed before any of these can boot from an AMI.
• fido-device — builds images/fido-device on top of ${REGISTRY}/${BOOTC_BASE_IMAGE}:${BOOTC_BASE_IMAGE_TAG} (i.e. on top of base by default) and pushes fido-device:latest.
• ami — pulls ${BOOTC_BASE_IMAGE}:aws and runs bootc-image-builder against it — same shape as the fdo-aio-server Makefile's ami target, but using the community quay.io/centos-bootc/bootc-image-builder image and an xfs rootfs.

For the FDO device path specifically, that's four commands in sequence:

make base REGISTRY=quay.io/<you>
make fido-device REGISTRY=quay.io/<you>
make cloudinit BOOTC_BASE_IMAGE=fido-device REGISTRY=quay.io/<you>
make ami AMI_NAME=fido-device BOOTC_BASE_IMAGE=fido-device

make ami needs an AWS account set up for VM import first — bootc-image-builder uploads the raw disk image to S3 and calls ec2:RegisterImage under a vmimport service role, which most accounts don't have by default. configure_aws.yaml creates the bucket, the vmimport role, and its policy in one pass:

export VAULT_SECRET=<your vault password>
ansible-playbook --vault-password-file <(echo "$VAULT_SECRET") configure_aws.yaml

Provisioning a device with the Ansible playbook

Device provisioning follows the same launch_instance.yaml shape as the FDO server itself, but parameterized differently — this repo's version takes user_data_template and device_count as variables rather than hardcoding them, since it provisions several different device flavors off one playbook. The FDO device's vars file, vars/fido-device.yaml:

instance_name: "fido-device-01"
ami: ami-03975b976bebb4cb8
instance_type: t3.medium
subnet_id: subnet-053842037f4ad3df0
security_group_id: sg-033428bf5df380f8b
storage: 128
microshift: false
device_count: 1
user_data_template: ./templates/user-data-fido.j2
manufacturing_server: http://ec2-13-202-44-205.ap-south-1.compute.amazonaws.com:8038

manufacturing_server is the FDO AIO server's manufacturing endpoint from earlier — the only FDO-specific configuration the device needs at launch time, since everything else (rendezvous address, owner config, FlightCTL credentials) is negotiated during onboarding rather than baked into the instance. templates/user-data-fido.j2 is much smaller than the template used for cloud-init-only devices, because it isn't carrying any FlightCTL secrets:

#cloud-config
users:
- name: "{{ admin_user }}"
  ssh_authorized_keys:
  - "{{ admin_user_ssh_pubkey }}"
  groups: ["wheel"]
  passwd: "{{ admin_user_password | password_hash('sha512') }}"
  shell: /bin/bash
  sudo: ['ALL=(ALL) NOPASSWD:ALL']

runcmd:
- echo "Welcome to FIDO Device" > /etc/motd
- go-fdo-client device-init '{{ manufacturing_server }}' --device-info '{{ instance_name }}' --key ec256 --blob /etc/fdo/cred.bin

Compare that to the non-FDO flow documented in the same repo, where write_files bakes FlightCTL's config.yaml — private key included — directly into the instance's user-data. Here, runcmd just runs go-fdo-client device-init against the manufacturing server, which creates a device credential and GUID but hands over no FlightCTL secrets at all. Those only land on the device after TO2 completes, delivered over FDO's own authenticated channel instead of sitting in EC2 user-data — which anyone with describe-instance-attributes permissions on the account can read back out. I could have also added step to start the oneshot fido-onboard systemd service we baked into the image that does the device onboarding. Once the ownership voucher is transferred over to owner server (TO0 protocol) the onboarding would complete successfully. For demonstration purposes we will manually start the service once we transfer over the ownership voucher.

Launch it the same way as any other device in this repo:

ansible-playbook --vault-password-file <(echo "$VAULT_SECRET") launch_instance.yaml -e @vars/fido-device.yaml

Screen capture below shows playbook executed successfully

Check if ownership voucher got created in manufacturing server by running command below.

curl http://ec2-13-202-44-205.ap-south-1.compute.amazonaws.com:8038/api/v2/vouchers

Grab the device guid and download the full ownership voucher from manufacturing server by running command below.

curl -v -H 'Accept: application/x-pem-file' http://ec2-13-202-44-205.ap-south-1.compute.amazonaws.com:8038/api/v2/vouchers/26bbe8572b8e21e066d4bfc2c5f4a7f5 > /tmp/26bbe8572b8e21e066d4bfc2c5f4a7f5

Send the ownership voucher to owner server by running command below

curl -X POST 'http://ec2-13-202-44-205.ap-south-1.compute.amazonaws.com:8043/api/v2/vouchers' -H 'Content-Type: application/x-pem-file' --data-binary @/tmp/26bbe8572b8e21e066d4bfc2c5f4a7f5

Make sure Owner server has the voucher by running command below

curl http://ec2-13-202-44-205.ap-south-1.compute.amazonaws.com:8043/api/v2/vouchers

SSH into device and manually start the fido-onboard service. If all goes well onboarding should complete successfully and we should see FlightCTL enrollment credentials for agent here /etc/flightctl/config.yaml

What actually happens at boot

Put together, a device goes through this sequence with nobody logged in:

1. cloud-init creates the admin user and runs go-fdo-client device-init against the manufacturing server, writing /etc/fdo/cred.bin.
2. cloud-final.service completes, satisfying fido-onboard.service's ordering requirement (once it's enabled — see above).
3. fido-onboard.service runs go-fdo-client onboard, performing TO1 (find the owner via rendezvous) and TO2 (mutual authentication and service info exchange) against the FDO AIO server.
4. During TO2, the owner's fsims run in order: it downloads FlightCTL's config.yaml to /etc/flightctl/config.yaml, fixes its ownership and permissions, and enables flightctl-agent.service.
5. flightctl-agent starts, reads its freshly delivered config, and enrolls with FlightCTL — showing up as a pending device in the UI, waiting for approval.

Keep in mind the onboarding will only be successful once the ownership voucher is sent over to owner server. Steps to do this are covered in this article.

None of the individual pieces here are new — cloud-init, systemd unit ordering, Ansible playbooks, an owner config file — but stacking them this way means the only thing that ever touches a device between launch_instance.yaml and "pending enrollment in FlightCTL" is the FDO protocol itself. The enrollment certificate never appears in user-data, the agent config never gets baked into an image, and there's no window where a device is up and reachable but not yet configured. What's still manual is generating that FlightCTL enrollment cert once per owner and dropping it where the download FSIM can find it — and, on the FlightCTL side, approving the pending device once it shows up.