XDI Systems: From Unpredictable AWS Bills to Bare Metal

XDI Systems provides physical climate risk assessments to major global banks and governments. Their models analyse individual assets such as harbours, buildings, and critical infrastructure against climate projections and produce detailed per-asset reports, which requires a large and growing amount of compute.

Over several years, their AWS setup had grown to include three separate EKS clusters across multiple sub-accounts, legacy servers, and a managed database service, with configuration spread across multiple systems. Their data scientists also needed Linux VMs for analysis work, and while the development team was already using Kubernetes, one engineer was managing the entire setup alongside other responsibilities.

Monthly AWS spend averaged $25,000 USD and spiked unpredictably. A single EFS transfer event cost $20,000 over three days, and legacy on-demand instances for data science work added another $5,000 to $10,000 per month without cost controls. XDI Systems asked us to take full operational ownership of their infrastructure.

We migrated XDI Systems from three separate AWS clusters to a 10-node, multi-AZ bare-metal cluster in Germany that runs our full managed Kubernetes stack (RKE2 and Cilium with WireGuard node-to-node encryption) and separates production, QA, and development into logical environments. We phased the migration over four months, starting with QA, followed by production and then development, and billing only started once workloads were running on the new platform. Production workloads get scheduling priority on the cluster, while data analysis workloads use spare capacity when it is available.

200 Gbps NVMe object storage and shared filesystem

We deployed an erasure-coded MinIO object storage cluster across dedicated 7.68 TB NVMe drives per node, paired with JuiceFS (backed by in-cluster Valkey Sentinel metadata and local NVMe read caches) as a POSIX-compatible shared filesystem to replace AWS S3, Cloudflare R2, and EFS. The object storage cluster benchmarks at 200 Gbps of aggregate throughput and handles 50,000 requests per second. At sustained throughput, the equivalent S3 GET request volume alone would cost approximately $55,700 USD per month on AWS, whereas our bare-metal cluster runs at a fixed monthly price.

KubeVirt VMs and custom Rust CLI for data scientists

XDI Systems’ data scientists need full Linux VMs for analysis work, which previously ran as on-demand AWS instances without cost controls. We added a 48-core, 1.1 TB RAM bare-metal host to run these VMs inside the cluster using KubeVirt, and we built a custom Kubernetes operator and CLI in Rust that handles provisioning, networking, resizing, monitoring, and automatic shutdown. When a data scientist provisions a VM through the CLI, it automatically joins their private Tailscale mesh network.

High-availability PostgreSQL

We migrated XDI Systems’ databases from their managed database vendor to in-cluster PostgreSQL on StackGres, backed by AES-256-GCM encrypted local ZFS NVMe storage, and we have since handled their major version upgrades. Every database runs as a primary and replica pair with automated snapshot backups, point-in-time recovery, and two to four times the resources of the managed instances it replaced.

Private network and ISO 27001 compliance

We deployed a self-hosted Headscale coordination server so the entire team can access the cluster and internal services through a Tailscale mesh network without managing a traditional VPN. Moving off AWS also removed the need for Security Hub, because there are no longer dozens of separate managed services to aggregate findings across. To meet their ISO 27001 requirements, we enforce network policies with Cilium and run Falco for kernel-level runtime threat detection alongside Kyverno for policy enforcement, which makes security events across the cluster auditable.

“Small configuration changes on AWS lead to them charging you orders of magnitude more money. That’s not going to happen on bare metal.”

Sohum Banerjea, Senior Architect, XDI Systems

We work with XDI Systems’ team every day in Slack and hold fortnightly calls to coordinate priorities and work through their infrastructure backlog. We respond to alerts, handle capacity planning, and help debug application-level issues, and we pass our findings back to their developers with specific fixes. Their engineers also use our Grafana observability stack directly, where we build custom dashboards so they can see how their applications behave across the cluster.

After migrating their workloads, we observed early batch runs and saw high context-switch rates as the Linux scheduler struggled with bursty CPU demand from individual climate-modelling processes. We built custom metrics to measure actual throughput against allocated resources, then constrained per-pod CPU allocations to slow individual processes down slightly. This made scheduling deterministic and allowed us to pack batch jobs tightly across all 10 nodes. We also tightened memory limits so that leaking processes are terminated before their memory usage balloons and sits idle, and we tuned those limits over several days to maximise total cluster throughput.

“Thank you very much for being so responsive, by the way. It is awesome.”

Tim McEwan, CTO, XDI Systems