CTO, ActualyzeAI

6 minute read

KubeCon + CloudNativeCon North America 2025 in Atlanta wasn’t just another cloud native conference. It was the event where the industry collectively acknowledged a fundamental shift: enterprises are bringing AI home.

After three days of keynotes, technical sessions, and countless hallway conversations, one theme dominated everything else: data sovereignty. As someone who’s spent years in infrastructure (AWS, Cisco, Metacloud) and now works with enterprises on their AI challenges, this resonated deeply.

The Numbers Tell the Story

Let’s start with what the CNCF presented:

  • 52% of cloud native developers are now running AI workloads
  • 18% more are planning to adopt AI in the near future
  • The ecosystem has grown to 230+ projects, 275,000+ contributors, and 3.35 million code commits

The Data Sovereignty Imperative

Throughout the conference, I heard the same concerns echoed across industries:

Healthcare: “We can’t send patient data to cloud AI services. HIPAA won’t allow it, and our patients wouldn’t accept it.”

Financial Services: “Our models are our competitive advantage. We need complete audit trails and zero data egress.”

EU Manufacturing: “The Cyber Resilience Act deadline is December 2027. We need compliant infrastructure now, not later.”

One of the most compelling demos came from Deloitte showing lightweight AI agents running at the edge using Kubernetes, Ollama, and K3s. These agents process insurance claims locally, where the data already lives, eliminating cloud latency and privacy concerns while addressing the $265+ billion annual cost of claims processing errors.

This is the pattern we’re seeing everywhere: run AI where your data lives, not where it’s convenient for the cloud provider.

The Technology Stack Has Matured

What’s changed in 2025 is that the technology to support this vision is finally production-ready. Here’s what stood out:

Dynamic Resource Allocation (DRA) Goes GA

Kubernetes 1.34 shipped with DRA as generally available, and this is a game-changer for AI workloads. The old model of static GPU allocation created massive waste. You’d reserve an entire GPU for a workload that only needed it 30% of the time.

AMD demonstrated 50-70% efficiency gains through intelligent memory partitioning using DRA with Kueue and fractional GPU resources. For enterprises running expensive AI hardware on-premises, this directly translates to cost savings and better utilization.

Multi-Cluster Orchestration with Karmada

Bloomberg and Huawei presented real-world implementations managing thousands of Kubernetes clusters with Karmada. The capabilities are impressive:

  • Federated resource quotas across heterogeneous capacity
  • Multi-cluster queuing with Volcano integration
  • Federated HPA for cross-cluster autoscaling
  • Application-level failover with automated migration

For enterprises with multiple data centers, edge locations, or regional deployments, this solves the coordination nightmare. You can distribute AI workloads based on cost, latency, data locality, and availability, all while maintaining sovereignty boundaries.

AI Inference Infrastructure

Two projects stood out for model serving:

AIBrix, newly donated to the CNCF by ByteDance, addresses critical challenges in AI inference:

  • KV cache offloading for efficient memory management
  • Multimodal serving support
  • Intelligent request routing
  • Support for NVIDIA, AMD, and Huawei Ascend hardware

KServe graduated to CNCF status with version 0.17, featuring:

  • Dedicated LLMInferenceService CRD
  • Disaggregated serving architecture
  • Model and KV caching
  • Multi-cluster inference gateway with dynamic routing

Combined with storage layers like MinIO and vector databases like PostgreSQL with pgvector, you now have all the pieces for complete, production-grade AI stacks, entirely on open source.

Platform Engineering: The New Buyer

Perhaps the most significant organizational shift at KubeCon was the emergence of platform engineering as its own discipline. The CNCF released a Platform Maturity Assessment framework, and the findings were revealing.

Platform teams consistently reported:

  • Lack of dedicated resources
  • Insufficient focus on developer productivity
  • Low adoption rates of the platforms they build
  • Difficulty proving value to leadership

The solution? Treat your platform as a product.

Three principles resonated throughout the conference:

  1. API-first self-service - Developers shouldn’t need to file tickets
  2. Full lifecycle business compliance - Security and compliance built in, not bolted on
  3. Fleet management by capability providers - Standardize on patterns, not just tools

For AI infrastructure specifically, this means platform teams are now being asked to provide model serving, vector databases, GPU management, and RAG pipelines as internal services. They need solutions they can deploy once and let their entire organization consume through self-service.

The “app store moment” for enterprise internal platforms is here.

The Compliance Timeline Is Real

If you’re doing business in the EU or serving EU customers, the Cyber Resilience Act (CRA) timeline demands attention:

  • June 11, 2026: Governments and assessment bodies must be ready
  • December 11, 2027: Full regulation application

The conference emphasized that while services and many device types are excluded, commercial products incorporating open source software are affected. For enterprises deploying AI systems, this means:

  • Implementing security.txt for vulnerability disclosure
  • Generating SBOMs (Software Bill of Materials) for every release
  • Establishing clear processes for CVE reporting
  • Maintaining audit trails for all AI model training and inference

Organizations that treat compliance as a design principle, rather than an afterthought, will have significant advantages. Those scrambling to retrofit compliance onto existing systems will face expensive, disruptive projects.

What This Means for Enterprise AI Strategy

Based on everything I saw at KubeCon 2025, here’s where I think enterprise AI infrastructure is heading:

The Hybrid Model is Dead for Sensitive Workloads

For workloads involving PII, PHI, financial data, or competitive intellectual property, the “train in cloud, inference at edge” model is giving way to “everything on-premises.” The technology is ready, and the regulatory environment demands it.

GPU Economics Favor Ownership

With DRA enabling 50-70% better utilization, the economics of owning AI hardware are increasingly favorable, especially for organizations with sustained inference workloads. Cloud GPU pricing makes sense for burst capacity, but baseline workloads should run on owned infrastructure.

Multi-Cluster is the New Normal

Enterprises don’t have one Kubernetes cluster. They have dozens or hundreds, across data centers, edge locations, cloud regions, and air-gapped environments. AI workloads need to flow across this infrastructure based on data locality, compliance requirements, and cost optimization.

Developer Experience is a Differentiator

The organizations winning at AI aren’t just the ones with the best models. They’re the ones where developers can actually use AI capabilities without friction. Platform teams that provide self-service AI infrastructure will see dramatically higher adoption than those requiring tickets and approvals.

Key Takeaways

If I had to distill KubeCon 2025 into actionable insights for enterprise teams, here’s what I’d prioritize:

For Infrastructure Teams:

  • Start planning for Kubernetes 1.34+ with DRA if you’re running GPU workloads
  • Evaluate Karmada if you’re managing multiple clusters
  • Look at KServe and AIBrix for model serving. They’re production-ready

For Platform Teams:

  • Take the CNCF Platform Maturity Assessment
  • Design for self-service from day one
  • Think about AI infrastructure as a platform capability, not a one-off project

For Security/Compliance:

  • Start SBOM generation now
  • Implement security.txt across your stack
  • Map your CRA obligations if you’re in EU markets

For Leadership:

  • The data sovereignty conversation is happening whether you lead it or not
  • GPU ownership economics have shifted. Run the numbers
  • Platform engineering needs dedicated investment, not side-of-desk effort

The Bottom Line

KubeCon 2025 marked the moment when enterprise AI infrastructure became real. The building blocks are production-proven. The patterns are established. The compliance requirements are clear.

The question is no longer “Can we run AI on our own infrastructure?” It’s “How quickly can we get there?”

For enterprises in healthcare, financial services, manufacturing, and any industry where data sovereignty matters, the answer needs to be “now.” The technology is ready. The regulatory pressure is mounting. And the competitive advantage goes to organizations that control their AI infrastructure.

What’s your organization’s approach to AI infrastructure ownership? I’d love to hear how others are thinking about this. Reach out on LinkedIn or drop a comment below.

Tags: KubeCon, CloudNativeCon, Enterprise AI, Data Sovereignty, Kubernetes, CNCF, AI Infrastructure, Platform Engineering, Compliance, HIPAA, GDPR, Cyber Resilience Act

You May Also Enjoy