Kafka and Kubernetes

Ondat brings performance, scale, resilience and significant cost savings to the perfect Cloud Native partnership.

Get Started

Kafka and the Cloud Native language of events

Whether it is as an event store, a framework for stream-processing, or a messaging bus for service integration, however, we choose to use Apache Kafka, there is no escaping its inherent strength and suitability for microservices and Cloud Native development.

Modern Cloud Native applications are distributed, loosely coupled, and event-driven. For many organizations this makes Kafka’s event-driven architecture the ideal backplane for monitoring and connecting containerized apps and services, without fettering them through deeper, legacy-style integration. Kafka provides a fast, intuitive solution for tracking, coordinating, and integrating distributed services in the common Cloud Native language of events.

Kafka can be adopted for metrics and log aggregation, or more complex integration utilizing Kafka Streams and Kafka Connect, but there is one very common pattern among users: Kafka soon proves itself to be the fastest, most simple solution to an ever-growing number of challenges. The tactical tool becomes a key strategic messaging platform: scale, resilience, efficiency, management, and cost become crucial factors.

The Kafka Kube Combination

Just as Kafka provides a fast, effective answer for many of the challenges in Kube Native development, Kubernetes returns the favour. Kube provides the ideal solution for scaling enterprise-wide provision and management of Kafka through containerization and automation.

Kubernetes Operators such as Strimzi and Confluent now make it easier to install and configure Kafka on Kubernetes than it is to implement it in a more traditional VM or server environment. Running on Kubernetes, platform teams automate Kafka to create new clusters and scale instances up or down with a few simple commands. And for organizations rolling out Kafka across a global estate, Kubernetes provides a unified, single point of control for monitoring, management, and operation across diverse infrastructures on public, private, and hybrid cloud.

Ondat users running Kafka on Kubernetes gain

Operational cost savings of up to 90%
Vital improvements in failover and resilience
Increased application performance
Additional data security
Data portability with complete freedom from storage and cloud provider lock-in

Persistent Volumes - The Key to the Combination

The combination of Kafka running on Kubernetes provides a highly reliable solution. And one of the core strengths of Kafka is its ability to scale, handling vast amounts of data concurrently and in real-time. However, underneath its sturdy architecture, Kafka is a stateful application that relies on resilient, performant persistent storage volumes.

The architectural choices behind delivering persistent storage volumes to applications running on Kubernetes has significant implications on performance, resilience, security, and ongoing operational costs. For users running Kafka on Kubernetes at scale, Ondat’s kube - native storage mesh delivers:

Kube-Native Persistent Storage with Ondat

Cost Savings of up to 90%

Our Kube-Native data mesh opens up new technical possibilities for dramatically reducing storage costs. Ondat’s ability to harness local storage to provide Kafka with resilient, highly performant persistent volumes delivers new options on-Prem and in the Cloud.

Users with enterprise infrastructure can optimize and re-utilize existing storage investments for Kube Native development. Meanwhile, Public Cloud customers gain new options for reducing dependency on high-cost, networked storage and other hosted services.

Performance

Ondat’s Kube-Native architecture, its ability to parallelize the performance of connected storage, and the capability to leverage faster local storage devices, all combine to deliver Kafka and other stateful applications with truly unbeatable performance.

In addition, Ondat provides engineers and developers with stable, deterministic storage performance necessary for high volume concurrency, QoS (quality of service) guarantees, and performance aspects of modern SRE (site reliability engineering).

Resilience

Ondat gives users the critical ability to extend Kafka’s in-built failover mechanisms across multiple AZs (datacenter availability zones). In the event of the permanent loss of a Kafka broker through node-level failure, Ondat’s cluster-wide data replication also ensures Kafka broker pods can be reassigned to a new worker node and brought back online in seconds. Ondat achieves this without the need for resource-hungry data rebalancing within Kafka, which adds latency and saps application performance.

Data Security

Kafka offers in-built encryption of all data in transit. But in many industries, best practice and even regulatory compliance also demand the encryption of all stored application data. This is not possible natively in Kafka. However, Ondat persistent volumes deliver automatic encryption of all data at rest.

Data Portability and Freedom from Lock-in

Ondat frees your data, ensuring users can move applications between any Kubernetes distribution running on any combination of public, private or hybrid cloud, as well as bare metal or more traditional enterprise architectures. We do not sell storage, we give users choice between the full range of hosted storage options from their cloud provider, or their own storage platform. Vitally, we give users the ability to optimize these choices, and to move applications, removing all data lock-in to specific cloud providers or storage vendors.

Slash Operational Cost for Public Cloud

Ondat offers an unbeatable value-add solution for organizations using managed Kubernetes services from cloud providers. Our kube-native storage layer lets you utilize your cloud provider’s services more effectively.

On its own, Kube Native Kafka gives users new options to replace high-cost, hosted, cloud provider services for streaming, event and data pipelines, and ETL. But for customers already running Kafka in the Public Cloud, Ondat opens up additional paths for further, significant cost savings.

The majority of organizations running Kafka on Kubernetes in the Public Cloud will be restricted to using their cloud provider’s network-attached storage to deliver persistent storage. These services, such as: AWS EBS, Google Persistent Disk, and Azure Managed Disk, can cost up to 200 times more than local storage from the same provider. Ondat lets organizations create a resilient, high-performance Kube-Native storage pool using any available storage, including local disk.

Users can leverage the storage that comes bundled with their existing compute nodes and often lies redundant in Kube-Native environments. New storage capacity can be added at a fraction of the cost of other storage services, and all this comes combined with the benefits of faster I/O performance.

Sign up for the Tech Preview

Learn how Ondat can help you scale persistent workloads on Kubernetes.

Get started for free