LinkedIn Introduces Northguard and Xinfra for Enhanced Log Storage and Pub/Sub

LinkedIn has introduced Northguard, a new log storage system that replaces Kafka, and Xinfra, a virtualized Pub/Sub layer. Northguard offers sharded data & metadata, log striping, strong consistency, and self-balancing clusters at a larger scale than Kafka. Xinfra enables seamless migration and unified access across Kafka and Northguard.

According to LinkedIn's engineers, Kafka had become challenging to manage at LinkedIn's scale. Northguard's architecture eliminates Kafka's limitations and provides a more flexible scaling mechanism.

Northguard's data model organizes logs into records, segments, ranges, and topics. This structure enables balanced load, high availability, and seamless scaling.

Northguard's metadata model uses sharded, Raft-backed replicated state machines distributed across vnodes. The protocols are optimized for performance and durability.

Migrating from Kafka to Northguard at LinkedIn's scale required seamless transitions. Xinfra, a virtualized Pub/Sub layer, was built to support this migration.

Segment storage in Northguard is pluggable, with the default implementation optimized for durability and latency.

To ensure reliability at scale, Northguard undergoes rigorous testing under deterministic simulation.

LinkedIn engineers have successfully migrated thousands of topics from Kafka to Northguard and over 90% of applications at LinkedIn are running Xinfra clients.

LinkedIn plans to finalize the implementation of Northguard and Xinfra internally before exploring the possibility of open-sourcing them.

According to the source: infoq.com.