Exploration of self-management for Kafka and stateful system at large scale in general

Jiangjie Qin

LinkedIn/Apache Kafka committer and PMC member

Jiangjie Qin is on the Data Infrastructure team at LinkedIn. He works on Apache Kafka and is a Kafka committer and PMC member. Previously, he worked at IBM, where he managed IBM’s zSeries platform for banking clients. Jiangjie holds a master’s degree in information networking from Carnegie Mellon’s Information Networking Institute.


LinkedIn runs more than 1800+ Kafka brokers that deliver more than 2 trillion messages/day. Running Kafka at such a scale makes automated operations a necessity. We will share the lessons we learned from operating Kafka at scale with minimum human intervention.


对于 Kafka 这一消息中间件的事实标准,学习超大规模集群下的如何进行自管理。