I felt that my team, which was supposed to be made up of distributed systems engineers, was really acting more as distributed system plumbers. Worse, the complexity meant that the data was … So in 2010 we decided to build a system that would focus on capturing data as streams and use this as both the integration mechanism between systems and also allow real-time processing of these same data streams. This was the origin of Apache Kafka. We imagined …