Building a Scalable Real-Time Fleet Management IoT Data Tracker with Kafka Streams and gRPC

In the IoT-enabled fleet management domain, real-time signal tracking is crucial. Signals refer to various datapoint readings from different sensors across the vehicle, like engine temperature, fuel level or braking force. Our solution processes batches of these signals, handling up to 8K batches (or 500K signals) per second in production every day. This talk explores our architectural journey, focusing on real-time, horizontal scalability, fault tolerance, monitoring and alerting. We utilized Kafka Streams' interactive queries API and a gRPC layer for Protobuf-formatted data storage and querying, achieving near-instantaneous data access. Key optimizations to both Kafka topology and cluster will be discussed, specifically aimed at reducing network overhead and controlling changelog size. These optimizations not only ensure resource efficiency but also enhance fault tolerance and rapid startups. Walk away with actionable insights for your own Kafka deployments.