You’re Spiky and We Know It

Twilio is a communications company with SMS, Voice APIs which see traffic spikes and troughs based on times of the day and week when companies send out communications(think early morning stock updates) or when end customers interact with companies(think restaurant delivery orders placed at lunch or dinner times) When building a generic monitoring system using Kafka Streams with exactly-once-processing, this spiki-ness can cause some set of our customers(think companies with a lot of interactions as above) to have more traffic at certain times of the day which can let them overpower traffic for other customers(think a group of hospitals sending communication about appointments). This talk elaborates the challenges that Twilio faced when building such a monitoring platform, which can aggregate customer data and send alerts in a timely manner under SLA. The optimizations we would go in depth are :-

1. TOPOLOGY - Spreading of work in Kafka Streams application topology
2. STORAGE - Statestore reads and writes from topology nodes to avoid data skewness
3. COMPUTE - Use of Punctuator from Processor API to avoid competition between customers

Do listen to this talk if building a real time alerting pipeline OR spreading a mix of global and local state stores OR aggregations over upserts OR use of punctuators in the wild piques your interest.