Level Up Your Kafka Skills in Just 5 Days | Join Season of Streaming
AI-powered customer intelligence platform Dialpad provides the tools and technology to manage contact centers and customer engagements.
Dialpad’s Data Platform Team is responsible for creating features that their developers can build off of with ease. The team handles how and where data is stored, how it’s formatted, and who has access. While this involves a lot of traditional data engineering tasks, including distributed systems and database engineering, it also includes policy-level aspects such as governance and compliance. Their primary focus internally is supporting the data science teams in the AI organization using data systems for machine learning, model training for natural language processing (NLP), as well as labeling and annotating data. Externally, the team supports call center managers who use dashboards and charts pulling metrics directly from the Dialpad data platform.
Dialpad proactively surfaces real-time insights for call center managers such as queue times, handle times, and hold times. This allows for down-to-the-minute decision-making for how to reassign and repurpose agents. AI-powered live sentiment analysis powers management dashboards that can also show which agents may need immediate assistance or intervention handling problematic situations. The Data Platform Team also captures data around customer sentiment beyond simple satisfaction or dissatisfaction. Dialpad uses its own AI customer satisfaction (CSAT) system to look deeper into interactions and infer its own scoring.
To successfully do this work, Dialpad modernized its data infrastructure to exceed customer needs and bring new features to market, such as identifying moments of coachability for call center agents. The team chose data streaming from Confluent and real-time analytics from Startree to speed up innovation.
To power all of these capabilities on the back end, Dialpad was using a closed-source, proprietary database to handle the data warehousing work connected to Google Pub/Sub. This system was not adequate for the task of real-time, user-facing analytics—ingesting streaming data, dealing with rapid changes, and creating results, all within sub-second latencies. It provided internal insights into data, but couldn’t support an unbounded number of external customer queries on top of that.
Performance suffered as the result of the volume of queries. Even establishing a hard cap on the number of queries—100 QPS—did not resolve the issue. Queries regularly exhibited P95 latencies exceeding 30 seconds, and sometimes over a minute. This was far too slow for the real-time conditions of call center management, and not scalable to the high query concurrency of user-facing analytics. This intrinsically limited Dialpad’s ability to create new products and features that relied on real-time metrics.
Dialpad required four core capabilities in a replacement system:
Focus on streaming data
Support for upserts
Flexible indexing capabilities
Open source core backed by a strong sponsoring company
Streaming data was a necessity for real-time management of call centers, agents, and all relevant information about customers and their needs. Flexible indexing was also a strong requirement due to the multivariate ways Dialpad needs to analyze data. The fourth was a philosophical stance of the company to support open source projects and the vendors behind them.
Dialpad has a deep commitment to open source—and in particular Apache Pinot—not just as a user or consumer, but as a contributing organization. “That’s one of the great things about open source. You have that opportunity to own your own destiny from that perspective,” says Dialpad Engineering Manager Evan Galpin. “We are an atypical use case and we have added some features to help support that. We have multiple producer groups all producing part of the row, and we added this feature to the multiple comparison column so that we can keep all of those ordered correctly even though they are distinct from one other. Without open source, we’d be one company out of thousands. A vendor could say, ‘Yeah, that’s low priority because we don’t need it for everyone.’ But because it’s open source we can do it.”
Dialpad turned to StarTree Cloud, powered by Apache Pinot, which was designed specifically for the scale and speeds required by real-time use cases and offers a bring-your-own-cloud option. Dialpad implemented Confluent Cloud for fast and scalable data streaming—it worked out of the box and Confluent’s Terraform support made onboarding easy.
Combining StarTree Cloud and Confluent Cloud provided Dialpad with an end-to-end solution for producing data insights from their customer engagement platform.
Dialpad still uses Google Pub/Sub to ingest streams of events from an array of distinct producers, including the tens of thousands of live agents using the Dialpad app. From there, data for live analytics moves through Google Dataflow, and, using Apache Beam, populates data into topics in Confluent. Confluent then passes data into StarTree Cloud.
The new architecture has reduced average end-to-end ingestion latencies from over one minute down to an average of 500 ms and as low as 10 ms, greatly improving data freshness. While there are still cases where data can take 30 to 60 seconds for ingestion, these are situations where Dialpad is intentionally waiting to deduplicate and de-noise events—resulting in higher data quality. Beam performs enrichment and pre-materialization of select columns, rather than performing query-time computation.
Dialpad is using the partial upsert functionality of Apache Pinot because data is incomplete at first as it populates into StarTree Cloud. Over time, subsequent upserts fill in the blanks by adding in additional columns as data becomes available. The team also uses Pinot’s offline table feature to ingest data from the data warehouse, combined with the live stream data in a hybrid table for a complete picture of customer call center activity.
Dialpad also uses a number of indexing capabilities within Apache Pinot: full-text search based on the embedded Apache Lucene engine, JSON indexing, and range indexing.
Next, the Dialpad Data Platform Team plans to use timestamp indexes, because so much of their data is time series-oriented, related to calls and meetings. They are also looking at the star-tree index unique to Apache Pinot to help quickly narrow down the result sets they need to analyze. Dialpad already plans to expand their existing datasets and add new datasets, plus expand the range of queries and use cases they support.
If you believe your own use case bears more than a passing resemblance to Dialpad’s, you are not alone. There is a growing movement to offload data warehouse systems and move analytical workloads to designed-for-purpose real-time OLAP databases.
StarTree Cloud is built for real-time, user-facing analytics—allowing sub-second queries with high concurrency against even petabyte-scale data. Plus, the easiest and most reliable way to interconnect such systems is with a fully managed data streaming solution like Confluent Cloud.
Whether your data sources are IoT-enabled devices, SQL or NoSQL databases, web apps, mobile devices, or other endpoints, you can flow it all into Confluent topics, which can then be immediately ingested and indexed in StarTree Cloud. You can also combine the power of real-time data with data found in your data warehouses to create hybrid tables, by using StarTree’s Delta Lake connector to ingest data from Delta Lake tables into StarTree Cloud.
Discover what you can achieve combining the powers of Confluent and StarTree
Discover how Apache Kafka and Apache Pinot work together seamlessly
Sign up for a free tier account on StarTree Cloud
Get started free on Confluent Cloud
This blog explores how cloud service providers (CSPs) and managed service providers (MSPs) increasingly recognize the advantages of leveraging Confluent to deliver fully managed Kafka services to their clients. Confluent enables these service providers to deliver higher value offerings to wider...
With Confluent sitting at the core of their data infrastructure, Atomic Tessellator provides a powerful platform for molecular research backed by computational methods, focusing on catalyst discovery. Read on to learn how data streaming plays a central role in their technology.