How Apna, Glance, & Meesho Are Innovating with Data Streaming For Consumers in India

What do Apna, India’s largest hiring platform currently connecting 40 million job seekers to 445,000 employers across 174 cities, Meesho, India’s fastest ecommerce platform to cross one million registered sellers, and Glance, InMobi-owned live lock screen infotainment platform operating in India and Indonesia with 200 million active users, all have in common?

The short answer? Use of data streaming.

These startups are all using data streaming technologies to transform and scale their business—and deliver unparalleled customer experiences.

At the recent Data in Motion Tour event in India, representatives from these businesses shared their data streaming journey, the challenges along the way, and how Kafka and Confluent Cloud are helping pave their path to success. 

How Hiring Platform Apna Uses Data Streaming for Unlocking New Opportunities for Its Users 

When jobs and professional networking platform Apna embarked on its mission to empower the rising workforce of India with better professional opportunities, relying on a monolithic architecture quickly proved to be a challenge. That was especially true when it came to scaling their hiring platform and meeting the demands of a rapidly growing user base.

After exploring options for a new approach to data infrastructure, Apna decided to pursue a multi-tier, event-driven, service-oriented architecture powered by Apache Kafka.

“Today, our architecture is service-oriented, meaning our applications are built as a suite of loosely coupled, independently deployable services. These services are generally serving either one or a group of tightly coupled business or product capabilities,” said Ravi Singh, principal architect at Apna.

“Our architecture is event-driven, meaning events are the primary mode of communication across these decoupled services,” Singh said. “It’s also multi-tier, which means these services are grouped into multiple layers or tiers depending on the functionality or the purposes they're solving for.”

But as a growing company, Apna didn’t want to invest a lot of resources into managing and scaling up Kafka. Apna adopted Confluent Cloud to achieve the scalability needed to power real-time microservices while offloading Kafka infrastructure maintenance to the experts.

Since implementing Confluent Cloud, Apna has largely transitioned away from its monolithic roots and is powering several critical microservices with real-time data streams in the cloud. A few of the company’s most high-impact use cases include job matching, job searching, application tracking, community feed, and data lakehouse.

Apna currently uses Kafka in multiple different ways:

• Kafka acts as an event manager for their event-driven service-oriented architecture. It solves for event sourcing, routing, transformation, and ordering. 

• Kafka is the default choice for async mode of communication between these decoupled services, majorly for implementing patterns like chain of responsibility, command pattern, etc.

• Kafka handles ephemeral storage.

• Using Kafka manages distributed task scheduling 

• Using Kafka helps the team build and use data pipelines

“The first thing that stands out is the resilient, highly available architecture with 99.99% uptime that Confluent has made possible for us,” Singh said. 

Plus, Confluent ensures:

• Scalability: >100k/sec supported with p95 latency of less than 5 milliseconds

• Security: Transit and end-to-end data encryption options

• Seamless integration with other open source technologies

• Apna doesn’t have to invest any bandwidth into cluster maintenance 

• Great availability and guidance from Confluent’s solution and support teams

What’s next for Apna?

“We’re continuously working towards enhancing our multi-objective job recommendation engine to provide the best possible job recommendations to our users,” Singh said. “We are introducing deep learning to our community feed models and continuously working on data democratization. That is how we can make data accessible to every employee at APNA so that we can implement this ideology of data-informed decision-making throughout the company.”

How Glance is Using a Data Streaming to Power Innovation in Digital Experience

The creation of smart lock screen provider Glance in 2019 marked a turning point in the mobile industry. By analyzing user behavior and interests, Glance delivers customized content—including news, sports, gaming, shopping, and live trends—on the lock screen that is tailored to the user's preferences. This makes it easier than ever for users to access the content they care about most without having to navigate through multiple apps.

And the lock screen product has resonated with the customers: Glance got to 100 million users in 21 months.

“An interesting statistic we found out is that people were unlocking their phone at least 150 times a day,” said Arvind Jayaprakash, senior vice president of technology at Glance. “Each of those moments is an opportunity for us. They're going to unlock their phone and move on and that's the amount of time we have to make an impression.”

This also poses a unique challenge: Trying to understand the user's taste and what they will care about in the next few minutes. This entails processing enormous amounts of data in the hopes of hooking people with really interesting content.

And it’s a data streaming infrastructure that powers all of this.

A single incoming consumer interaction event has multiple users internally, meaning different applications within the organization may have varied purposes for consuming and analyzing the event. And those applications are going to evolve over time.

Kafka plays a crucial role in enabling this by allowing Glance to capture those events without the need for prior knowledge of the specific consumer app that’s going to consume the event, because those apps are going to evolve over time. Similarly, the consuming applications don't have to know about what created the event.

Kafka helps decouple data producers and data consumers from each other and serves as the central nervous system facilitating the flow of information and events within the organization’s architecture, and enables real-time consumption of data by a diverse range of applications.

But there are data governance challenges that come with setting up an organization that has made it possible for “anyone to consume data in a decoupled manner,” Jayaprakash said, mostly around data governance. For example:

• Schema management: A consumer interested in reading a message on a particular topic needs to know about the message schema and the deserialization library to be able to programmatically consume the message.

• Data lineage: What happens when the pipeline from which a consumer is consuming a stream of data suddenly goes dry? Who do they reach out to for resolving the issue?

This is where using a fully managed service provider for Kafka makes a difference.

“Confluent goes beyond Kafka in that it helps us ensure governance in a decentralized manner to safely democratize access to data, enables a microservices architecture that helps create self-contained teams, and data observability, which benefits the entire data pipeline performance,” Jayaprakash said.

How Meesho is Using Data Streaming to Democratize E-commerce in India 

With a lofty vision of democratizing e-commerce in India, SoftBank-backed online retailer Meesho has been experiencing immense growth in recent years.

During the 2022 Diwali sale event, for example, the company hit 1 million peak requests per second.

“When that happens, you need to scale the platform and be vigilant 24/7 to avoid disruptions. All this while making sure standard day-to-day operations aren’t hampered,” said Katreddi Kiran Kumar, vice president of data platform engineering at Meesho.

This requires building a platform that’s scalable, extensible, resilient, agile, and has observability built into it.

Plus, Meesho users are completely unique in that the majority of them come from tier 2+ towns, meaning they are usually new internet users and first-time online shoppers or sellers. For Meesho, this means implementing features that make app usage and interactions easy for the user.

And data is at the core of it all—be it using data for fueling Meesho’s real-time recommendation engine, enriching the entire user experience for sellers, re-sellers, and consumers, and supporting business decisions.

“At Meesho, we collect all the data through our ingestion platform and then the first hop is Kafka, where we dump all the data. And all other use cases pick up from there,” he said.

And while Kafka is built to be easy to get started with, there are many operational and management challenges that ensue as Kafka usage and use cases grow within a company.

While Meesho is a big advocate of open source software, as a high-growth startup they wanted their engineering resources to focus on driving customer use cases rather than managing Kafka challenges—and offload infrastructure overhead as much as possible.

“We were grappling and building, rebuilding our data architecture, even our platform architecture multiple times in the last two years because of the exponential growth of data,” Kumar said.

“Ultimately, we wanted a scalable Kafka architecture where … we shouldn't be sitting and scaling our Kafka clusters and only provisioning for peaks on the platform and wasting money as a result. We wanted our platform to be completely scalable and at the same time it should happen automatically in the platform.”

Enter Confluent Cloud.

Today, Confluent Cloud is at the core of Meesho’s ingestion layer, eliminating the headache of managing rapid scale, enabling faster time to market, and accelerating Messho’s goal of democratizing data within the organization. Confluent helps: 

• Meesho to efficiently manage throughput bursts and enable ease of scalability with incremental expansion

• Meesho’s data team to handle 3x load during regular sale days with elastically scalable Confluent Cloud clusters

• Meesho to minimize operational burden with its elastic scaling capability and help maintain consistency across services

• Meesho to save its engineering teams’ time from maintaining infrastructure so they can focus on other high-value tasks

“Our total cost of ownership has gotten much better than it used to be and actually more competitive than running open source clusters on our own. Together with Confluent we were able to solve the network challenges we had and to run in an optimal manner,” Kumar said. “Ultimately, our vision is to build for a billion Indians and we are optimistic of achieving that goal.”

What will you build with data streaming? 

Not sure yet? Read the “2023 Data Streaming Report: Moving Up the Maturity Curve” to see how data streaming is powering real-world benefits and use cases for companies today.