Building Customer Engagement One Store at a Time

Personalization has been the gold standard for online customer interactions for a decade. Companies began with rough segmentation based on tagging content to measure marketing interactions and online behavior patterns. Now, with increased data across companies, a huge SaaS footprint of business and marketing operations tools, sophisticated consumer analytics capabilities and bottomless storage, we’re in the era of one-to-one personalization.

Customers now expect retailers to discern what’s relevant and interesting. They expect companies to reward loyalty and engagement, and use what they know to help consumers find exactly what they’re looking for right when they need it. Much of this evolution has been driven by online-first interactions.

Personalization is a force multiplier—and business necessity—one that more than 70 percent of consumers now consider a basic expectation. Organizations able to build and activate the capability at scale can put customer lifetime value on a new trajectory—driving double-digit revenue growth, superior retention, and richer, more nurturing long-term relationships.” — McKinsey

In addition to this digital transformation, with delivery options being more accessible, going to a retail outlet is increasingly a special trip. Customers start online with pricing and product research, then head to a physical location to handle merchandise, re-evaluate their wishlists, find additional discounts, and bring their purchase home right away. They’re much more informed about their options than ever before, and expect the same high-quality and personalized experience in store as they get online.

Mastercard recently found that customers want the same highly personalized experience in person as they do online; and they come to stores looking for it. “Personalization lets customers enjoy a hybrid electronic-physical shopping experience in which their digital curiosities are satisfied by actual goods.”

“For example, a select group of loyal customers can choose to receive personalized digital coupons upon entering the store. Alternatively, the shopper who's been ‘geofenced’ in the winter coat section can be offered a price that aligns with the cost of the last coat she purchased, or a price that reflects the deals she's explored on the store's mobile app.” — Mastercard

In a noisy digital environment, surrounded by constant email and SMS notifications from dozens of brands, many companies miss the opportunity to turn offline into a special, differentiated experience. 

The good news is that brick-and-mortar interaction can be improved, and can even win over digital-first customers—even if you have a few gaps or lags in in-store purchase data.

Enter the customer data platform

Customer data platforms bring together disparate sources of data across multiple channels. A full view of a customer’s activity allows companies to build rich behavioral and interaction profiles. The most effective profiles require a company to unite data from in-store POS and loyalty cards, then combine it with clickstream, direct marketing email opens, coupon conversions, and app installs to drive real-time personalization online.

This all requires a firm grasp of the company’s data model, governance, and how it all fits together. Older POS systems or legacy back-offices often make it hard to ingest real-time data—they move data hourly or daily in big batch jobs. So in-store purchasing intelligence lags the e-commerce arm, and as a result, brick-and-mortar customer experiences get lumped into that struggle too. They’re data sources to analyze later—but the online experience is where marketing teams put their time and effort, and do cool real-time stuff around customer personalization efforts.

But it’s possible, still, to create good hybrid shopping experiences even despite having a different cadence for data flow.

Requirements for the hybrid shopping streaming solution

With data streaming, you can react to a customer’s presence in your store in real time, with the right insights, the right interaction, and the right level of detail. The systems needed to do so will have the following features:

• High availability (HA) – According to Gartner, every minute of downtime can mean hundreds of thousands of dollars lost, and this is particularly true in the retail world where customers lost sometimes don’t come back. Any system that is orchestrating customer transactions and data should be treated as mission-critical; even if they’re not handling transactional data directly. A missed or late-arriving promotion or alert has a conversion and reputation cost: “We already had that conversation with the rep!,” “The store is now closed?,” “Oh… that was yesterday’s hours.” HA is essential, and a system that quickly scales out to multi-zone and even multi-region architectures will eventually be needed.

• Scalability – It’s a week before Black Friday and things are starting to get weird on the website. Can the system handle sudden spikes and scale up as loyal customers flock to the turkey aisle? Based on the true story of the author spending a 70h week before Thanksgiving pinpointing slow database queries to ensure 500K holiday turkey orders could get placed … systems must scale seamlessly during demand spikes or infrastructure failures. If you’re helping customers get around your store during a promotional event, delivering the latest crucial information is imperative—“e.g., turkeys now out of stock at X location! Go to Y instead.” Scalability is essential to ensure this, and it should be quantified and tested end to end.

• Low latency – To ensure a good user experience, latency is essential to send the right message to the user at the right time. A minute or two late, and the user has walked off, lost interest, or changed their mind. Someone can move several feet in just a second. Latency should be subsecond at a minimum, in the two-digit ms range ideally. For low-latency workloads, direct microservices communication almost immediately hits bottlenecks: like a relay race, one slow runner slows down everybody else too. Vertical scaling can help, but has to be done across multiple systems and can be a tricky dance of tracking down the slowest system. However, horizontal scalability can also become an issue with more traditional pub/sub services as they often rely on a centralized broker to handle message routing—a decoupled streaming data model solves both problems.

What does the hybrid shopping experience entail?

The keys to building this experience for customers include:

1. Data architecture and interaction strategy: To create a strong hybrid shopping experience, a strong centralized data governance practice is required—at both the business and IT levels, and across sources and systems both online and off. Data discoverability allows the organization to process and combine interactions without false-positives, do deduplication, and discover gaps in data sources. Data that’s not understood and cataloged can’t be used. In-store stock location data is essential, also, and may not exist in quite the right formats for geolocation yet.

As for strategy: what kinds of personalized messages will customers respond to well, and where? Last year Best Buy reported a 72% uptick in customers using the Best Buy mobile app while inside their stores. Likewise, Nike found that their mobile users spend nearly triple on their app than what they spend on Nike.com. 

So now that you have the right data and strategy, get tactical. What coupons, offers, and loyalty programs will best increase engagement? Could communication be via app, SMS, or email? Should any data be sent to store associates to give them tips about how to interact or direct the customer who has walked in, for a more boutique shopping experience? All of this, then, needs an app or website and a notification subscriber like Ably to power the full experience.

2. Real-time integration platform and insights: Next, you need a place for the data to interact with other data. Many CDPs and CRMs unite customer information across disparate systems, and provide BI analysis for later marketing intelligence or action. But the real key for a quality in-store experience is to be able to take action on customer activity as it happens—while leveraging customer data and machine learning (ML) insights like likelihood to buy or cohort analysis.

To get started, customer and transaction data can be streamed into Confluent Cloud from source databases or systems as it’s created (even if it is a bit stale, e.g., coming from a batch process) using change data capture, and transformed and standardized to be joinable and usable.

Views of current status of customer info, product, and revenue counts can then be materialized in ksqlDB, and a few other key BI aggregations to power some basic personalizations. These will then be joined with real-time customer movements later, and also used to find business intelligence correlations.

To take BI one step further, ML insights can be generated by a tool like Google ML Studio and be queried in real time, or specific propensities added/updated on a customer’s profile for every action they take (click, purchase, browse).

And finally, location can be treated like another clickstream from an app, and happen via geolocation—e.g., when a customer visits a site for promotions on their phone as they enter the store, if they haven’t yet installed your app.

3. Stream processing and message delivery: Once the data is flowing, you need to be able to take fast action on it using a stream processing solution like Kafka Streams or Flink. As soon as a customer opens the app or scans a QR code, or uses the in-store Wi-Fi to look up product information, the technology needs to keep up with their physical browsing—and be able to unite where they are (literally) in their shopping journey with known customer and inventory data. The results can then be pushed into a notification channel for delivery to the customer or staff’s device.

Implementation example

In the case of a hybrid retail customer, the goal is (most likely) to increase conversions by geo-targeting app users when they enter a store. One organization found that their customers would look at their app for coupons and deals of the day before heading in—but memories are short, and without a reminder or a nudge, it is too easy to forget what they came in for, and not be aware of updated promotions.

To solve for this they combined three things:

1. An app framed around couponing and loyalty, and including a product wishlist. This allows customers to create and check off their list while in the store, and find product locations by aisle or rack.

This has the bonus effect of encouraging the customer to keep the app open for more precise location information throughout—e.g., if background location is disabled. Using a scalable notification platform like Ably seamlessly handles routing push and app actions; the app then can tell Apache Kafka® when the user enters/leaves a geofence or the user performs an action like opening a product details view to find inventory.

Even absent an app, though, in-store Wi-Fi and call-to-action signs encourage a customer to check the site for details. That can get you 90% of the way to the same functionality via HTML5 geolocation, and that’s a great start.

2. Geofencing parts of the store that have items of interest, tied to a product category or, even better (but much more ambitious), SKU. This requires a location lookup for wishlist items, and a list of locations for products, promotions, and stores. When a user triggers a geofence on the app, the fence ID is sent to stream processing, triggering a join on products and/or segments currently on promotion.

For larger retail footprints with faster-changing stocking, store layouts, and inventory, the matching work can be distributed. The user’s location gets streamed to a decisioning engine every few seconds, to update the search for related promotions.

Operating systems (OSs) usually have limits for when geofence actions (entry, exit) are refreshed to reduce duplicates and bandwidth used, so it might be too slow for some use cases; instead, the geofence match can be done on the server-side with the user’s GPS location.

This does take more sophisticated logic, but is also more flexible. It offers huge benefits for precision and downstream analytics! But, it can introduce privacy issues too. So be sure to consider how you use and retain the data. It also requires adding a bit of deduplication logic into the stream processing, but that is straightforward with a tool like ksqlDB. The most sophisticated use case will call for writing custom functions to handle complex geometries; this lets you get down to the level of ultra-personalized to-the-shelf offers. But, it creates a much richer experience.

But a “lite” version of the functionality is great for getting started—e.g., for pickup notifications, or when the user accesses the website from a HTML5Geo lat/long-enabled browser, so you know their area of the store.

A stream processing engine like ksqlDB can match current promotions or product locations within a rough distance from a point; you might not always get to-the-meter data (e.g., if they’re on Wi-Fi only), but it’s also quite straightforward to do. Even simple geolocation that tracks distance from a store can make a good first step to send a simple welcome or an update of the latest relevant promotions.

3. Customer insights based on next-best product ML propensity from online and in-store purchase and browsing habits. This can be combined with a next-best channel ranking and likelihood to buy score lookup in ksqlDB, Kafka Streams or Flink. That can trigger an alert in the app, a chat pop-up on the site, or an email and ad retargeting for later (or no action at all, if an opt-out is set!).

Loyalty score threshold can unlock flash specials, and a message can also be sent to a coupon app to generate a unique code if the user clicks “apply.” This makes it straightforward to measure conversion and effective discount rates.

These three tools combine to create a way to alert customers when they qualify for deals, need help, are near an item they want, or to show alternative items as they browse.

A simplified demo using Confluent Cloud, ksqlDB, and Ably

Here’s something simple that you can build as a proof of concept to match promotions, location, and segmentation together in real time.

Components

Follow the instructions in the demo to create your Confluent cluster.

1. Confluent REST Produce, Postgres CDC, or Datagen Connector to mock up the location events.

2. The Ably Connector on Confluent Cloud, installed as a Custom Connector. Ably lets you do edge interaction fast with browsers, mobile devices, and IoT.

Ably sends promotions when product segment location matches the customer’s.

You can also use it to gather locations in the other direction—e.g., location data passed from the website or app via Ably Firehose service back into Kafka. For the example, we’ll mock that part of the data.

• Grab the connector from the Ably repository: https://github.com/ably/kafka-connect-ably#confluent-cloud-custom-connector.

• Go to Connectors, select “Add Connector,” then “Add Plugin.” Follow the instructions in the repo to install the connector.

• Set it to read from the customers.promotions topic.

• Configure two channels: Location and Promotions.

3. Create topics:

• stores.products: SKU, segment, variant, display lat/long

• products.promotions: pid, SKU, segment

• customers.locations: lat/long, (optional)uid

4. ksqlDB. We’ll leverage the “geo_distance” function for simple distance-from-location calculation. However, in advanced use cases this could be done with a UDF or a Kafka Streams app.

Logic: If a customer is within a three-feet radius of a product that matches a segment in the promotions topic, emit an event with product info and discount.

Confluent features

For those looking to go beyond our demo, it’s worth noting some of Confluent’s features which enable the delivery of in-store personalization: 

• Kora Engine – Confluent Cloud is built on Kora–the cloud-native engine for Apache Kafka®. Powered by Kora engine, Confluent has delivered GBs/sec for global retail giants, scaling seamlessly with increased demand. 

• Pre-built connectors – Confluent offers a library of over 120 pre-built connectors, 70 of which are fully-managed. 

• Multi-AZ dedicated clusters –  Confluent offers multi-AZ dedicated clusters, which ensure the high availability and reliability of streaming data pipelines for in-store personalization.

Conclusion

Building even a simple hybrid in-store personalized experience quickly shows benefits to the bottom line: customers are more likely to find what they need, when they need it. Managing expectations and improving their experience in the store leads to increased satisfaction, better conversion, and it builds trust that their needs are known and understood.

On the tech side, retail organizations can feed this added data into business intelligence applications, and improve campaign outcomes via better offers and more relevant products. Location data can improve demand prediction for staffing, and can improve a staff’s ability to help a customer in-store, too. This leads to increased employee engagement as well, which also has an impact on the bottom line.

Try out the demo at the repository below to see Confluent connectors and stream processing in action!

Partners — Ably

Ready to try Confluent Cloud?