Level Up Your Kafka Skills in Just 5 Days | Join Season of Streaming On-Demand
ETL
Streaming SFDC Data to BigQuery for Real-Time Customer Analytics
A common challenge organizations face is how to extract, transform, and load (ETL) Salesforce data into a data warehouse, so that the business can use the data. Salesforce (SFDC) is […]
Detecting Patterns of Behaviour in Streaming Maritime AIS Data with Confluent
This is part two in a blog series on streaming a feed of AIS maritime data into Apache Kafka® using Confluent and how to apply it for a variety of […]
Streaming ETL and Analytics on Confluent with Maritime AIS Data
One of the canonical examples of streaming data is tracking location data over time. Whether it’s ride-sharing vehicles, the position of trains on the rail network, or tracking airplanes waking […]
Streaming ETL with Confluent: Routing and Fan-Out of Apache Kafka Messages with ksqlDB
In the world of data engineering, data routing decisions are crucial to successful distributed system design. Some organizations choose to route data from within application code. Other teams hand off […]
Building a Cloud ETL Pipeline on Confluent Cloud
As enterprises move more and more of their applications to the cloud, they are also moving their on-prem ETL pipelines to the cloud, as well as building new ones. There […]
Pipeline to the Cloud – Streaming On-Premises Data for Cloud Analytics
This article shows how you can offload data from on-premises transactional (OLTP) databases to cloud-based datastores, including Snowflake and Amazon S3 with Athena. I’m also going to take the opportunity […]
How to Use Single Message Transforms in Kafka Connect
Kafka Connect is the part of Apache Kafka® that provides reliable, scalable, distributed streaming integration between Apache Kafka and other systems. Kafka Connect has connectors for many, many systems, and […]
🚂 On Track with Apache Kafka – Building a Streaming ETL Solution with Rail Data
Trains are an excellent source of streaming data—their movements around the network are an unbounded series of events. Using this data, Apache Kafka® and Confluent Platform can provide the foundations […]
Deploying Kafka Streams and KSQL with Gradle – Part 1: Overview and Motivation
Red Pill Analytics was recently engaged by a Fortune 500 e-commerce and wholesale company that is transforming the way they manage inventory. Traditionally, this company has used only a few […]
The Changing Face of ETL
The way in which we handle data and build applications is changing. Technology and development practices have evolved to a point where building systems in isolated silos is somewhat impractical, […]
Data Wrangling with Apache Kafka and KSQL
KSQL, the SQL streaming engine for Apache Kafka®, puts the power of stream processing into the hands of anyone who knows SQL. It’s fun to use for exploring data in […]
Apache Kafka vs. Enterprise Service Bus (ESB) – Friends, Enemies or Frenemies?
Typically, an enterprise service bus (ESB) or other integration solutions like extract-transform-load (ETL) tools have been used to try to decouple systems. However, the sheer number of connectors, as well […]
KSQL in Action: Real-Time Streaming ETL from Oracle Transactional Data
In this post I’m going to show what streaming ETL looks like in practice. We’re replacing batch extracts with event streams, and batch transformation with in-flight transformation. But first, a […]
Building a Real-Time Streaming ETL Pipeline in 20 Minutes
There has been a lot of talk recently that traditional ETL is dead. In the traditional ETL paradigm, data warehouses were king, ETL jobs were batch-driven, everything talked to everything […]