Meet the Oracle XStream CDC Source Connector

Organizations today face challenges keeping their various data systems updated with fresh data from operational stores. While many companies rely on Oracle® databases for storing mission-critical customer, transaction, and business data, extracting real-time data at scale remains difficult, and results in missed opportunities to take timely action.

Today, we're excited to announce our Oracle XStream CDC Source Connector for both Confluent Cloud and Confluent Platform, with general availability coming soon. This connector represents a significant advancement in capturing Oracle database changes, delivering enterprise-grade performance, reliability, and cost-effectiveness for modern data streaming requirements.

With the Oracle XStream CDC Source Connector, you can:

• Achieve high performance and reliable streaming of change events, delivering 2-3x improvement in throughput and latency

• Reduce total cost of ownership with simplified licensing and lower operational overhead

• Unlock real-time operational data to create reusable data products for downstream systems

Be sure to tune in to our upcoming webinar to see the connector in action.

Sound interesting? Well, there’s even more to unpack! In this blog, we'll dive deeper into the challenges of real-time Oracle data streaming, and explore how our new connector addresses these pain points. We'll examine the technical foundations of XStream technology, walk through deployment steps with a technical demo, and provide implementation guidance. Continue reading to discover how you can achieve high-performance streaming with Oracle XStream CDC.

CDC and Oracle®'s XStream Technology: Purpose-Built for Real-Time Streaming

Change data capture (CDC) addresses the challenge of keeping various systems updated by efficiently identifying and capturing data that has been added to, updated, or removed. By streaming changes from Oracle tables to a data streaming platform, these change events can be written as Apache Kafka® messages, and made available to data lakes, data warehouses, or event-driven applications, with minimal latency. In 2021, we released our Oracle CDC Source Connector based on LogMiner technology, which provided a robust foundation for reliable database replication and change data capture implementation.

However, as data volumes grew and organizations expanded their real-time use cases with higher performance requirements, existing solutions faced significant limitations in certain scenarios. To overcome these performance hurdles, Oracle developed XStream technology, specifically designed for high-throughput, low-latency operations.

XStream technology offers three key advantages:

• Purpose-built for real-time streaming use cases: The XStream Out server pushes changes in real-time to the connector with minimal latency, due to improved redo log querying

• High-performance design for mission-critical transactions: Flexible deployment options and optimized memory management enable best-in-class throughput and latency for both high-volume and long transactions

• Rich productivity features with continuous Oracle investments: There are advanced capabilities beyond basic CDC features, including downstream capture, support for various data types, and native integration with Oracle Real Application Clusters (RAC)

Unlock High-Performance Data Streaming From Oracle® Databases

Confluent's new Oracle connector leverages the XStream technology to provide high-performance, reliable change data capture, while also tackling the key performance limitations that the old technology couldn’t handle. Thanks to our partnership with Oracle, we’re able to deliver significant cost savings compared to traditional licensing, along with pre-built integrations for modern data systems. This connector represents a huge advancement in how organizations can stream Oracle data changes more efficiently.

Achieve High Performance and Reliable Streaming

Traditional CDC approaches work well for many use cases. But they can hit throughput limitations during long or large transactions, which can be a challenge for advanced implementations. As data volumes continue to grow, and more complex processing requirements emerge, you need a solution that can scale as your enterprise workloads expand, without compromising performance.

Our new connector solves these scalability challenges by delivering 2-3x improvement in throughput and latency as compared to traditional CDC approaches, enabling true, real-time data processing at scale. XStream Out technology provides high-performance, multi-threaded capture, optimized for large or long transactions.

It also comes with built-in checkpointing and state management for automated recovery, and native RAC support to seamlessly integrate with Oracle RAC architecture. If your team is hitting the ceiling with conventional CDC solutions, this connector removes your bottlenecks, ensuring your real-time data pipelines can keep pace with even the most demanding enterprise requirements.

Let’s take a look under the hood at how this works with Apache Kafka®:

(see full-size image)

What makes Oracle XStream technology so powerful is that it’s purpose-built for CDC Streaming:

• Direct integration with Oracle DB: XStream Out Server is deployed as a service in Oracle DB to capture changes from database instances.

• Efficient change capture: Any changes in the database are captured by the outbound server and sent to the connector task as logical change records (LCRs).

• Optimized handling of large transactions: To handle large volumes or lengthy transactions, XStream leverages disk storage, avoiding memory exhaustion. It then accesses data from disk with minimal latency, maintaining high transaction throughput.

• Efficient topic routing: The connector task creates a change event queue and distributes these changes to individual Kafka topics, with schema changes automatically tracked in a history topic.

This architecture enables high-throughput data capture with minimal latency, optimized for both high-volume and long-running transactions, without requiring a separate redo log topic.

Reduce Total Cost of Ownership

When it comes to implementing robust CDC solutions for Oracle databases, many organizations face a major roadblock: cost. Existing Oracle CDC solutions come with hefty licensing fees that can easily climb into the millions each year. This financial burden often forces teams to scale back their real-time data initiatives or settle for less fresh data.

Our new connector changes the game by making CDC more affordable. First, using this connector doesn’t require a separate Oracle Golden Gate license. Instead, the cost of leveraging XStream technology is built into our connector pricing—something you won’t find with other XStream-powered CDC connectors. Second, the connector provides enterprise-ready features out of the box, including state management, trusted user model security, and performance monitoring.

These capabilities help save you the time and effort. And if you choose our fully managed version of the connector, you don’t have to worry about the ongoing operational burden of managing Kafka Connect infrastructure and the connector, freeing up your development time to focus on more important initiatives.

By reducing the total cost of ownership, we’re helping organizations unlock the full potential of their Oracle data in real-time applications.

Unlock Real-Time Operational Data

Since Oracle stores critical customer and transaction data, there’s likely an ever-growing list of data systems and applications waiting for Oracle change data, to stay up to date. But processing and governing data only after it reaches downstream systems leads to data inconsistencies, duplicated processing efforts, and increased costs that compound throughout the pipeline, limiting your ability to use the data effectively.

Using our Oracle XStream CDC Source Connector enables you to unlock operational data from a heavyweight database and transform the change events into reusable data products before landing in downstream systems. How? With Schema Registry, you can enforce data contracts to ensure high data quality, while Apache Flink® processes real-time data as it’s produced.

The connector allows you to capture Oracle table changes into separate Kafka topics for different lines of business, and you could use stream processing to join, filter, and enrich data from various Oracle tables or other data sources. Once your data products are ready, you can easily push updates downstream using sink connectors for various modern data systems and apps, such as AWS S3, MongoDB Atlas, and Google BigQuery, or use this optimized and contextual data for your event-driven applications.

(see full-size image)

Ultimately, this transforms your "data mess" into reusable data products that deliver instant value across your organization—enabling real-time intelligence and operational capabilities through custom applications and microservices that weren't previously possible.

Demo: Configuring the Oracle® XStream CDC Connector

The Oracle XStream CDC Connector is designed for straightforward implementation while delivering powerful capabilities. Our technical demo walks you through a comprehensive setup, but here are some key deployment steps while setting up the connector:

Step 1: Setting Up the Database (Prerequisites)

Before the connector can be deployed, certain prerequisites must be met within the database to ensure compatibility and proper functionality. The steps for this preparation are:

1. Create user accounts:

   1. The XStream Administrator (sysdba): This account is essential for initiating and managing the capturing processes. It's a high-privilege account, but its utilization is strictly confined to activities within the source database.

   2. The XStream User: This role is designated for interaction with the XStream API. It's configured to have lesser access, focusing only on consuming the output provided by the XStream API. This ensures a minimal yet sufficient access level for the connector's operation, aligning with best security practices.

2. Activate XStream: This step involves executing a specific command designed to enable XStream functionality within the Oracle database. XStream plays a crucial role in facilitating real-time data changes capture and propagation.

3. Configure Archivelog mode: Archivelog mode is essential for the comprehensive logging of all transactions occurring within the database. This mode provides the foundational data necessary for XStream's change data capture (CDC) functionality.

4. Implement supplemental logging: Supplemental logging is an additional layer of data detail logging. It captures extended information not tracked by default, ensuring that all necessary context for data changes is available for accurate replication and analytics.

5. Set up the XStream Out instance: The final preparatory step involves configuring the XStream Out instance appropriately. This setup is critical for establishing a pipeline that streams database changes outwards, ready to be consumed by the Oracle XStream CDC Source connector.

   ‎ 

Step 2: Preparing the Connect Worker Node (Confluent Platform Only)

For Confluent Platform deployments, your environment needs specific preparation. Follow these steps:

1. Install Oracle Instant Client: This client software provides the necessary functionality for the connect worker node to communicate efficiently with the Oracle database, ensuring stable and robust connectivity.

2. Install required packages: To facilitate optimal operation of the connector, various dependent packages and libraries may need to be installed. These dependencies are critical for the execution environment, providing the necessary support for the connector's functionality.

   ‎ 

Step 3: Deploying the Connector

The deployment process follows standard connector practices with key Oracle-specific considerations:

1. Acknowledgement of support limitations: It's important to understand that the connector is designed to support a one-to-one relationship with the pluggable databases (PDBs) on Oracle. This means that each connector task can manage data capture for only one specific PDB.

2. Familiar deployment experience: The deployment process aligns with the standard practices established for other connectors. Users are offered multiple methods for deployment, including through a User Interface (UI), Command Line Interface (CLI), or Application Programming Interfaces (APIs).

3. Comprehensive deployment steps:

   1. Configure Kafka authentication: This step involves setting up a secure authentication mechanism that governs the communication between the source database and the connector, ensuring data security.

   2. Configure connection with the source database: For the connector to access the database changes, proper credentials for the XStream user must be shared. This ensures that the connection is authenticated and authorized.

   3. Configure task and properties: This involves a detailed setup of the connector task, specifying how the connector should behave, including which data to capture and how to process it. Configuring properties tailors the connector's operation to meet specific data pipeline requirements.

   4. Set up single message transforms (SMTs): If data transformation or manipulation is needed before the data reaches its destination, SMTs can be configured. This step allows the customization of data as it flows through the connector, enhancing flexibility and utility.

   5. Finalize deployment of the connector: Completing the outlined steps finalizes the connector deployment, making it operational and ready to stream change data from the Oracle database.

Ready to Stream Oracle® Data?

In this blog, we introduced our new Oracle XStream CDC Source Connector, available both on Confluent Cloud and Platform, which is designed to tackle the challenges organizations face when keeping various systems updated with fresh data from Oracle databases. We explored how XStream technology provides significant advantages over traditional CDC approaches, delivering 2-3x improvements in throughput and latency at a significantly lower price point.

Here are two ways you can get started.

First, register for the upcoming Oracle XStream Connector webinar to get hands-on with a technical demo that showcases the full capabilities of the connector.

Second, try the connector yourself! Sign up for a free trial of Confluent Cloud and start using it within a matter of minutes. And with the promo code CCBLOG60, you’ll get an additional $60 of free Confluent Cloud usage*.

Oracle® and XStream are either registered or unregistered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Apache®, Apache Kafka®, Kafka®, Apache Flink®, and Flink® are registered trademarks of the Apache Software foundation.