Why Confluent over MSK for Streaming Data Pipelines at Security Scorecard

When a business wants to understand their credit risk, they consult one of three major credit scoring companies. When companies want to see the security risks of their 1st party and 3rd party systems in their data infrastructure, they turn to SecurityScorecard. Accurate, up-to-date data is the lifeblood of any modern business, and it needs to come from myriad sources. SecurityScorecard continuously monitors and rates the cyber posture of more than 12 million companies worldwide.

Up until a few years back, the teams at SecurityScorecard relied on batch pipelines to push data to and from Amazon S3 as well as expensive REST API based communications to carry data between systems. They also spent significant time and resources on Kafka upgrades and cluster management initially using Amazon MSK before turning to Confluent’s fully managed solution.

SecurityScorecard’s Brandon Brown, lead of the company’s Pipelines team, pivoted away from antiquated batch processing and toward Confluent’s streaming approach for the development of a new Automatic Vendor Detection (AVD) product. His team continues to leverage the full breadth of Confluent Cloud’s connectors to deliver several new use cases, including:

1. AVD to onboard new sources of data quickly via topics and refresh that data every hour
2. A source of truth using PostgreSQL and pushing data to downstream systems using CDC connectivity
3. A future use case using ksqlDB to have consolidated observability, and save on infrastructure costs

Join us and hear the reasons why SecurityScorecard switched to Confluent.