Confluent CLI 1.0 is Now Generally Available for Cloud and Platform

Over a year ago, Confluent set out on a mission to improve user experience by empowering developers, operators, and architects with intuitive command line interfaces (CLIs) for managing their Confluent deployments, whether in the cloud or on prem. We engineered the ccloud and confluent CLIs from the ground up to be production-grade tools ready to run on a variety of platforms and to expose a rich set of features that make creating and maintaining an event streaming system as easy as a few commands. Having just reached the v1.0 milestone, with both CLIs declared generally available, we’re taking a few minutes to review some of the main features of the CLIs, as well as highlighting some of the brand new features you may not have explored yet.

Getting started with Confluent’s CLIs

Although the cloud and on-prem CLIs offer different feature sets, they’re built on the same codebase and have a very similar user experience (we’ve found this especially useful for hybrid customers who have to context switch between their on-prem and cloud deployments of the Confluent stack). To start, you can download both CLIs using a one-line terminal command or by downloading a tarball. For example, you can install the Confluent Cloud CLI via:

curl -L --http1.1 https://cnfl.io/ccloud-cli | sh -s -- -b /usr/local/bin

Both CLIs are available and officially supported on Windows, Mac, and Linux. Additionally, the Confluent Platform archives include a confluent CLI for that version of the platform. Hence, if you are an on-prem user, we recommend using the (potentially older) CLI versions bundled with the platform, since the latest bleeding-edge CLI versions may have incompatibilities. In the cloud, you can always update to the latest CLI version by running:

ccloud update

When updates are available, customers will be automatically notified in the CLI output once per day.

Once the CLI is installed, it’s easy to get started working with Apache Kafka^®, Confluent Schema Registry, ksqlDB, Kafka Connect, and other components provided by Confluent. For instance, to create a new Kafka cluster in Confluent Cloud, check out the following demo (which also demonstrates machine-readable output):

Note that all commands in the CLIs follow the same command grammar:

<cli name> <resource> [<sub-resource>] <operation> [arguments and --flags]

And all commands have a --help flag to display usage information. For a great starter tutorial that uses the Cloud CLI to subsequently create API keys and produce to/consume from topics in a new cluster, see this tutorial by our rockstar Developer Relations Team. Note: if you’re already familiar with the basics, we recommend another tutorial that shows the power of the CLI for creating a complete ETL (extract, transform, load) pipeline in the cloud, including using the CLI to interact with Confluent Cloud ksqlDB and fully managed connectors.

Key Cloud features

As hinted above, the Confluent Cloud CLI unlocks a powerful set of features for both the day-to-day activities of a developer as well as managerial functions for operators, architects, and analysts. A sampling of features in the cloud includes:

• Full CRUD operations on Kafka clusters and topics, including the management of clusters across all major cloud providers—Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP)
  • Pro tip: ccloud kafka region list displays all the regions Confluent Cloud currently supports
•  Management of service accounts and their associated API keys and Kafka ACLs
• Creation and control (including pausing or stopping) of connectors, including viewing all connector types that are available to create
• Full CRUD operations for Schema Registry clusters, schemas, and subjects
• Management and switching between environments
• Administration of ksqlDB apps, including automatically configuring the appropriate ACLs

There are also features to make your interaction with the CLI easier (currently limited to bash or zsh shell users), such as autocompletion (ccloud completion --help) and a fully customizable PS1 prompt for adding ccloud information to your terminal (ccloud prompt --help).

Key on-prem features

The on-prem CLI experience is quite similar, though the feature set is tailored towards Confluent Platform deployments (though many such features, like Role-Based Access Control (RBAC), are planned for Confluent Cloud as well). Current features include:

• Rich identity access management (IAM), currently in the form of RBAC and Kafka ACLs, and full management of centralized ACLs, RBAC roles, and RBAC role bindings (assignments of user and group principals to roles)
• File-based secret management for Confluent Platform using envelope encryption
• Cluster metadata discovery
• “Local” commands for experimenting with a development instance of Confluent Platform running locally on your developer machine

New feature: Machine-readable output

A common request we received from users is that the table-formatted “human-readable” output generated by most CLI commands made it difficult to script around the CLIs, e.g., to automatically parse the output of CLI commands from shell or Python scripts in a production environment. In order to address the need for scriptability, commands in Confluent’s CLIs now accept the -o flag, with values such as json, yaml, or human. As an example, we can write this simple bash expression, which continuously ensures that our Kafka cluster hosted in Confluent Cloud is staying up (you can transparently verify we’re upholding our rock-solid 99.95% SLA!):

# Assumes cluster ID is lkc-abc123 (use `ccloud kafka cluster list` to get your cluster’s ID)
watch -n5 "ccloud kafka cluster describe lkc-abc123 -o json | jq '{status}[]' | xargs -I{} echo 'Your Kafka cluster is {}.'"

You can build on these simple ideas, such as querying the CLI to report information back to your company’s internal monitoring, alerting, and dashboard systems.

New feature: Long-lived authentication

Initially, the CLI copied the authentication model of Confluent’s GUI, which meant that all sessions timed out after an hour. However, this prevented the scriptability of the CLIs since it’s unreasonable to ask you to log in again every hour in your production environments. We designed a solution that handles each of the various types of authentication you can do with the CLIs:

• A regular Confluent Cloud account (email and password)
• An SSO-enabled Confluent Cloud account (authentication token returned from IdP)
• An on-prem Confluent Server/metadata service instance (username and password)

In order to achieve consistent long-lived authentication for all of these mechanisms, we’ve added the --save flag to the login command of both ccloud and confluent. When used, the --save flag persists the relevant credentials to your local ~/.netrc file (~/_netrc on Windows). Then, if the CLI ever encounters an authentication error, it attempts to use the stored values to automatically re-authenticate and will silently succeed if matching credentials are found.

We note that, in addition to the above mechanisms, it’s also possible to log into a single Kafka cluster in Confluent Cloud using just a Kafka API key and secret (no Confluent Cloud credentials). The ccloud init command makes this possible and only offers the functionality of consuming from/producing to that cluster.

Closing thoughts and looking ahead

With both machine-readable output and long-lived authentication, Confluent’s CLIs are now able to be used in production environments for both interactive and automated use cases. Moreover, with support across a variety of operating systems and platform versions, the CLIs are a robust, reliable, developer-friendly way to interact with on-prem, cloud, or hybrid Confluent deployments. The CLIs are mature and stable enough to declare the v1.0 milestone; however, the CLIs are under active development, and the CLI Team has a number of exciting new features on the roadmap in the coming months. For example:

• Integrating data from the forthcoming Confluent Cloud Metrics API into ccloud
• Supporting RBAC management in the cloud, not just on prem, for fine-grained permissions management
• Adding additional on-prem cluster discovery features to confluent

We’re excited for these features—and more we can’t announce yet! Keep an eye out for new features or updates released every week or two in the cloud, as well as with each new Confluent Platform release on prem.

Get in touch

The CLI Team is highly responsive to user feedback and is interested in hearing about both bug reports and feature requests—any way to help us serve our users better. If you would like to provide any suggestions, or if you’d just like to give some feedback, please feel free to file a ticket through Confluent Support, reach out on the #confluent-cli Slack channel, or send us an email at cli-team@confluent.io. We look forward to hearing from you!