[Webinar] How to Protect Sensitive Data with CSFLE | Register Today
Containerized Testing with Kerberos and SSH
Kerberos authentication is widely used in today’s client/server applications; however getting started with Kerberos may be a daunting task if you don’t have prior experience. Information on setting up Kerberos with an SSH server and client on the web is fragmented and hasn’t been presented in a comprehensive end-to-end way on a simple local setup.
At Confluent, several of our connectors for Apache Kafka® support Kerberos-based authentication. For development and testing of these connectors, we often leverage containers due to their fast, iterative benefits. This tutorial aims to provide a simple setup for a Kerberos test environment with SSH for a passwordless authentication that uses Kerberos tickets. You may use this as a guide for testing the Kerberos functionality of SSH-based client-server applications in a local environment or as a hands-on tutorial if you’re new to Kerberos. To understand the basics of Kerberos before diving into this tutorial, you may find this video helpful. Additionally, if you are looking for a non-SSH-based setup, the setup below for the KDC server container may also be useful.
The setup will consist of three parts:
- Client: a client SSH application that authenticates via Kerberos to the SSH server
- KDC server: a local KDC server in a container
- Server: an SSH server with a user
The software packages used in this tutorial are OpenSSH Server and MIT Kerberos. The configuration may be different for other package distributions. The following steps are performed on a Mac, but a few tweaks may be needed for other platforms.
We will explore two ways for the client to be set up: (1) as a client Docker container and (2) as a local client on your host machine. The latter is useful for conveniently testing applications from a local run environment, such as running from an IDE without having to configure a client Docker container.
The container images for this tutorial have been pre-configured with the required package dependencies, but Kerberos needs to be set up manually. This tutorial explains how to configure the containers in the following way:
- KDC server: We will configure the hosts file for a correct Kerberos lookup in a Docker environment, as well as adding the user and service principals
- SSH server: We will review how to configure the SSH server to support Kerberos authentication and add a user and the server Kerberos sshserver.keytab file
- SSH client: We will review how to configure the client to use Kerberos tickets as part of the SSH authentication and how to configure the Kerberos client to point to the correct KDC in the client krb5.conf file
Approach 1: Triple container setup with the container client
As a first step, start all three required containers. To start the KDC server, use the provided kdc-server image with docker-compose up from the subdirectory. The KDC server container also maps the localhost ports 749 and 88 in case you would like to perform testing or operations on the KDC from your local machine. To start the SSH client and server, simply call docker-compose up again from the ssh-container subdirectory. A successful start of the containers should resemble the below output on docker ps.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8845aa0f05cf kdc-server_kdc-server "/bin/sh -c /tmp/ini…" 3 seconds ago Up 3 seconds 0.0.0.0:88->88/udp, 0.0.0.0:749->749/tcp kdc-server_kdc-server_1 8fb55d49237a ssh-container_ssh-server "/usr/sbin/sshd -D" 4 seconds ago Up 4 seconds 0.0.0.0:2222->22/tcp ssh-container_ssh-server_1 e154b902e3de ssh-container_ssh-client "/bin/bash" 14 minutes ago Up 4 seconds 22/tcp ssh-container_ssh-client_1
KDC
Once the KDC container is running, switch into the container’s bash session.
docker exec -it kdc-server_kdc-server_1 /bin/bash
From within the container, start the kadmin shell using the command kadmin.local.
If you list the principals, you can see the defaults as below:
kadmin.local: listprincs K/M@EXAMPLE.COM kadmin/admin@EXAMPLE.COM kadmin/changepw@EXAMPLE.COM kadmin/kdc-server@EXAMPLE.COM krbtgt/EXAMPLE.COM@EXAMPLE.COM noPermissions@EXAMPLE.COM
Here, the kadmin/admin@EXAMPLE.COM and noPermissions@EXAMPLE.COM were added by the init script of the container and may be used for KDC admin testing purposes. kadmin/kdc-server@EXAMPLE.COM is the principal for the Docker container and is added automatically. You can see that the host of the principal is the same as the host name of the container. The rest of the principals are created by default and are not used in the rest of this tutorial.
Using kadmin, we will now add the principals for the SSH server and client.
In order to add the principal for the SSH server, we first need to identify the host name of the SSH server container. In our case, this will be ssh-server. If you are using your own container, it may be the same as the container ID.
kadmin.local: addprinc -randkey host/ssh-server@EXAMPLE.COM WARNING: no policy specified for host/ssh-server@EXAMPLE.COM; defaulting to no policy Principal "host/ssh-server@EXAMPLE.COM" created.
Next, we will create the keytab file for the service principal that will need to be placed on the SSH server container.
kadmin.local: ktadd -k /sshserver.keytab host/ssh-server@EXAMPLE.COM Entry for principal host/ssh-server@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/sshserver.keytab.
The command above will save the keytab file as sshserver.keytab in the root of the KDC server container.
Now we will add the principal for the (future) user on the SSH server. The username here will need to be identical to the username on the SSH server. Here, we will simply use sshuser as the username.
Method 1: Keytab
This method will save the Kerberos key of the user principal in the same way as above for the host and will need to be transferred to the SSH client.
kadmin.local: addprinc -randkey sshuser WARNING: no policy specified for sshuser@EXAMPLE.COM; defaulting to no policy Principal "sshuser@EXAMPLE.COM" created. kadmin.local: ktadd -k /sshuser.keytab sshuser@EXAMPLE.COM Entry for principal sshuser@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/sshuser.keytab.
Method 2: Password
This method will assign a password to the Kerberos user principal and will require that you specify the password instead of a keytab file on the SSH client when initializing using kinit.
kadmin.local: addprinc sshuser WARNING: no policy specified for sshuser@EXAMPLE.COM; defaulting to no policy Enter password for principal "sshuser@EXAMPLE.COM": Re-enter password for principal "sshuser@EXAMPLE.COM": Principal "sshuser@EXAMPLE.COM" created.
After adding the principals, the results of the listprincs command should look like what you see below:
kadmin.local: listprincs K/M@EXAMPLE.COM host/ssh-server@EXAMPLE.COM kadmin/admin@EXAMPLE.COM kadmin/changepw@EXAMPLE.COM kadmin/kdc-server@EXAMPLE.COM krbtgt/EXAMPLE.COM@EXAMPLE.COM noPermissions@EXAMPLE.COM sshuser@EXAMPLE.COM
Once the service and user principals are created, copy the keytab file(s) from the KDC container to your local machine. The sshserver.keytab will need to be copied to the SSH server, and if you are using the keytab Method 1 above, the sshuser.keytab to the SSH client.
docker cp kdc-server_kdc-server_1:/sshserver.keytab ./sshserver.keytab docker cp kdc-server_kdc-server_1:/sshuser.keytab ./sshuser.keytab
Final step
To make sure the KDC is able to look up the SSH server container, we will need to create an entry in the /etc/hosts file on the KDC container. We will need to find the IP address of the SSH server container, which can be done from the SSH container or using the handy command below:
docker inspect -f '{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq)
/kdc-server_kdc-server_1 - 172.21.0.2 /ssh-container_ssh-server_1 - 172.31.0.3 /ssh-container_ssh-client_1 - 172.31.0.2
We see that our SSH server has the IP 172.31.0.3.
Now all we need to do is add the entry that maps the SSH server hostname to its IP address in /ets/hosts on the KDC server. Note that if you are using your own container, the host name may be different, but the default case is the same as the container ID. The KDC container has Vim installed for convenience. After the changes, the /etc/hosts file should look similar to the one below, where the last entry is the SSH container:
127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 172.21.0.2 kdc-server 172.31.0.3 ssh-server
SSH server
The setup for the SSH server is simpler than the KDC server. Keep in mind that the file sshd_config is responsible for SSH server configurations (note the extra letter d), while ssh_config contains the configurations for the SSH client.
In order to enable the SSH server to accept Kerberos tickets as part of the authentication, we must enable the following configurations in /etc/ssh/sshd_config.
GSSAPIAuthentication yes GSSAPICleanupCredentials yes
These configurations are automatically turned on in the ssh-server container based on the sshd_config in the ssh-container subdirectory, but on other systems, they may need to be added or uncommented manually. Keep in mind that for changes in sshd_config to take effect, you will need to reload the SSH server. The relevant configurations for ticket-based authentication will be the ones starting with GSSAPI*. For more on the configurations, refer to the manual. Note that the configuration #KerberosAuthentication no remains off as this would require the re-entry of our Kerberos password and would not allow using only existing tickets, which would impede our passwordless authentication goal.
Now we will configure the /etc/krb5.conf file on the SSH server container to point to the correct KDC and allow for forwardable tickets. If you are using the container from this tutorial, you will need to overwrite the existing krb5.conf file that was created by the krb5-user package. You can switch into the container using the command below:
docker exec -it ssh-container_ssh-server_1 /bin/bash
In a real-world scenario, the IP address of the KDC server would be used instead of the host.docker.internal address. We’ll use the host.docker.internal address because the KDC container’s ports are exposed on the host machine.
[libdefaults]
default_realm = EXAMPLE.COM
forwardable = TRUE
[realms]
EXAMPLE.COM = {
kdc_ports = 88
kadmind_port = 749
kdc = host.docker.internal
admin_server = host.docker.internal
}
[domain_realm]
host.docker.internal = EXAMPLE.COM
The host.docker.internal host name is specific to Docker on Mac and Windows. If you are using Linux, you will need to configure your /etc/hosts file or apply an alternate workaround to get the setup working.
Next, we will copy the sshserver.keytab that we created earlier into the SSH server container under the path /etc/krb5.keytab. Note the name change of the file—this is so that the SSH server can find the Kerberos host keytab on the default path that it checks for.
docker cp ./sshserver.keytab ssh-container_ssh-server_1:/etc/krb5.keytab
Lastly, we will create the user sshuser on the SSH server:
root@ssh-server:/# adduser sshuser Adding user `sshuser' ... Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully Changing the user information for sshuser Enter the new value, or press ENTER for the default Full Name []: test Room Number []: Work Phone []: Home Phone []: Other []: Is the information correct? [Y/n] Y
As a sanity check, we can try to SSH into the new user account with the password that was created above. This is also the same password that is required to SSH from another client when Kerberos ticket authentication is not used.
root@ssh-server:/# ssh sshuser@localhost The authenticity of host 'localhost (127.0.0.1)' can't be established. ECDSA key fingerprint is ad:48:a6:ca:20:b4:97:12:40:78:2f:52:15:47:12:d8. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts. sshuser@localhost's password: ... sshuser@ssh-server:~$
The SSH server is now ready to accept a ticket-based login for the user ssshuser. If for any reason you would like to test SSH from your host machine, you may use ssh sshuser@localhost -p 2222, as the container maps the port 2222 by default to the host machine.
SSH client
Let’s bring it all together. Now that we have the KDC server and the SSH server setup, the only remaining element is the client. First, let’s switch into the SSH client container:
docker exec -it ssh-container_ssh-client_1 /bin/bash
Configure /etc/hosts by adding the mapping for the SSH server container’s host name and IP.
172.31.0.3 ssh-server
As a check, we will attempt to SSH directly to the server with the username and password. For this, we will use the host name of the server container, in our case, simply ssh-server:
root@ssh-client:/# ssh sshuser@ssh-server sshuser@ssh-server's password: ... sshuser@ssh-server:~$
This confirms that the user was set up correctly and that our SSH connection to the server works properly. We can exit from the session back to the container shell.
The important configurations for our use case in the SSH client config /etc/ssh/ssh_config start with GSSAPI*. For our tutorial, we will need minimal configuration.
GSSAPIAuthentication yes
The configuration above is configured for the ssh-client container automatically during the build. For the other GSSAPI* configurations applicable in the SSH client, please refer to the documentation.
We will now configure the /etc/krb5.conf file used by the client to establish communication with the KDC server. If you are using the container from the tutorial, you will (again) need to overwrite the existing krb5.conf file.
[libdefaults]
default_realm = EXAMPLE.COM
forwardable = TRUE
[realms]
EXAMPLE.COM = {
kdc_ports = 88
kadmind_port = 749
kdc = host.docker.internal
admin_server = host.docker.internal
}
[domain_realm]
host.docker.internal = EXAMPLE.COM
If you used Method 1 for the sshuser principal, you may now copy the sshuser.keytab to the SSH client container:
docker cp ./sshuser.keytab ssh-container_ssh-client_1:/sshuser.keytab
We can now initialize our Kerberos ticket using kinit.
If you used a keytab for the user principal, the command will look like this:
root@ssh-client:/# kinit sshuser -k -t sshuser.keytab
If you used a password (Method 2), the command will prompt you for the Kerberos password that you have configured on the KDC. Note that this password is different from the user password on the SSH server.
root@ssh-client:/# kinit sshuser Password for sshuser@EXAMPLE.COM:
If the above kinit command returns without connection errors, this means that the client was able to reach the KDC successfully based on the krb5.conf settings. After initializing the TGT ticket for the user principal, you can verify if the ticket was cached correctly using klist.
root@ssh-client:/# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: sshuser@EXAMPLE.COM
Valid starting Expires Service principal 01/03/21 05:21:10 01/04/21 05:21:06 krbtgt/EXAMPLE.COM@EXAMPLE.COM
Finally, we can now connect to our SSH server using Kerberos tickets and without needing a password.
root@ssh-client:/# ssh sshuser@ssh-server
The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Sun Jan 3 05:20:33 2021 from ssh-container_ssh-client_1.ssh-container_default sshuser@ssh-server:~$
Approach 2: Single-container setup with a local client
This container setup combines the KDC and SSH server into one container. It enables you to use your host machine as the SSH client and test applications from your local environment. To start the container, simply use docker-compose up from the kdc-ssh-server directory.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 42dbfa15c4b4 kdc-ssh-server_kdc-ssh-server "/bin/sh -c /tmp/ini…" 53 seconds ago Up 5 seconds 88/tcp, 0.0.0.0:88->88/udp, 0.0.0.0:749->749/tcp, 0.0.0.0:2222->22/tcp kdc-ssh-server_kdc-ssh-server_1
KDC in the combined container
This KDC setup is similar to part 1 of the tutorial but differs in some of the key steps. To switch into the container, use the following command:
docker exec -it kdc-ssh-server_kdc-ssh-server_1 /bin/bash
Initially, the KDC principals should be the ones listed below:
kadmin.local: listprincs K/M@EXAMPLE.COM kadmin/admin@EXAMPLE.COM kadmin/changepw@EXAMPLE.COM kadmin/kdc-ssh-server@EXAMPLE.COM krbtgt/EXAMPLE.COM@EXAMPLE.COM noPermissions@EXAMPLE.COM
We need to add a service principal for ssh-server, along with a keytab, which will be saved in the default /etc/krb5.keytab location. Note that this step differs from part 1 of the tutorial in the keytab location. Since the SSH server will be running on the same container, the keytab can be saved directly to the default location.
kadmin.local: addprinc -randkey host/ssh-server@EXAMPLE.COM WARNING: no policy specified for host/ssh-server@EXAMPLE.COM; defaulting to no policy Principal "host/ssh-server@EXAMPLE.COM" created. kadmin.local: ktadd -k /etc/krb5.keytab host/ssh-server@EXAMPLE.COM Entry for principal host/ssh-server@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/krb5.keytab.
You can create the sshuser principal in the same way that we did in part 1. After creating the principals, the list should look like this:
kadmin.local: listprincs K/M@EXAMPLE.COM host/ssh-server@EXAMPLE.COM kadmin/admin@EXAMPLE.COM kadmin/changepw@EXAMPLE.COM kadmin/kdc-ssh-server@EXAMPLE.COM krbtgt/EXAMPLE.COM@EXAMPLE.COM noPermissions@EXAMPLE.COM sshuser@EXAMPLE.COM
Next, we need to edit the /etc/hosts file. On the first line, add a mapping for the container’s IP to the ssh-server host name (the same IP as the last mapping). The SSH command from the host machine will use the ssh-server host name, and this mapping will enable the KDC to look up and find the SSH server on the same container.
172.22.0.2 ssh-server 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 172.22.0.2 kdc-ssh-server
SSH server in the combined container
In this version of the container, the build and init scripts take care of the krb5.conf configuration and the sshd_config. We only need to create the user and make sure the host keytab that we created in the KDC step is under /etc/krb5.keytab. We can create the user the same way that we did in part 1.
Local SSH client
This part may require some system-specific configuration. Make sure that your system has a Kerberos client and SSH client installed. Your setup should be ready if the commands kinit and ssh work.
Add an entry to your local /etc/hosts file. The entry will map the localhost’s IP address 127.0.0.1 to the ssh-server host name.
127.0.0.1 ssh-server
To test if the user was created successfully earlier and that the container’s SSH connection is open, you can try to SSH from your host machine into the container.
The exposed container port for the SSH server is 2222, so the command will look like this:
ssh sshuser@ssh-server -p 2222 sshuser@localhost's password: ... sshuser@kdc-ssh-server:~$
Let’s configure the SSH client to attempt ticket-based authentication. This may be configured in either /etc/ssh/ssh_config globally or in ~/.ssh/config for a specific user. The key line to add is the one below, which enables the client to use the tickets. There may be additional GSSAPI* options that you may want to configure depending on your preferences. For our example, we only use the minimum requirement.
GSSAPIAuthentication yes
Now we need to configure the /etc/krb5.conf to point to the KDC at the localhost. This configuration will be similar to that in part 1, but the IP address used will be for the host machine’s localhost.
[libdefaults]
default_realm = EXAMPLE.COM
forwardable = TRUE
[realms]
EXAMPLE.COM = {
kdc_ports = 88
kadmind_port = 749
kdc = 127.0.0.1
admin_server = 127.0.0.1
}
[domain_realm]
127.0.0.1 = EXAMPLE.COM
Initialize the TGT with kinit. Depending on whether you created a keytab or password for the user principal, you will need to use the correct method for kinit, which is explained in part 1 under the SSH client section.
kinit sshuser sshuser@EXAMPLE.COM's password:
A klist command should provide the result below:
Credentials cache: API:CC2F710D-E71E-4F56-A1B0-75144E3D0015
Principal: sshuser@EXAMPLE.COM
Issued Expires Principal
Jan 4 01:59:42 2021 Jan 4 11:59:37 2021 krbtgt/EXAMPLE.COM@EXAMPLE.COM
Finally, we can SSH to the server using the Kerberos ticket and without a password.
ssh sshuser@localhost -p 2222
The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Mon Jan 4 01:20:15 2021 from 172.22.0.1 sshuser@kdc-ssh-server:~$
Helpful notes for troubleshooting
If you currently have a cached ticket and it’s interfering with the Kerberos authentication, or if you made a mistake earlier, you may remove it using the command below at your own risk:
kdestroy -A
If you’re having issues at the SSH stage or would like to see the process of ticket authentication, you can get a verbose output from the SSH client command:
ssh sshuser@ssh-server -vvv
A successful Kerberos authentication will resemble the following:
debug3: authmethod_is_enabled gssapi-with-mic debug1: Next authentication method: gssapi-with-mic debug3: send packet: type 50 debug2: we sent a gssapi-with-mic packet, wait for reply debug3: receive packet: type 60 debug3: send packet: type 61 debug3: receive packet: type 61 debug3: send packet: type 66 debug3: receive packet: type 52 debug1: Authentication succeeded (gssapi-with-mic).
A failed authentication will look similar to the one below—in some cases, stating the cause of the error, though in others, the issue may be less obvious. In general, issues such as the one below arise from a misstep during configuration. In this case, the SSH server’s IP was mapped incorrectly on the KDC server in /etc/hosts.
debug1: Next authentication method: gssapi-with-mic debug3: send packet: type 50 debug2: we sent a gssapi-with-mic packet, wait for reply debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug1: An invalid name was supplied unknown mech-code 0 for mech 1 2 752 43 14 2debug1: Miscellaneous failure (see text) unknown mech-code 0 for mech 1 3 6 1 5 5 14
debug1: Miscellaneous failure (see text) unknown mech-code 2 for mech 1 3 6 1 4 1 311 2 2 10
For a verbose output from kinit, you may use the command below. This may be helpful when you have trouble reaching the KDC or looking up the user principal.
kinit sshuser -k -t sshuser.keytab -V
If you encounter the unable to reach KDC error below with tried 1 KDC, the issue may be a misconfigured KDC container. In this case, you should verify the /etc/hosts configuration. If nothing helps, you can try to stop and remove the container completely and rebuild it from scratch.
kinit sshuser sshuser@EXAMPLE.COM's password: kinit: krb5_get_init_creds: unable to reach any KDC in realm EXAMPLE.COM, tried 1 KDC
Next steps
Now that you have a Kerberos setup and an understanding of the basic concepts, we encourage you to try our Elasticsearch, SFTP, and Cassandra connectors, all with support for Kerberos authentication.
Special thanks to Sajana Weerawardhena for providing resources and help during my Kerberos ramp up.
Did you like this blog post? Share it now
Subscribe to the Confluent blog
Introducing Apache Kafka® 3.9
We are proud to announce the release of Apache Kafka 3.9.0. This is a major release, the final one in the 3.x line. This will also be the final major release to feature the deprecated Apache ZooKeeper® mode. Starting in 4.0 and later, Kafka will always run without ZooKeeper.
Preparing the Consumer Fetch: Kafka Producer and Consumer Internals, Part 3
In this third installment of a blog series examining Kafka Producer and Consumer Internals, we switch our attention to Kafka consumer clients, examining how consumers interact with brokers, coordinate their partitions, and send requests to read data from Kafka topics.