Apache Kafka (AlmaLinux 8)

Software included

Package	Version
Kafka	latest
Docker	3.20.10
Docker compose	2.12.2
containerd.io	1.6.10

Initial start of the service

This application uses Docker to provide faster startup of the adjacency software (Kafka) and start using it directly. This eliminates the need to familiarize yourself with its specifics in terms of configuration and installation. To start working directly with the application, all you need to do is connect your application that will use Apache Kafka by giving it the IP address assigned to your virtual machine and the port on which Kafka is listening (described below, default is 9092).

Quick settings and frequently asked questions

• Logging into the cloud service is done using an SSH key or password that you have specified.
• Docker is installed as recommended by the official docker documentation.
• The standard docker and docker compose commands are now available in this application.
• When the virtual machine is first started, it takes a few minutes before a Docker container containing Kafka is started.
• Kafka data is set to persistent and is located in /opt/kafka/data/.
• By default, Kafka runs on all available interfaces on the machine. Kafka access port is 9092.

Additional Docker settings

Additional Kafka settings that you can change and that are controlled directly by Docker are set as variable values in the configuration file /opt/docker-kafka/docker-compose.yml:

Note: After changing any of the values, you must recreate the Docker container on which the application is running. For more information, please read the Recreating Docker containers section.

• ALLOW_PLAINTEXT_LISTENER: Allow to use the PLAINTEXT listener. Default: no.
• KAFKA_INTER_BROKER_USER: Apache Kafka inter broker communication user. Default: user.
• KAFKA_INTER_BROKER_PASSWORD: Apache Kafka inter broker communication password. Default: bitnami.
• KAFKA_CERTIFICATE_PASSWORD: Password for certificates. No defaults.
• KAFKA_HEAP_OPTS: Apache Kafka's Java Heap size. Default: -Xmx1024m -Xms1024m.
• KAFKA_ZOOKEEPER_PROTOCOL: Authentication protocol for Zookeeper connections. Allowed protocols: PLAINTEXT, SASL, SSL, and SASL_SSL. Defaults: PLAINTEXT.
• KAFKA_ZOOKEEPER_USER: Apache Kafka Zookeeper user for SASL authentication. No defaults.
• KAFKA_ZOOKEEPER_PASSWORD: Apache Kafka Zookeeper user password for SASL authentication. No defaults.
• KAFKA_ZOOKEEPER_TLS_KEYSTORE_PASSWORD: Apache Kafka Zookeeper keystore file password and key password. No defaults.
• KAFKA_ZOOKEEPER_TLS_TRUSTSTORE_PASSWORD: Apache Kafka Zookeeper truststore file password. No defaults.
• KAFKA_ZOOKEEPER_TLS_VERIFY_HOSTNAME: Verify Zookeeper hostname on TLS certificates. Defaults: true.
• KAFKA_ZOOKEEPER_TLS_TYPE: Choose the TLS certificate format to use. Allowed values: JKS, PEM. Defaults: JKS.
• KAFKA_CFG_SASL_ENABLED_MECHANISMS: Allowed mechanism when using SASL either for clients, inter broker, or zookeeper communications. Allowed values: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 or a comma separated combination of those values. Default: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
• KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL: SASL mechanism to use for inter broker communications. No defaults.
• KAFKA_TLS_CLIENT_AUTH: Configures kafka brokers to request client authentication. Allowed values: required, requested, none. Defaults: required.
• KAFKA_TLS_TYPE: Choose the TLS certificate format to use. Allowed values: JKS, PEM. Defaults: JKS.
• KAFKA_CLIENT_USERS: Users that will be created into Zookeeper when using SASL for client communications. Separated by commas. Default: user
• KAFKA_CLIENT_PASSWORDS: Passwords for the users specified atKAFKA_CLIENT_USERS. Separated by commas. Default: bitnami
• KAFKA_CFG_MAX_PARTITION_FETCH_BYTES: The maximum amount of data per-partition the server will return. Default: 1048576
• KAFKA_CFG_MAX_REQUEST_SIZE: The maximum size of a request in bytes. Default: 1048576
• KAFKA_ENABLE_KRAFT: Whether to enable Kafka Raft (KRaft) mode. Default: no
• KAFKA_KRAFT_CLUSTER_ID: Kafka cluster ID when using Kafka Raft (KRaft). No defaults.

Working with Docker

The organization of this application is entirely done using Docker. In this way, we can provide you with ready-made applications faster. In addition, their configuration is more flexible and quite controllable. To be able to control this application, however, you need to know some basic Docker functionalities and features.

Restarting all services

Restarting the database and the Docker container that contains Kafka itself, along with the rest of the software, can be done in two ways:

1. By restarting the entire virtual machine
2. Restarting the Docker containers, which is the faster option. To do this, access your machine via SSH, and run the following commands:

sudo su -
cd /opt/docker-kafka
docker compose restart

Recreating Docker containers

Recreation of a docker container may be necessary if you've changed the configuration inside the docker-compose.yml file. If there are changes, please follow these steps:

cd /opt/docker-kafka/
docker compose stop
docker compose up -d

Checking the status of the Docker application

To check, you need to run the following commands:

sudo su -
docker ps -a

Please pay attention to the STATUS column. It should show how long the service has been running, for example Up X minutes. If the status is Restarting then you need to check what is wrong with the application. In such a case, you can check by following the application logs as follows:

1. Get the application name that you can see in the NAMES column from the output of the docker ps -a command. View the logs with the docker logs <NAME> command, where is the name of the container.

Changing settings

The settings in section Additional Docker settings can be set in the docker-compose.yml file. The path to it is described again in this section. To change a setting, open the configuration file and note the environment section under kafka. Each configuration option must be described on a new line with a dash in front, as in the following example:

    environment:
      - ALLOW_PLAINTEXT_LISTENER=no
      - KAFKA_HEAP_OPTS='-Xmx1024m -Xms1024m'

Please note that the correct indentation (number of spaces) must also be observed, as this is important for yaml configuration files.