Integrating Kafka with Google Cloud Pub/Sub

Tudip

17 April 2020

What is Kafka?

In order to understand Kafka, first we need to understand the publish-subscribe system. In the publish-subscribe system the messages which need to be sent are persisted in a topic. Unlike the point-to-point system, consumers can subscribe to one or more topics and consume all the messages in that topic. For example: Dish TV. Apache Kafka is a distributed publish-subscribe messaging system which is capable of handling a high volume of data and it works for both offline and online messages. It is highly reliable, scalable and very durable as well. It guarantees zero data loss as well.

What is Cloud Pub/Sub?

Cloud Pub/Sub is a fully managed messaging system, which is a Google Cloud service which deals with sending and receiving messages through independent applications.

Need of integration

Although, Kafka has provided a huge help with coming together with Google cloud as Kafka can be used with the Google cloud tools very easily to achieve the desired result and Kafka, being a better messaging system then Cloud pub/sub, it provides more options as well but still the use of Cloud Pub/Sub in various application running in Google cloud becomes more convenient as managing Kafka can become difficult sometimes although Could Pub/sub is fully managed. Also, Cloud Pub/sub can easily be better integrated with other Google Cloud services.

Integration

In order to integrate Kafka with Cloud Pub/sub first we need to launch Kafka through the Cloud marketplace. We simply need to search Kafka in Cloud Marketplace and then click on launch. We can customize the setting by simply selecting the zone which is close to us. Once it is deployed a VM would automatically be created in the console for Kafka.
Then we need to configure the VM so that it could be integrated with Cloud Pub/Sub topic which we will be creating later. The configuration part is done through the SSH client for the VM.

After this we will set up the cloud Pub/sub topic and subscription in the Cloud shell as it is a Google Cloud service, so that Kafka can be integrated with it.

Now we will set up the connector file and start the connector service in the SSH client for Kafka so that the integration can take place. After that we will open another window of the SSH client for Kafka and initiate the Kafka console. In the Kafka console we will enter the data elements.

These message elements would display in Cloud shell where the Cloud Pub/Sub is set up.

This way, Kafka is integrated with the Cloud Pub/Sub.