Kafka Connect

Kafka Connect - Learn How to Source Twitter Data, Store in Apache Kafka Topics & Sink in ElasticSearch and PostgreSQL

Last updated 2022-01-10 | 4.5

- Configure and run Apache Kafka Source and Sink Connectors
- Learn concepts behind Kafka Connect & the Kafka Connect architecture
- Launch a Kafka Connect Cluster using Docker Compose

What you'll learn

Configure and run Apache Kafka Source and Sink Connectors
Learn concepts behind Kafka Connect & the Kafka Connect architecture
Launch a Kafka Connect Cluster using Docker Compose
Deploy Kafka Connectors in Standalone and Distributed Mode
Write your own Kafka Connector

* Requirements

* Good understanding of Kafka Basic is REQUIRED to take this course
* A recent Windows / Mac / Linux machine with minimum 4GB of RAM
* 5 GB of disk space
* Good to have knowledge about Linux command line
* Good to have knowledge about Docker
* A note book and pen is recommended to take notes during the entire course


A Comprehensive and Brand New Course for Learning Apache Kafka Connect Framework with Hands-on Training – (Launched in April 2017)

Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems. Apache Kafka Connect is a common framework for Apache Kafka producers and consumers.  

Apache Kafka Connect offers an API, Runtime, and REST Service to enable developers to define connectors that move large data sets into and out of Apache Kafka in real time. It inherits strong concepts such as fault-tolerance and elasticity thanks to being an extension of Apache Kafka. Kafka Connect can ingest entire databases, collect metrics, gather logs from all your application servers into Apache Kafka topics, making the data available for stream processing with low latency.  

Kafka Connect standardises integration of other data systems with Apache Kafka, simplifying connector development, deployment, and management.  

In this course, we are going to learn the Kafka Connector deployment, configuration and management with hands-on exercises. We are also going to see the distributed and standalone modes to scale up to a large, centrally managed service supporting an entire organisation or scale down to development, testing, and small production deployments. The REST interface to submit and manage connectors to your Kafka Connect cluster via an easy to use REST API’s.  

Students Loved this course – Students have rated the course with 5 stars...!!

Tin Pavlinic says “This course is an excellent resource for those getting started with Apache Kafka-Connect. Stephane explains concepts very clearly and provides plenty of practical examples. The setup is easy. You can follow the examples along on your own machine, and then experiment further. I highly recommend this Apache Kafka course.

Alan V says "This course is GREAT: 1. The teacher provides clear explanations of the role to Kafka Connector in Kafka Streaming Solution architecture. 2. The course uses docker-compose.yml to help learners quick build a practice environment. 3. The course is well organized and easy to understand the idea, behind Kafka Connector. The teacher guides you how to use the tool step by step. 4. The provided shell commands are all well commented and steps by steps to guide users. If you read the comments in the shell script files after the course, it will remind you what the teacher just taught in the videos. 5. The teacher is very responsible to learners' questions and very helpful. If you are a professional developer, I highly recommend you to attend this course.

Sagar Joshi says "It was very engaging and interesting course. I recommend taking this Apache Kafka course. Stephane has provided all the required details and its really a hands-on training course. Thank you..!!" 



My name is Stephane Maarek, and I'll be your instructor in this course. I teach about Apache Kafka, the Kafka ecosystem and Kafka Certifications with my focus always on helping my students improve their professional proficiencies. I am also the co-founder of Conduktor: an enterprise Apache Kafka platform & UI to help everyone use Kafka.

Throughout my career in designing and delivering these certifications and courses, I have already taught 1,000,000+ students and gotten 350,000+ reviews!

With Apache Kafka becoming much more than a buzzword out there, I've decided it's time for students to properly learn how to be a Kafka professional. So, let’s kick start the course! You are in good hands!


This Course Also Comes With:

  • Lifetime Access to All Future Updates

  • A responsive instructor in the Q&A Section

  • Links to interesting articles, and lots of good code to base your next applications onto

  • Udemy Certificate of Completion Ready for Download

This is the course that could improve your career!

Apache Kafka is a skill in high demand and there are not enough people to fulfill all the open positions. You can boost your income, take on new roles and fun challenges. Many of my students are now the Kafka experts of their companies! You can be the next!

I hope to see you inside the course!


Note: Looking for more advanced Kafka concepts? There are many volumes in the Apache Kafka Series:

  • Learn Kafka for Beginners v2 (great to start)

  • Kafka Connect Hands On Learning

  • Kafka Streams for Data Processing

  • KSQL on ksqlDB - Hands On!

  • Kafka Cluster Setup & Administration

  • Confluent Schema Registry & Kafka REST Proxy

  • Kafka Security (SSL SASL ACL)

  • Kafka Monitoring and Operations

Happy learning!


Who this course is for:

  • Developers who wants to learn Apache Kafka Connect Framework and get hands-on with it
  • Professionals who are good at Apache Kafka ecosystem and basic core concepts
  • Architects who want to understand how Kafka Connect fits in their Solution Architecture

Course content

9 sections • 55 lectures

Important Pre-Requisites Preview 01:09

Pre-requisites for this Kafka Connect course

Course Objectives Preview 01:20

Learn about the course objectives:

- Learn about Kafka Connect concepts
- Install Docker and launch our Kafka cluster and Kafka Connect Cluster
- Deploy your first Source connectors
- Deploy your first Sink connectors

Course Structure Preview 01:55

Learn about the course structure

About Your Instructor Preview 02:20

Learn a bit about your instructor

What is Kafka Connect? Preview 04:50

Understand what is Kafka Connect, the problems it solves in the Kafka world

Kafka Connect Architecture Design Preview 03:20

Learn about the typical Kafka Connect Architecture in your enterprise

Connectors, Configuration, Tasks, Workers Preview 03:03

Deeper dive into Kafka Connect concepts, learn about Connectors, Configurations, Tasks and Workers

Standalone vs Distributed Mode Preview 02:11

Learn about the two modes to launch Kafka Connect, Standalone mode and Distributed Mode, and their pros and cons

Distributed Architecture in Details Preview 02:58

Visit the Kafka Connect distributed architecture and concepts in details

Important information about installation Preview 01:01

Docker on Mac (recent versions) Preview 03:54

Install Docker on the more recent Mac versions

Docker Toolbox on Mac (older versions) Preview 05:16

Install Docker Toolbox on the less recent Mac versions

Docker on Linux (Ubuntu as an example) Preview 04:56

Install Docker on Linux

Docker on Windows 10 64bit Preview 03:42

Install Docker on Windows 10 64bit

Docker Toolbox on Windows (older versions) Preview 05:37

Install Docker Toolbox on the other Windows versions

Important note for Docker Toolbox users Preview 00:28

Note for Docker Toolbox users

Starting Kafka Connect Cluster using Docker Compose Preview 03:40

Instructions to start our Kafka connect cluster using the Docker installation we have just created

It's not working! What to do? Preview 02:49

Steps to troubleshoot an error

Where to view logs? Preview 04:05

Where to view the Kafka Connect logs

Asking a question in the Q&A Preview 02:39

How to ask a question in the Q&A

Kafka Connect Source Architecture Design Preview 01:14

Looking at the Connect Source Architecture design that we will go over in this section

FileStream Source Connector - Standalone Mode - Part 1 Preview 09:05

Running our first source connector in standalone mode!
This goes over the setup of the worker and the setup of the standalone connector

FileStream Source Connector - Standalone Mode - Part 2 Preview 07:56

Running our first source connector in standalone mode! 
This goes over the deployment of that connector

FileStream Source Connector - Distributed Mode Preview 11:29

Learn how to run our file source connector in distributed mode

List of Available Connectors Preview 05:04

Overview of the list of available connectors online, and the list of connectors available on our docker image

Twitter Source Connector - Distributed Mode - Part 1 Preview 07:37

Look at the configuration for the Twitter Source Connector

Twitter Source Connector - Distributed Mode - Part 2 Preview 06:11

Deploy and analyse the output of the Twitter Source Connector on our Kafka Connect Cluster

Section Summary Preview 01:36

Summary of what we have learned for the Section on Kafka Connect Source Connectors

Kafka Connect Sink Architecture Design Preview 01:43

Architecture discussion for the sink connectors in distributed mode

ElasticSearch Sink Connector - Distributed Mode - Part 1 Preview 08:58

Launching our first Sink Connector in Distributed mode, to put data into ElasticSearch

ElasticSearch Sink Connector - Distributed Mode - Part 2 Preview 07:31

Kafka Connect REST API Preview 11:08

Learn about the Kafka Connect REST API, so you can speed up and automate your connector deployments and monitoring

JDBC Sink Connector - Distributed Mode Preview 13:06

Learn about the JDBC Sink Connector, that we will launch in distributed mode. The data will go directly into PostgreSQL

Goal of the section: GitHubSourceConnector Preview 01:34

Overview of the Source Kafka Connector, the GitHubSourceConnector

Finding the code and installing required software Preview 03:09

Where to download the code

Description of the GitHub Issues API Preview 03:55

Description of the GitHub API

Using the Maven Archetype to get started Preview 05:43

Using the Maven Archetype to create your source and sink connectors

Config Definitions Preview 12:31

Creating our first Config Definition and define all the parameters needed for our application

Connector Class Preview 06:57

Analysis of the Source Connector Class

Writing a schema Preview 07:31

Writing our different schemas that will be used for keys and values

Data Model for our Objects Preview 06:10

Creating the Plain Old Java Objects for our data

Writing our GitHub API HTTP Client Preview 07:03

Writing the GitHub API HTTP Client

Source Partition & Source Offsets Preview 04:22

Learn what Source Partition and Source Offsets are used for

Source Task Preview 13:40

Source Tasks analysis

Building and running a Connector in Standalone Mode Preview 04:59

Learn how to use Maven to build and run our connector

Command for next lecture Preview 00:20

Deploying our Connector on the Landoop cluster Preview 06:30

Learn how to deploy the jars onto the Connect Cluster

More Resources for Developers Preview 01:57

Links to more resources available online to write your own connectors

Setting up Kafka Connect in Production (1/2) Preview 14:19

Learn how to start your own Kafka Connect Cluster

Setting up Kafka Connect in Production (2/2) Preview 06:15

Learn how to start your second Connect Worker and the Kafka Connect UI

What's next? Preview 01:46

Next steps in your learning of Kafka Connect

Congratulations and Thank you! Preview 01:18


THANK YOU! Preview 01:32

Bonus Lecture: Student Special Coupons for my Other courses Preview 01:05

Check out my other courses and keep on learning! Coupons inside :)