How to modernize your information system thanks to KAFKA?

Kafka is the major component of a distributed business event streaming platform.

Kafka enables complex processing of real-time data streams. It is used more and more often:

• For the implementation of data pipelines in real time and with large volumes to manage, in order to enable an efficient communication between the business silos of the information system

• In microservices architectures as an exchange system replacing synchronization batches

• In real-time supervision of systems based on sensors and effectors (supervision of networks, supervision of robot fleets, supervision of factories, etc.)

• In the IOT (internet of things): “smart cities”, etc.

Apache Kafka is an open source project developed by the Apache Software foundation. The project was initially developed by LinkedIn before its code was opened in 2011. In 2014 the main creators of Kafka founded the company Confluent which offers an enterprise streaming platform based on Apache Kafka.

Kafka provides five basic APIs:

• Producer: used to produce data flows in topics

• Consumer: allows you to consume data streams from topics

• Streams: used to transform (“Transform”) data streams

• Connect: allows you to interface with external systems to extract (“Extract”) data from a source system to kafka or to load (“Load”) a target system from kafka

• AdminClient: allows you to manage the kafka components (topics, brokers, clusters, connectors, diagrams)

Kafka also allows the very rapid development of microservices for filtering, sorting and processing on-the-fly streams using the KSQLDB tool.

EMOXA offers consultancy services to define Kafka flows, performance optimization, assistance in the supervision of a KAFKA infrastructure and training.