Kafka is not the best anymore, Meet Pulsar !

  1. Scaling Kafka is difficult, this is due to the way Kafka stores data within the broker as distributed logs that stores as messaging persistence store. Spinning off another broker means it has to replicate the topic partitions and replicas completely which takes time to complete the broker spin off.
  2. Changing partition sizes to allow more storage can mess the message order by conflicting the message indexes. This is critical if the message order is a prime concern.
  3. Leader election can go crazy if the partition replicas are not in ISR (In Sync) state. Basically there should be at least one ISR replica to be elected as the leased when the original leader partition fails, but this cannot be guaranteed. there is a setting to disable this but then if enabled a non-ISR replica will get elected as the leader which is much worse since that impose a worse situation than a service outage without a leader partition.
  4. Ideally you must plan and calculate number of brokers, topics, partitions and replicas in first place (that fits planned future usage growth) to avoid scaling problems, but now adays with unpredictable traffic demands and spike its is hard to plan.
  5. Kafka cluster rebalancing can impact the performance of connected producers and consumers
  6. Kafka topics can lose messages in failure scenarios (specially in point 3)
  7. Working with offsets is a pain because Kafka is dumb (don’t be angry for saying that, because the “dumb” comes from the architectural concept “dumb broker and intelligent client”)
  8. Old messages has to be deleted pretty soon if the usage is high to avoid disk space problems
  9. Kafka’s native across region geo replication mechanism (MirrorMaker) is notoriously problematic, even within just two data centers. Because of this even Uber has created another solution to overcome this called uReplicator as an example
  10. You must use another realtime event analyzer tool such as Apache Storm, Apache Heron or Apache Spark if you need to do so, also that should be in a strength to support the incoming traffic rates
  11. No native multi-tenancy capabilities with complete isolation of tenants . It is done with the use of security features such as topic authorization

What is Apache Pulsar ?

Pulsar Performance !

Apache pulsar advantageous and features

package org.example.functions;import org.apache.pulsar.functions.api.Context;
import org.apache.pulsar.functions.api.Function;
import java.util.Arrays;public class WordCountFunction implements Function<String, Void> {
// This is invoked every time messages published to the topic
@Override
public Void process(String input, Context context)
throws Exception {
Arrays.asList(input.split(" ")).forEach(word -> {
String counterKey = word.toLowerCase();
context.incrCounter(counterKey, 1);
});
return null;
}
}
show tables in pulsar."public/default"

Summary

--

--

--

an enterprise architect, technology enthusiast, dog lover , music maniac, a husband & a father ! 🐶

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Moving to Hybrid Cloud with Google Anthos

Lab 7: Output: DC Motors

How to Validate an Email Address for Correct Syntax in C# .NET Framework

How to convert JSON to XML format in Python

How to increase your LinkedIn network?

Helm Chart For Creating and Publishing Jenkins on artifacthub.io

Build a Twitter Chatbot with PubNub Functions

How to update a VM to Send to Log Analytics after Provisioning on Azure

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Anuradha Prasanna

Anuradha Prasanna

an enterprise architect, technology enthusiast, dog lover , music maniac, a husband & a father ! 🐶

More from Medium

File automation and ingestion platform

file automation platform

How to integrate AWX with WSO2 API Manager?

Getting Started with Application Containerization

Everything about Monolithic Architecture