Clustering Hazelcast on Kubernetes

I’ve had experience with clustering Hazelcast in Google Cloud Engine before, but right now, in the advent of containers, Kubernetes is becoming a standard for app/container orchestration and therefore my previous effort has become somewhat obsolete – unless you’re limited to VMs and have to use TCP-based clustering for Hazelcast, since multicast traffic may be blocked (it is on most cloud providers).

Given this, I’ve assembled a Dockerfile which will run a small app (hazelcast-kubernetes-bootstrapper) on boot. This app queries Kubernetes API to discover all nodes in the [Kubernetes] cluster that are acting as an Hazelcast node, retrieves their IP addresses, configures and instantiates Hazelcast with TCP configuration, accordingly.

Please test it and give me feedback. All you need is Docker and a Kubernetes cluster (I’ve tested it locally and in Google Container Engine).

More information, including step-by-step instructrion can be found at https://github.com/pires/hazelcast-kubernetes.

Cheers,
Paulo Pires

Advertisements

6 thoughts on “Clustering Hazelcast on Kubernetes

  1. Himanshu Jaiswal says:

    Hi Pires,

    I have set up hazelcast cluster (using minikube) using the steps given at https://github.com/pires/hazelcast-kubernetes . In addition to that I have exposed the service there on NodePort for accessing from the outside. But when I am creating a client (using java api) to access the cluster, I am only able to connect to one of the instances and showing error like this –

    Members [2] {
    Member [172.17.0.4]:5701 – af48cda4-a581-48c2-84ef-9454f437e98d
    Member [172.17.0.5]:5701 – a1ea8e1e-8a82-4307-95cb-1dd3f6a2d7b2
    }

    Jan 08, 2017 4:38:43 PM com.hazelcast.core.LifecycleService
    INFO: hz.client_0 [someGroup] [3.7.4] HazelcastClient 3.7.4 (20161209 – 3df1bb5) is CLIENT_CONNECTED
    jjjj
    Exception in thread “main” com.hazelcast.core.HazelcastException: java.io.IOException: No available connection to address [172.17.0.5]:5701

    Can you help me here?

    • This happens because Hazelcast leaks the cluster topology so clients can connect to other nodes. However, Hazelcast only knows about intra-cluster IPs and not external IPs, hence your client can connect only to one node, the one exposed with NodePort.

      • Himanshu Jaiswal says:

        Hi Pires,

        Thanks for the reply. I have some questions-
        1) All the nodes are replicas, they all should be accessible by the service which I exposed at Nodeport with the external ip of minikube?
        2) How to connect the other nodes since internal ip address of kubernetes cannot be accessed from outside?

  2. 1 – All nodes are replicas and are accessible through the NodePort *but* Hazelcast itself doesn’t know about the external addresses but only the intra-cluster addresses. Therefore, leaking the topology will lead to this problem. It’s something you’ll need to workaround with Hazelcast itself.

    2 – Short answer, you can’t.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s