Are the services in Rancher natively load balanced?

Create the headless service

To start with, we'll create a headless Kubernetes service called that defines a DNS domain for the 3 pods. A headless service does not load balance or has no static IP. To learn more about headless services, see the official Kubernetes documentation.

Open a file named with your favorite editor:

Add the following Kubernetes service YAML:

Then save and close the file.

We define a name in the namespace and give it the name. We then set up for the service to select pods with the label. When we map our Elasticsearch StatefulSet to this service, the service returns DNS A records that point to Elasticsearch Pods with the label.

We then set what makes the service headless. Finally, we define the ports and that are used to interact with the REST API or for communication between nodes.

Build the service with:

You should see the following output:

Finally, check that the service was created successfully with:

You should see:

After we have set up our headless service and a stable domain for our pods, we can create the StatefulSet.

Create a StatefulSet

With a Kubernetes StatefulSet, you can assign a stable identity to pods and give them stable, permanent storage space. Elasticsearch requires stable storage in order to preserve the data through the pod rescheduling and reboot. For more information about the StatefulSet workload, see the Statefulsets page in the Kubernetes documents.

Open a file with the name in your favorite editor:

We'll go through the StatefulSet object definition section by section and add blocks to this file.

Start pasting in the following block:


In this block we define a StatefulSet with the name in the namespace. We then link it to our previously created service via the field. This ensures that each Pod in the StatefulSet can be accessed using the following DNS address: where is the ordinal integer assigned by the pod.

We specify 3 (pods) and set up the selector, which we then mirror in the section. The fields and must match.

We can now proceed with the object specification. Add the following YAML block just below the previous block:


Here we define the pods in the StatefulSet. We name the containers and choose the Docker image. At this point, you can change this image tag to match your internal Elasticsearch image or a different version. Note that for the purposes of this guide, only Elasticsearch has been tested.

We then use the field to indicate that the container is guaranteed to require at least 0.1 vCPU and can burst up to 1 vCPU (which limits the pod's resource usage when doing an initial large pickup or handling a load spike). You should change these values ​​depending on your anticipated workload and the available resources. For more information on resource requirements and limits, see the official Kubernetes Documentation.

Then we open and name the ports and for the REST API or communication between nodes. We specify a with the name that integrates the PersistentVolume with the name under the path into the container. We will define the VolumeClaims for this StatefulSet in a later YAML block.

Finally we put some environment variables in the container:

  • : The name of the Elasticsearch cluster, in this guide.

  • : The name of the node that we put on the field. This is resolved into depending on the assigned ordinal number of the node.

  • : This field specifies a list of the master capable nodes in the cluster that will trigger the node discovery process. In this guide, thanks to the previously configured headless service, our pods have domains of the form, so we set this variable accordingly. With the DNS resolution of the local Kubernetes namespace, we can shorten this to. For more information on Elasticsearch detection, see the official Elasticsearch documentation.

  • : This field also provides a list of the master-able nodes that will participate in the master electoral process. Note that for this field, you should identify nodes by their hostnames, not their hostnames.

  • : Here we set this on, which tells the JVM to use a minimum and maximum heap size of 512MB. You should adjust these parameters based on the availability of resources and the requirements of your cluster. For more information, see Setting the heap size.

The next block we insert looks like this:


In this block we define several init containers that are executed before the main app container. These init containers are executed in the order in which they were defined. To learn more about Init Containers, see the official Kubernetes Documentation.

The first command by name runs a command to change the owner and group of the Elasticsearch data directory to, the UID of the Elasticsearch user. By default, Kubernetes mounts the data directory as, which makes it inaccessible to Elasticsearch. For more information on this step, see "[Notes for production use and defaults" by Elasticsearch.

The second command, named, runs a command to increase the operating system's limits on mmap counts, which are too low by default and cause memory errors. To learn more about this step, consult the official Elasticsearch documentation.

The next init container to run is that will run the command to increase the maximum number of open file descriptors. For more information on this step, see the official Elasticsearch documentation at "[Notes for Production Use and Defaults]".

[.grade] #Grade: ElasticsearchNotes for Production Use also mentions that swapping will be disabled for performance reasons. Depending on your Kubernetes installation or your provider, swapping may already be disabled. To verify this, type in a running container and run to list active swap devices. If you don't see anything there, the swap is disabled.

Now that we've defined our main app container and the init containers that will run previously to optimize the container operating system, we can add the final piece to our StatefulSet object definition file:

Insert the following block:


In this block we define the StatefulSet. Kubernetes uses this to create PersistentVolumes for the Pods. In the block above we name it (these are the ones we refer to in the label as StatefulSet.

We then specify the access mode as, which means that it can only be provided as read / write access by a single node. In this guide, we define the storage class as because we are using a DigitalOcean Kubernetes cluster for demonstration purposes. You should change this value depending on where you are running your Kubernetes cluster. For more information, see the Persistent Volume documentation.

Finally, we specify that each PersistentVolume should be 100 GB. You should adjust this value to suit your production needs.

The full StatefulSet specification should look something like this:


When you are happy with your Elasticsearch configuration, save and close the file.

Now provide the StatefulSet with:

You should see the following output:

You can monitor the StatefulSet during the rollout with:

You should see the following output when the cluster is provisioned:

Once all of the pods have been provisioned, you can verify that your Elasticsearch cluster is working properly by making a request for the REST API.

To do this, first forward the local port to the port on one of the Elasticsearch nodes () with:

Then make a request for the REST API in a separate terminal window:

You should see the following output:

This indicates that our Elasticsearch cluster was successfully created with 3 nodes :, and. The current main node is.

Now that your Elasticsearch cluster is up and running, you can set up a Kibana front end for it.

[[Step 3 - Create the Kibana Deployment and Service]] == Step 3 - Create the Kibana Deployment and Service

To start Kibana on Kubernetes, we create a service named and a deployment that consists of a replica pod. You can scale the number of replicas based on your production needs and optionally specify a type so that the service can balance requirements for load balancing in the deployment pods.

This time we create the service and the deployment in the same file. Open a file with the name in your favorite editor:

Insert the following service description:

Then save and close the file.

In this specification we have defined a service called in the namespace and given it the name.

We have also specified port to be accessed and use the label to select the service's target pods.

In the spec, we define a deployment by name and state that we want 1 pod replica.

We use the image of. At this point, you can put your own private or public Kibana image to use.

We state that at least 0.1 vCPU should be guaranteed for the pod, exceeding a limit of 1 vCPU. You can change these parameters depending on your anticipated workload and the available resources.

Next, we'll use the environment variable to set the endpoint and port for the Elasticsearch cluster. When using Kubernetes DNS, this endpoint is its service name. This domain is resolved into a list of IP addresses for the 3 Elasticsearch pods. For more information about Kubernetes DNS, see DNS for Services and Pods.

Finally, we set up the Kibana container port to which the service forwards inquiries.

Once you are happy with your Kibana configuration, you can start providing service and deployment with:

You should see the following output:

You can verify that the rollout was successful by running the following command:

You should see the following output:

To access the Kibana interface, we again forward a local port to the Kubernetes node running Kibana. Get the Kibana Pod Details with:

Here we observe that our Kibana Pod is called.

Forward the local port to port on this pod:

You should see the following output:

Now visit the following URL in your web browser:

If you see the following Kibana welcome page, you have successfully deployed Kibana to your Kubernetes cluster:

You can now continue with the final component of the EFK stack: the Fluentd log collector.

[[Step-4 - Creating the Flowing Demon Set]] == Step 4 - Creating the Flowing Demon Set

In this guide, we'll set up Fluentd as a DaemonSet, a type of Kubernetes workload that runs a copy of a particular Pod on each node in the Kubernetes cluster. This DaemonSet controller is used to deploy a Fluentd Logging Agent Pod on each node in our cluster. For more information on this logging architecture, see "[Using a node logging agent]" in the official Kubernetes docs .

In Kubernetes, for container applications that log in to and, the log streams are captured and redirected to the node in JSON files. The Fluentd Pod monitors these log files, filters log events, transforms the log data and sends it to the Elasticsearch logging backend that we provided in Step 2.

In addition to the container logs, the Fluentd agent tracks the logs of the Kubernetes system components such as Kubelet, Kube proxy and Docker logs. For a complete list of the sources provided by the Fluentd logging agent, see the file used to configure the logging agent. For more information on logging in Kubernetes clusters, see the official website at "[Logging at the node level" ] "Kubernetes Documentation.

First, open a file with the name in your favorite text editor:

Again we will insert the Kubernetes object definitions block by block, specifying the context. In this guide we use the Fluentd DaemonSet spec provided by the Fluentd maintainers. Another helpful resource provided by the Fluentd maintainers is Kuberentes Fluentd.

First, add the following ServiceAccount definition:

Here we will create a service account with the name that the Fluentd Pods will use to access the Kubernetes API. We create it in the namespace and give it the name again. For more information about service accounts in Kubernetes, see Configure Service Accounts for Pods in the official Kubernetes documents.

Next, add the following block:

Here we define a ClusterRole named, to which we grant the permissions and for the objects and. You can use ClusterRoles to grant access to cluster Kubernetes resources such as nodes. For more information about role-based access control and cluster roles, see Using RBAC Authorization in the official Kubernetes documentation.

Now insert the following block:

In this block we define a name that binds the ClusterRole to the service account. This grants the ServiceAccount of the permissions listed in the cluster role of.

At this point we can start adding in the current DaemonSet specification:

Here we define a DaemonSet with the name in the namespace and give it the name.

Next, add the following section:

Here we agree with the designation defined in and assign the service account to the DaemonSet. We also select the Pods being managed by this DaemonSet.

Next, we'll define a tolerance to match the equivalent taint on Kubernetes master nodes. This ensures that the DaemonSet is also distributed to the Kubernetes masters. If you don't want to run a Fluentd Pod on your master nodes, remove this tolerance. For more information on Kubernetes errors and tolerances, see the official Kubernetes documents at "[Taints and Tolerations]".

Next we define the pod container that we will name.

We are using the official v1.4.2 Debian image provided by the Fluentd maintainers. If you want to use your own private or public Fluentd image, or a different version of the image, change the tag in the container specification. The Docker file and the content of this image are available from influentd-kubernetes-daemonset Github rep from Fluentd.

Next, let's configure Fluentd using a few environment variables:

  • : We set this to the previously defined headless service address of Elasticsearch :. This will create a list of IP addresses for the 3 Elasticsearch pods. The actual Elasticsearch host is most likely the first IP address returned on this list. To distribute logs across the cluster, you need to change the configuration for Fluentd's Elasticsearch output plug-in. To learn more about this plugin, see Elasticsearch Output Plugin.

  • : We set this on the previously configured Elasticsearch port.

  • : We put this on.

  • : We put this on to suppress the output related to that is not set up in the container.

Finally, add the following section:

Here we specify a storage capacity of 512 MiB for the FluentD Pod and guarantee 0.1 vCPU and 200 MB memory.You can optimize these resource limits and requirements based on your expected log volume and the available resources.

Next we mount the host paths and into the container using and. These are defined at the end of the block.

The last parameter we define in this block is what gives Fluentd 30 seconds to gracefully shut down when it receives a signal. The containers receive a signal after 30 seconds. The default value for is 30s, so this parameter can be omitted in most cases. For more information on how to properly terminate Kubernetes workloads, see "[Kubernetes-Best Practices: Terminate with reservations" .

The entire Fluentd spec should look something like this:

After you have configured the Fluentd DaemonSet, save and close the file.

Now roll out the DaemonSet with:

You should see the following output:

Make sure that your DaemonSet was successfully implemented with:

You should see the following status output:

This indicates that 3 pods are running, which is the number of nodes in our Kubernetes cluster.

We can now review Kibana to make sure the log data is being properly collected and sent to Elasticsearch.

With it still open, navigate to.

In the left navigation menu, clickDiscover:

You should see the following configuration window:

This is a great way to define the Elasticsearch indexes that you want to explore in Kibana. For more information, see Defining your index patterns in the official Kibana documents. For now, we're just using the wildcard pattern to collect all of the log data in our Elasticsearch cluster. Enter in the text box and clickNext step.

You will then be redirected to the following page:

This allows you to configure in which field Kibana filters the log data by time. Select the field from the drop-down list@timestamp and pressCreate index pattern.

Now press in the left navigation menuDiscover.

You should see a histogram graph and some recent log entries:

At this point, you have successfully configured and provisioned the EFK stack on your Kubernetes cluster. For information on using Kibana to analyze your historical data, see the Kibana User Guide.

In the next optional section, we're going to provide a simple counter pod that prints numbers on stdout and finds its logs in Kibana.

[[Step-5-optional -—- testing the container logging]] == Step 5 (optional) - testing the container logging

To demonstrate a basic Kibana use case for browsing the latest logs for a particular pod, we provide a minimal counter pod that outputs sequential numbers by standard.

Let's start by creating the pod. Open a file with the name in your favorite editor:

Then paste in the following pod specs:

Save and close the file.

This is a minimal pod called that does a loop and prints numbers one at a time.

Provide the pod with:

Once the pod is created and running, navigate back to your Kibana dashboard.

Enter on the sideDiscover in the search bar. This will filter the log data for Pods named.

You should then see a list of log entries for the Pod:

You can click in any of the log entries to view additional metadata such as the container name, Kubernetes node, namespace, and so on.