Exposing services outside of kubernetes with MetalLB load balancer

Sat, Feb 19, 2022 6-minute read

Introduction

When you want whatever you host inside a kubernetes cluster to be accessible from the outside - that being either on the LAN or on the internet, then you need a way for traffic to flow into the cluster, hit the correct pods and preferably all in a nice HA way.

There are several options you can use out of the box when you want to host services in a kubernetes cluster.

ClusterIP

This is the default way services are exposed and what you get if you do not define anything. This configuration makes the service only available on the inside of the cluster. So if you want to access your service from outside the cluster - this is not an option.

NodePort

This setting is used if you are okay with exposing ports directly from the individual pods - so what happens is that the cluster exposes the IP Address of the underlying pod - with what ever configured ports you have defined for your service. This works fine for testing purposes, if you just want to see if your service behaves as it should inside the cluster.

But if you want to have any kind of HA - then suddenly you have to manage that from the outside of the cluster - with the added disadvantage that if a pod gets moved to another node because it fails or for any other reason - then you have to update your HA setup on the outside of the cluster to point to the new node it runs on. So obviously this is not very HA.

LoadBalancer

This is the only real solution if you want to expose your service outside of the cluster and you want any HA. Without any other configuration the LoadBalancer type only works with external load balancers that kubernetes can update - so basically only in a cloud environment.

Luckily there are others that think LoadBalancer would be a useful usage scenario if you are not running your cluster in the cloud, but instead on bare metal.

Solution

Enter MetalLB. This is a load balancer that you deploy inside the kubernetes cluster that works seamlessly - so if you configure a service with LoadBalancer type, then MetalLB assigns an ip address to that service - and provide load balancing between all the pods that run that service.

Prequisites

Kube-proxy needs to be tweaked to get the load balancer working.

To quote exactly what is on the metallb website you have to do the following:

kubectl edit configmap -n kube-system kube-proxy

And find strictARP and set to to true, so it looks like:

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
  strictARP: true

Installation

To configure MetalLB you deploy it via the manifests simply by connecting to a control plane node and doing the following:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/metallb.yaml

Now the load balancer is deployed inside the cluster - but its not fully configured yet - it needs ip addresses it can load balance between.

Configuration

These ip addresses can be internal ip addresses, external ip addressess - it does not matter as long as they are routable on the local network where you want to access the services inside the cluster.

In my kubernetes cluster i have assigned the following network to MetalLB:

192.168.6.0/24

Which is the addresses: 192.168.6.1 -> 192.168.6.254.

Its very easy to configure - just create a yaml file with contents similar to:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.6.0/24
      avoid-buggy-ips: true
    - name: production-public-ips
      protocol: layer2
      addresses:
      - 85.204.132.112/28    

This configuration should be written to a file and applied to the cluster via:

kubectl apply -f metal-lb.yml

As you can see from the configuration - I have just added the network I want to use for my load balanced ip addresses - I could also have used:

      addresses:
      - 192.168.6.1-192.168.6.254

I have also added an address pool called “production-public-ips” - which is just another pool that I will show how to use later in this post.

Single IP’s

If you like we want to move existing servers to run as pods inside kubernetes, then its awesome that you can just tell metallb to re-use single ip-addresses as part of its pool.

The syntax follows the normal CIDR syntax.

So you simply add a new address as:

data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.6.0/24
      - 192.168.0.2/32    

As you can see I have added the single ip address 192.168.0.2 to the pool, which means I can simply request that ip address for a service, if that corresponded to a server I had running outside of kubernetes. The advantage of this is that I don’t have to update the configuration of all my clients, since the IP address of the service stays the same whether or not its running outside of kubernetes or inside.

To see all the possible configurations you can check the configuration example

Exposing services

So with the above steps we now have a working load balancer in the cluster that can expose services with ip addresses in the range 192.168.6.1 -> 192.168.6.254.

To expose a service all you have to do is change the configuration of any existing service you have - and change type: ClusterIP to LoadBalancer.

That will make MetalLB assign an ip address for the service and you should immediately be able to access the service using the “External IP” of the service

An example from my own cluster:

NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/rancher           LoadBalancer   10.107.48.153    192.168.6.1   80:31082/TCP,443:30314/TCP   3h6m

My Rancher service is accessible on the LAN on the IP Address 192.168.6.1 and port 80 or port 443.

It is also possible to request a specific IP Address from MetalLB - so you e.g. can update external DNS to point to your now load balanced kubernetes service.

This is done by annotating your service to tell MetalLB that you want either a specific ip address, a specific ip range or a specific ip pool.

If you want to use a specific address pool, you do something similar to this

apiVersion: v1
kind: Service
metadata:
  name: My-Awesome-Service
  annotations:
    metallb.universe.tf/address-pool: production-public-ips
spec:
  type: LoadBalancer

Where the value of the metallb.universe.tf/address-pool is the pool you have created in MetalLB configuration.

If you want a specific ip address you do:

apiVersion: v1
kind: Service
metadata:
  name: My-Awesome-Service
  annotations:
    metallb.universe.tf/loadBalancerIPs: 192.168.6.99,192.168.6.100
spec:
  type: LoadBalancer

This can either be a single ip address, or a comma separated list of ip addresses.

If you don’t care - you just omit the annotation of your service - and MetalLB just picks an address for your service from its default address pool.

Of course if you want to update external DNS with a specific ip address, it makes sense to request a “static” ip address from MetalLB for your service - then outside callers will always be able to acces your service, since the IP Address will not change.