Introduction

In my kubernetes cluster I am running Prometheus as my time series storage for the metrics that servers & services create, ceph is one example of this.

Earlier today I wanted to look into historical stats, but found out that prometheus by default truncates data after 14 days.

This is done because its optimized for short time periods, so I did not want to tweak the configuration to enable it to store data for longer times.

So I looked into how I could store my data for a longer time - and I found Mimir - and since prometheus have an option to write to remote storage I wanted to configure that.

But it was not as easy as I initially thought, since the documentation for this is lacking to say the least.

So this post is a simple guide for others that want to do what I did - install Mimir and get Prometheus to write its metrics for longer storage.

Guide

Prequisites

Obviously it requires that you have a Mimir deployment running somewhere - I deployed it in my kubernetes cluster - and had some issues with that as well - since it turns out that the nginx pod they deploy for ingress purposes gets deployed with an invalid DNS configuration.

I fixed the ConfigMap according to this issue

Which allowed my Mimir deployment to be fully working.

Prometheus configuration

To enable prometheus to write to mimir - you have to add a remote_write section to the configuration. The docs says:

remote_write:
  - url: http://<ingress-host>/api/v1/push

And also mentions:

From outside the cluster via ingress:
http://myhost.mynetwork/prometheus

This got me confused - but I finally found the correct url to add to my prometheus configuration:

Which is:

http://<chart name>.mimir-nginx.<namespace>.svc.cluster.local:80/api/v1/push

Reading the documentation I thought first the correct url was:

http://<chart name>.mimir-nginx.<namespace>.svc.cluster.local:80/prometheus - then I thought it was: http://<chart name>.mimir-nginx.<namespace>.svc.cluster.local:80/prometheus/api/v1/push

But finally I tried without prometheus in the path - and I could see the prometheus logs not being full of 404’s.

In my cluster my full url is: http://mimir-mimir-nginx.r00t.svc.cluster.local:80/api/v1/push

With the correct url in had we can modify the prometheus configuration - which is luckily very simple.

Just find any top level and add the remote_write.

I have added mine near the top of the file:

rule_files:
  - /etc/config/recording_rules.yml
  - /etc/config/alerting_rules.yml
  - /etc/config/rules
  - /etc/config/alerts
remote_write:
  - url: http://mimir-mimir-nginx.r00t.svc.cluster.local:80/api/v1/push

This new mimir storage allows me to add a new datasource to grafana - which uses the /prometheus path: http://mimir-mimir-nginx.r00t.svc:80/prometheus

And with the new datasource, I can switch datasource when I look at my dashboard - if I want to look at stuff that is older than 14 days.

I hope you enjoyed this post and if you spot errors, please let me know in the comments below on on email directly.