How to use mimir as a remote storage for prometeus
Introduction
In my kubernetes cluster I am running Prometheus as my time series storage for the metrics that servers & services create, ceph is one example of this.
Earlier today I wanted to look into historical stats, but found out that prometheus by default truncates data after 14 days.
This is done because its optimized for short time periods, so I did not want to tweak the configuration to enable it to store data for longer times.
So I looked into how I could store my data for a longer time - and I found Mimir - and since prometheus have an option to write to remote storage I wanted to configure that.
But it was not as easy as I initially thought, since the documentation for this is lacking to say the least.
So this post is a simple guide for others that want to do what I did - install Mimir and get Prometheus to write its metrics for longer storage.
Guide
Prequisites
Obviously it requires that you have a Mimir deployment running somewhere - I deployed it in my kubernetes cluster - and had some issues with that as well - since it turns out that the nginx pod they deploy for ingress purposes gets deployed with an invalid DNS configuration.
I fixed the ConfigMap according to this issue
Which allowed my Mimir deployment to be fully working.
Prometheus configuration
To enable prometheus to write to mimir - you have to add a remote_write
section to the configuration. The docs says:
remote_write:
- url: http://<ingress-host>/api/v1/push
And also mentions:
From outside the cluster via ingress:
http://myhost.mynetwork/prometheus
This got me confused - but I finally found the correct url to add to my prometheus configuration:
Which is:
http://<chart name>.mimir-nginx.<namespace>.svc.cluster.local:80/api/v1/push
Reading the documentation I thought first the correct url was:
http://<chart name>.mimir-nginx.<namespace>.svc.cluster.local:80/prometheus
- then I thought it was: http://<chart name>.mimir-nginx.<namespace>.svc.cluster.local:80/prometheus/api/v1/push
But finally I tried without prometheus in the path - and I could see the prometheus logs not being full of 404’s.
In my cluster my full url is: http://mimir-mimir-nginx.r00t.svc.cluster.local:80/api/v1/push
With the correct url in had we can modify the prometheus configuration - which is luckily very simple.
Just find any top level and add the remote_write.
I have added mine near the top of the file:
rule_files:
- /etc/config/recording_rules.yml
- /etc/config/alerting_rules.yml
- /etc/config/rules
- /etc/config/alerts
remote_write:
- url: http://mimir-mimir-nginx.r00t.svc.cluster.local:80/api/v1/push
This new mimir storage allows me to add a new datasource to grafana - which uses the /prometheus path:
http://mimir-mimir-nginx.r00t.svc:80/prometheus
And with the new datasource, I can switch datasource when I look at my dashboard - if I want to look at stuff that is older than 14 days.
I hope you enjoyed this post and if you spot errors, please let me know in the comments below on on email directly.