K8s Centralized Monitoring using Prometheus

For whatever reason, if you want to collect metrics of multiple k8s clusters and store them all in one place, this article is for you.

We’ll be using remote write, because, it’s the easiest way to implement!

STEP1: Setting up prometheus operator on your k8s clusters

Before you start setting up prometheus-operator, it’s suggested you check the compatible version for your k8s cluster here .

Let’s set up prometheus-operator using kube-prometheus

git clone https://github.com/prometheus-operator/kube-prometheus.git -b release-0.10 # I picked 0.10 version here

After cloning the repo, inside manifests/setup folder you can find the CRDs for different components we’ll be installing on the cluster and inside manifests you can see all the deployments, stateful-sets and daemon-sets we’ll be setting up.

Setting up Centralized Prometheus server

Ways this can be achieved using prometheus:

• Prometheus Federation — Federation | Prometheus
• Prometheus Remote Write — Remote write tuning | Prometheus

Prometheus Federation -
“Federation allows a Prometheus server to scrape selected time series from another Prometheus server.”

Prometheus Remote Write -
“The remote write allows Prometheus to accept remote write requests from other Prometheus servers.”

So, Prometheus Federation can scrape other prometheus servers of different clusters and store all of them in one place
OR, even better, all your prometheus servers from different clusters can write to a centralized prometheus server using remote write feature.

We’ll be using remote write, because it’s easy to allow incoming traffic at just the centralized prometheus instead of allowing federation prometheus at each and every cluster.

Also, because, it’s the easiest way to implement!

First, we’ll have to make sure the centralized prometheus(receiver) has remote-write-receive feature enabled from the startup command.

For the newer versions of prometheus:

 — web.enable-remote-write-receiver

For the older versions:

--enable-feature=remote-write-receiver

add, the flag above to prometheus startup command and start the prometheus server, we can verify whether it’s working only after we attempt to remote write to this prometheus server, so let’s move on to next step.

Now, while you deploy your prometheus(sender) to your k8s cluster, in your prometheus.yml config file, add the following at the end

remote_write:
    - url: http://CENTRALIZED_PROMETHEUS_SERVER_IP:9090/api/v1/write

If your prometheus is protected with basic auth, you can pass the login credentials with additional config mentioned here: Configuration | Prometheus

That’s all! Yes, I’m not kidding, that’s all you have to do.

Of course you need to add the remote write config to all your k8s cluster prometheus(senders) config files, but, that’s all.

Now you can head over to your central prometheus and check the metrics from metrics explorer.

Note: The above setup assumes you already have exporters and prometheus configured to collect k8s cluster metrics within the cluster.

If you’re starting with a fresh setup for monitoring your k8s cluster, you can setup prometheus using kube-prometheus, which provides node exporter, kube state metrics, black box exporter etc out of the box.

Here’s how to set it up on your k8s cluster: GitHub — kubernetes/kube-state-metrics: Add-on agent to generate and expose cluster-level metrics.