Loki + Grafana + Prometheus is a powerful combination for monitoring and logging. I found there were very few up-to-date instructions on how to set up all these three components with helm on Kubernetes. So I decided to write down my experience here.

Prerequisites

You need to have helm installed. If not you can follow the instructions here Add corresponding helm repositories:

for Grafana and Loki:

helm repo add grafana https://grafana.github.io/helm-charts

for Prometheus:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

It’s good practice to create a separate namespace for monitoring:

kubectl create namespace monitoring

Install Grafana

Grafana connects to Loki and Prometheus to display logs and metrics. It can be installed standalone or as part of the Loki stack. I prefer to install it standalone as it gives me more flexibility on configuration such as the data volume.

here is the minimum helm chart values used for Grafana:

persistence:
  type: pvc
  enabled: true
  storageClassName:  # insert storage class for the data volume
  accessModes:
    - ReadWriteOnce
  size: 10Gi

for storageclass you can create one using your cloud provisioner. Please refer to the documentation of your cloud provider for more information.

then just run the following command to install Grafana:

helm install grafana grafana/grafana --namespace monitoring -f values.yaml

so this will create a 10Gi PVC for Grafana data. By setting this up, you will not lose your dashboards and settings when the pod is restarted or recreated.

Install Loki stack

Loki stack includes Loki which is the aggregation layer and query engine for logs, Promtail which is the agent that collects logs and sends them to Loki, and Grafana. Since we have already installed Grafana, we can install Loki and Promtail with the following values:

loki:
  image:
    tag: 2.9.7
  persistence:
    enabled: true
    storageClassName: # insert storage class for the data volume
    size: 50Gi

The tricky part is you need to assign loki version to go along with Grafana version as the default version could be outdated which may cause compatibility issues like this one https://github.com/grafana/loki/issues/12566

For the storage size, you need to estimate on your case considering the log volume and retention period.

Then run the following command to install Loki stack:

helm install loki grafana/loki-stack --namespace monitoring -f values.yaml

if all goes well you should see both loki and promtail up and running.

Then on Grafana, you can add Loki as a data source and start querying logs.

Install Prometheus

Prometheus is used for metrics collection. It also needs storage for the data and for most cases you want to config the scrape targets. I recommend you read the official values template file https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml which is well annotated.

What I did is just copy this file to local and modify the sections like persistentVolume and scrapeConfigs to suit my needs. Similar to Loki you need to estimate the storage size based on your metrics volume and retention period. You might want to turn off the alert manager if you intend to use Grafana alerting which can tap into prometheus metrics.

again you can install Prometheus with the following command:

helm install prometheus prometheus-community/prometheus --namespace monitoring -f values.yaml

After checking the pods are up and running, you can add Prometheus as a data source in Grafana.

You should have a basic monitoring setup with Loki, Grafana, and Prometheus. You can start exploring the metrics and logs and create your own dashboards. Of course there are more advanced configurations you can do like setting up alerting, using Grafana Loki plugin, etc. But I hope this guide can help you get started.