Prometheus and Grafana

🚀 Introduction

In this HashiQube DevOps lab, you will get hands-on experience with Grafana and Prometheus. Together, they provide a powerful monitoring and alerting solution for your infrastructure.

📊 Monitoring Components

Grafana

Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.

Prometheus

Prometheus is an open source monitoring system for which Grafana provides out-of-the-box support. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts when specified conditions are observed.

🛠️ Provision

In order to provision Prometheus and Grafana, you need basetools, docker, and minikube as dependencies.

💡 We enable Vault, Consul and Nomad, because we monitor these with Prometheus. We also enable Minikube because we host Grafana and Prometheus on Minikube and deploy them using Helm.

Choose one of the following methods to set up your environment:

bash docker/docker.sh
bash vault/vault.sh
bash consul/consul.sh
bash nomad/nomad.sh
bash minikube/minikube.sh
bash prometheus-grafana/prometheus-grafana.shCopy to clipboardErrorCopiedCopy to clipboardErrorCopied

vagrant up --provision-with basetools,docker,docsify,vault,consul,nomad,minikube,prometheus-grafanaCopy to clipboardErrorCopiedCopy to clipboardErrorCopied

docker compose exec hashiqube /bin/bash
bash hashiqube/basetools.sh
bash docker/docker.sh
bash docsify/docsify.sh
bash vault/vault.sh
bash consul/consul.sh
bash nomad/nomad.sh
bash minikube/minikube.sh
bash prometheus-grafana/prometheus-grafana.shCopy to clipboardErrorCopiedCopy to clipboardErrorCopied

🔍 Accessing Monitoring Tools

After provisioning, you can access the interfaces at:

• Prometheus: http://localhost:9090
• Alertmanager: http://localhost:9093
• Grafana: http://localhost:3000
  • Username: admin
  • Password: Look for the password displayed in the terminal output

Look at the Minikube dashboard for progress updates and check the terminal output for details:

...
hashiqube0.service.consul: ++++ Waiting for Prometheus to stabalize, sleep 30s
hashiqube0.service.consul: NAME                                            READY   STATUS    RESTARTS   AGE
hashiqube0.service.consul: grafana-557fc9455c-67h4s                        1/1     Running   0          90s
hashiqube0.service.consul: hello-minikube-7bc9d7884c-fks85                 1/1     Running   0          3m36s
hashiqube0.service.consul: prometheus-alertmanager-76b7444fc5-8b2sq        2/2     Running   0          100s
hashiqube0.service.consul: prometheus-kube-state-metrics-748fc7f64-hxcvj   1/1     Running   0          100s
hashiqube0.service.consul: prometheus-node-exporter-xm6fw                  1/1     Running   0          100s
hashiqube0.service.consul: prometheus-pushgateway-5f478b75f7-j9tpj         1/1     Running   0          100s
hashiqube0.service.consul: prometheus-server-8c96d4966-bv24c               1/2     Running   0          100s
...
hashiqube0.service.consul: ++++ Prometheus http://localhost:9090
hashiqube0.service.consul: ++++ Grafana http://localhost:3000 and login with Username: admin Password:
hashiqube0.service.consul: N6as3Odq7bprqVdvWV5iFmwhOLs8QvutCJb8f2lSCopy to clipboardErrorCopiedCopy to clipboardErrorCopied

🔄 Prometheus Targets

You can open the Prometheus web interface and look at Status -> Targets to verify that all monitored services are properly configured:

📈 Grafana Configuration

💡 The Grafana datasource is configured automatically during the provisioning step in the grafana-values.yaml file.

[filename](grafana-values.yaml ':include :type=code')Copy to clipboardErrorCopiedCopy to clipboardErrorCopied

To access Grafana, go to http://localhost:3000 and log in with:

• Username: admin
• Password: The token displayed in the terminal output

📊 Monitoring HashiCorp Products

Monitoring Vault

HashiCorp Vault is configured with telemetry for Prometheus monitoring:

# https://developer.hashicorp.com/vault/docs/configuration/telemetry
# https://developer.hashicorp.com/vault/docs/configuration/telemetry#prometheus
telemetry {
  disable_hostname = true
  prometheus_retention_time = "12h"
}Copy to clipboardErrorCopiedCopy to clipboardErrorCopied

The Prometheus values file is configured to scrape Vault metrics:

[filename](prometheus-values.yaml ':include :type=code')Copy to clipboardErrorCopiedCopy to clipboardErrorCopied

You should see the Vault target in Prometheus web interface at http://localhost:9090/targets:

To view the Vault dashboard in Grafana:

1. Navigate to Grafana
2. Click on the + symbol in the top menu
3. Select "Import Dashboard"
4. Enter Dashboard ID: 12904
5. Click "Load"

Monitoring Nomad

HashiCorp Nomad is configured with telemetry for Prometheus monitoring:

# https://developer.hashicorp.com/nomad/docs/configuration/telemetry
telemetry {
  collection_interval = "1s"
  disable_hostname = true
  prometheus_metrics = true
  publish_allocation_metrics = true
  publish_node_metrics = true
}Copy to clipboardErrorCopiedCopy to clipboardErrorCopied

To view the Nomad dashboard in Grafana:

1. Navigate to Grafana
2. Click on the + symbol in the top menu
3. Select "Import Dashboard"
4. Enter Dashboard ID: 12787
5. Click "Load"

Monitoring Consul

HashiCorp Consul is configured with telemetry for Prometheus monitoring:

# https://developer.hashicorp.com/consul/docs/agent/telemetry
telemetry {
  prometheus_retention_time = "24h"
  disable_hostname = true
}Copy to clipboardErrorCopiedCopy to clipboardErrorCopied

To view the Consul dashboard in Grafana:

1. Navigate to Grafana
2. Click on the + symbol in the top menu
3. Select "Import Dashboard"
4. Enter Dashboard ID: 10642
5. Click "Load"

Monitoring Docker

Docker is configured to expose metrics for Prometheus:

{
  "metrics-addr": "0.0.0.0:9323",
  "experimental": true,
  "storage-driver": "overlay2",
  "insecure-registries": ["10.9.99.10:5001", "10.9.99.10:5002", "localhost:5001", "localhost:5002"]
}Copy to clipboardErrorCopiedCopy to clipboardErrorCopied

To view the Docker dashboard in Grafana:

1. Navigate to Grafana
2. Click on the + symbol in the top menu
3. Select "Import Dashboard"
4. Enter Dashboard ID: 10619
5. Click "Load"

⚙️ Prometheus Grafana Provisioner

The provisioning script handles the installation and configuration of Prometheus and Grafana:

[filename](prometheus-grafana.sh ':include :type=code')Copy to clipboardErrorCopiedCopy to clipboardErrorCopied

📚 Resources

• Prometheus Documentation
• Grafana Documentation
• HashiCorp Vault Telemetry
• HashiCorp Nomad Telemetry
• HashiCorp Consul Telemetry
• Docker Prometheus Metrics

<!DOCTYPE html>
<html>
<head>
  <title>HashiQube - A Hands on DevOps Development Lab Using All the HashiCorp Products and other Popular Applications such as Docker, Kubernetes, Traefik, Ansible, AWX Ansible Tower and loads more.</title>
  <!-- Google tag (gtag.js) -->
  <script async src="https://www.googletagmanager.com/gtag/js?id=G-LWNG2RV7KC"></script>
  <script>
    window.dataLayer = window.dataLayer || [];
    function gtag(){dataLayer.push(arguments);}
    gtag('js', new Date());

    gtag('config', 'G-LWNG2RV7KC');
  </script>
  <!-- script async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-5855657010614500" crossorigin="anonymous"></script -->
  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
  <meta name="viewport" content="width=device-width,initial-scale=1">
  <meta property="og:url" content="https://hashiqube.com/" />
  <meta property="og:type" content="website" />
  <meta property="og:title" content="HashiQube - The Ultimate Hands on DevOps Lab in a Docker Container" />
  <meta property="og:description" content="HashiQube is a Hands on DevOps Development Lab Using All the HashiCorp Products and other Popular Applications such as Docker, Kubernetes, Traefik, Ansible, AWX Ansible Tower and loads more." />
  <meta property="og:image" content="https://hashiqube.com/images/youtube-hashiqube-the-jedi-devops-lab.png" />
  <meta charset="UTF-8">
  <link rel="icon" type="image/png" href="/images/logo-bright.png">
  <link rel="stylesheet" href="/docsify/lib/vue.css" disabled>
  <link rel="stylesheet" href="/docsify/lib/buble.css" disabled>
  <link rel="stylesheet" href="/docsify/lib/dark.css" disabled>
  <link rel="stylesheet" href="/docsify/lib/pure.css" disabled>
  <link rel="stylesheet" href="/docsify/lib/mermaid.min.css">
  <link rel="stylesheet" href="/docsify/lib/docsify-accordion-style.css">
  <!-- Theme: Simple (latest v0.x.x) -->
  <link
    rel="stylesheet"
    href="/docsify/lib/docsify-darklight-theme-style.min.css"
    title="docsify-darklight-theme"
    type="text/css"
  />
  <script src="/docsify/lib/mermaid.min.js"></script>
  <style>
    /*
    Hashiqube Theme
    Version 0.0.3
    */
    :root {
      --hashi-gradient: 90deg,
        #370960 0%,
        #ea216b 10%,
        #ff5800 20%,
        #fefb00 30%,
        #61e190 40%,
        #5cdeff 50%,
        #61e190 60%,
        #fefb00 70%,
        #ff5800 80%,
        #ea216b 90%,
        #370960 100%;
      --hashi-blue: #5cdeff;
      --hashi-green: #61e190;
      --hashi-yellow: #fefb00;
      --hashi-orange: #ff5800;
      --hashi-pink: #ea216b;
      --hashi-purple: #370960;
      --docsifytabs-border-color: #ededed;
      --docsifytabs-tab-highlight-color: purple;
    }

    .yellow {
      background: var(--hashi-yellow);
    }

    .orange {
      background: var(--hashi-orange);
    }

    .blue {
      background: var(--hashi-blue);
    }

    .green {
      background: var(--hashi-green);
    }

    .purple {
      background: var(--hashi-purple);
    }

    .pink {
      background: var(--hashi-pink);
    }

    .header img {
      width: 80%;
      display: none;
      top: 0px;
      right: 0px;
    }

    .app-name-link img {
      max-width: 160px;
      -webkit-transition-property: all;
      -webkit-transition-duration: 0.3s;
      -webkit-transition-timing-function: ease;
    }

    .app-name-link img:hover {
      transform: scale(1.3);
    }

    .sidebar-nav ul ul,
    .sidebar nav ul ul {
      margin-left: 0px;
    }

    .search .input-wrap {
      margin-top: 10px !important;
    }

    aside.sidebar {
      width: auto;
      display: flex;
      flex-direction: column;
      padding-top: 110px;
    }

    aside.sidebar :nth-child(1) {
      order: 2;
    }

    aside.sidebar :nth-child(2) {
      order: 1;
    }

    aside.sidebar :nth-child(3) {
      order: 3;
    }

    .sidebar-toggle {
      position: fixed !important;
      bottom: auto;
      left: 0px;
      right: 0px;
      background: linear-gradient(var(--hashi-gradient));
      display: block;
      height: 70px;
      padding: 0;
      width: 100% !important;
    }

    .sidebar-toggle-button {
      display: flex;
      width: 70px;
      height: 70px;
      text-align: center;
      flex-direction: column;
      justify-content: center;
      align-items: center;
      background: var(--hashi-blue);
      right: 0;
      top: 0;
      position: absolute;
    }

    .sidebar-toggle span {
      background-color: #000000;
      display: block;
      margin-bottom: 4px;
      width: 20px;
      height: 2px;
    }

    .sidebar ul li a {
      font-size: 18px;
    }

    #docsify-darklight-theme {
      background-color: #000000;
      background-repeat: no-repeat;
      background-size: 30px;
      background-position: center;
      position: fixed;
      right: 140px;
      top: 0;
      width: 70px;
      left: auto;
      height: 70px;
      z-index: 1000;
    }

    .content {
      padding-top: 10px;
      left: 240px;
    }

    .markdown-section {
      padding-top: 110px;
      max-width: 1050px;
    }

    .markdown-section iframe {
      border: 0px solid #eee;
    }

    .markdown-section pre>code,
    .markdown-section code {
      font-size: 1rem;
    }

    .footer {
      position: relative;
      /* display: none; */
      text-align: center;
      bottom: 10px;
    }

    .github-corner svg {
      fill: transparent;
      color: #ffffff;
      padding: 0px;
      margin: 0px;
      margin-left: 0px;
      position: relative;
      right: 15px;
      height: 70px;
      width: 70px;
    }

    .github-corner {
      border-bottom: 0;
      position: fixed;
      right: 70px;
      text-decoration: none;
      top: 0;
      z-index: 1;
      box-sizing: border-box;
      width: 70px;
      height: 70px;
      margin-left: -12px;
      overflow: hidden;
      background: var(--hashi-purple);
      z-index: 100;
      box-sizing: border-box;
      padding: 8px 0 0 5px;
    }

    .progress-bar {
      background: linear-gradient(var(--hashi-gradient));
      height: 7px;
      position: fixed;
      bottom: 0px;
      left: 0px;
      width: 100%;
      padding-bottom: 10px;
    }

    .hasinav {
      position: fixed;
      left: auto;
      right: 210px;
      z-index: 100;
      top: 0;
      height: 70px;
      display: flex;
    }

    .hasinav .qube {
      width: 70px;
      display: inline-flex;
      box-sizing: border-box;
      height: 70px;
      align-items: center;
      justify-content: center;
    }

    .hasinav .qube:hover {
      opacity: 0.8;
    }

    .hasinav .qube img {
      max-width: 30px;
      max-height: 30px;
      width: auto;
      height: auto;
    }
  </style>
<body>
  <!-- Google Tag Manager (noscript) -->
  <noscript>
    <iframe src="https://www.googletagmanager.com/ns.html?id=GTM-MQ56PNF"
    height="0" width="0" style="display:none;visibility:hidden"></iframe>
  </noscript>
  <!-- End Google Tag Manager (noscript) -->
  <div class="header"><img title="HashiQube - DevOps Lab" alt="HashiQube - DevOps Lab" src="/images/hashiqube-banner.png"></div>
  <div class="hasinav">
    <a href="https://www.youtube.com/@hashiqube/featured" target="_blank" class="qube yellow"><img src="/images/youtube-logo.svg" border="0" alt="Youtube Channel" title="Youtube Channel"></a>
    <a href="https://medium.com/search?q=hashiqube" target="_blank" class="qube green"><img src="/images/medium-logo.svg" border="0" alt="Medium Posts" title="Medium Posts"></a>
    <a href="https://www.linkedin.com/in/riaannolan/" target="_blank" class="qube blue"><img src="/images/linkedin-logo.png" border="0" alt="Riaan Nolan Linkedin" title="Riaan Nolan Linkedin"></a>
    <a href="https://www.hashicorp.com/ambassadors/directory?q=riaan" target="_blank" class="qube pink"><img src="/images/hashicorp-logo.svg" border="0" alt="Riaan Nolan Hashicorp Ambassador" title="Riaan Nolan Hashicorp Ambassador"></a>
  </div>
  <div id="app"></div>
  <article class="markdown-section" id="main"></article>
  <script>
    window.$docsify = {
      routerMode: 'history', //hash (default) or history, history mode (SEO friendly) is possible if you host on https://www.netlify.com/ see rewrite rules in _redirects file
      notFoundPage: '/',
      nameLink: '/',
      basePath: '/',
      alias: {
        '/.*/SUMMARY.md': '/SUMMARY.md'
      },
      topMargin: 80,
      logo: '/images/logo-bright.png',
      name: 'HashiQube',
      repo: 'https://github.com/star3am/hashiqube',
      search: 'auto',
      auto2top: true,
      relativePath: false,
      homepage: 'README.md',
      loadSidebar: 'SUMMARY.md',
      subMaxLevel: 3,
      formatUpdated: '{MM}/{DD} {HH}:{mm}',
      plugins: [
        function (hook, vm) {
          hook.beforeEach(function (html) {
            return html.replace(/{% *.* %}/mg, '')
          })
        }
      ],
      markdown: {
        renderer: {
          code: function(code, lang) {
            if (lang === "mermaid") {
              return (
                '<div class="mermaid">' + mermaid.render('mermaid-svg-' + num++, code) + "</div>"
              );
            }
            if (lang === 'drawio') {
              if (window.drawioConverter) {
                console.log('drawio')
                return window.drawioConverter(code)
              } else {
                return `<div class='drawio-code'>${code}</div>`
              }
            }
            return this.origin.code.apply(this, arguments);
          }
        }
      },
      themeable: {
        readyTransition : true,
        responsiveTables: true
      },
      // Style Guide: https://github.com/dracula/dracula-theme
      darklightTheme: {
        siteFont: "PT Sans",
        defaultTheme: 'light',
        codeFontFamily: 'Roboto Mono, Monaco, courier, monospace',
        bodyFontSize: '18px',
        dark: {
          accent: '#8be9fd',
          toggleBackground: '#f8f8f2',
          background: '#240129',
          textColor: '#f8f8f2',
          codeTextColor: '#5cdeff',
          codeBackgroundColor: '#000524',
          borderColor: '#0d2538',
          blockQuoteColor: '#5cdeff',
          highlightColor: '#fefb00',
          sidebarSublink: '#61e190',
          codeTypeColor: '#5cdeff',
          coverBackground: 'linear-gradient(to left bottom, hsl(118, 100%, 85%) 0%,hsl(181, 100%, 85%) 100%)',
          toggleImage: 'url(/docsify/lib/docsify-darklight-theme-sun.svg)'
        },
        light: {
          accent: '#ed2f74',
          toggleBackground : '#091a28',
          background: '#ffffff',
          textColor: '#444444',
          codeTextColor : '#370960',
          codeBackgroundColor : '#f8f8f8',
          borderColor : 'rgba(0, 0, 0, 0.07)',
          blockQuoteColor : '#858585',
          highlightColor : '#d22778',
          sidebarSublink : '#34495e',
          codeTypeColor : '#370960',
          coverBackground: 'linear-gradient(to left bottom, hsl(118, 100%, 85%) 0%,hsl(181, 100%, 85%) 100%)',
          toggleImage: 'url(/docsify/lib/docsify-darklight-theme-moon.svg)'
        }
      },
      tabs: {
        persist: true, // default
        sync: true, // default
        tabComments: true, // default
        tabHeadings: true // default
      },
      // pagination: {
      //   crossChapter: true
      // },
    }

    var num = 0;
    mermaid.initialize({ logLevel:0, startOnLoad: false, themeCSS:'.label { font-family: Source Sans Pro,Helvetica Neue,Arial,sans-serif; }' });

  </script>
  <script src="/docsify/lib/docsify.min.js"></script>
  <script src="/docsify/lib/plugins-emoji.min.js"></script>
  <script src="/docsify/lib/docsify-copy-code.js"></script>
  <script src="/docsify/lib/plugins-search.min.js"></script>
  <script src="/docsify/lib/prism-bash.js"></script>
  <script src="/docsify/lib/prism-yaml.js"></script>
  <script src="/docsify/lib/prism-log.js"></script>
  <script src="/docsify/lib/prism-hcl.js"></script>
  <script src="/docsify/lib/prism-docker.js"></script>
  <script src="/docsify/lib/prism-python.js"></script>
  <script src="/docsify/lib/prism-ruby.js"></script>
  <script src="/docsify/libprism-groovy.min.js"></script>
  <script src="/docsify/libprism-typescript.min.js"></script>
  <script src="/docsify/lib/docsify-darklight-theme-index.min.js"></script>
  <!-- script src="/docsify/lib/docsify-pagination.min.js"></script -->
  <script src="/docsify/lib/docsify-slides.min.js"></script>
  <script src="/docsify/lib/docsify-accordion.js"></script>
  <script src="/docsify/lib/docsify-drawio-viewer.min.js"></script>
  <script src='/docsify/lib/docsify-drawio-drawio.js'></script>
  <script src="/docsify/lib/docsify-tabs.js"></script>
  <div class="footer"></div>
  <div class="progress-bar"></div>
</body>
</html>
Copy to clipboardErrorCopiedCopy to clipboardErrorCopied