Lewati ke isi

prometheus

Prometheus is a free software application used for event monitoring and alerting. It records metrics in a time series database (allowing for high dimensionality) built using an HTTP pull model, with flexible queries and real-time alerting. The project is written in Go and licensed under the Apache 2 License, with source code available on GitHub, and is a graduated project of the Cloud Native Computing Foundation, along with Kubernetes and Envoy.

instalasi prometheus. node exporter. konfigurasi. Securing Prometheus. Securing node_exporter.

Create Prometheus system group

sudo groupadd --system prometheus

sudo useradd -s /sbin/nologin --system -g prometheus prometheus

sudo mkdir /var/lib/prometheus

for i in rules rules.d files_sd; do sudo mkdir -p /etc/prometheus/${i}; done

sudo apt update
sudo apt -y install wget curl vim

mkdir -p /tmp/prometheus && cd /tmp/prometheus

curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi -

tar xvf prometheus*.tar.gz
cd prometheus*/

sudo mv prometheus promtool /usr/local/bin/

Check installed version:

$ prometheus --version
prometheus, version 2.39.1 (branch: HEAD, revision: dcd6af9e0d56165c6f5c64ebbc1fae798d24933a)
  build user:       root@273d60c69592
  build date:       20221007-15:57:09
  go version:       go1.19.2
  platform:         linux/amd64

$ promtool --version
promtool, version 2.39.1 (branch: HEAD, revision: dcd6af9e0d56165c6f5c64ebbc1fae798d24933a)
  build user:       root@273d60c69592
  build date:       20221007-15:57:09
  go version:       go1.19.2
  platform:         linux/amd64

sudo mv prometheus.yml /etc/prometheus/prometheus.yml

sudo mv consoles/ console_libraries/ /etc/prometheus/

cd $HOME

sudo nano /etc/prometheus/prometheus.yml

You can edit the file to your default liking and save it.

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

You can edit the file to your default liking and save it.

Create a Prometheus systemd Service unit file

sudo tee /etc/systemd/system/prometheus.service<<EOF
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP \$MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.external-url=

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target
EOF

Change directory permissions.

for i in rules rules.d files_sd; do sudo chown -R prometheus:prometheus /etc/prometheus/${i}; done
for i in rules rules.d files_sd; do sudo chmod -R 775 /etc/prometheus/${i}; done
sudo chown -R prometheus:prometheus /var/lib/prometheus/

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus

Check status using systemctl status prometheus command:

systemctl status prometheus
● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2020-01-19 14:36:08 UTC; 14s ago
     Docs: https://prometheus.io/docs/introduction/overview/
 Main PID: 1397 (prometheus)
    Tasks: 7 (limit: 2377)
   Memory: 21.7M
   CGroup: /system.slice/prometheus.service
           └─1397 /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.templates

Securing Prometheus (opsional)

sudo apt-get update
sudo apt-get install apache2-utils

sudo htpasswd -c /etc/nginx/.htpasswd sammy

sudo cp /etc/nginx/sites-available/default /etc/nginx/sites-available/prometheus

sudo nano /etc/nginx/sites-available/prometheus

##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# https://www.nginx.com/resources/wiki/start/
# https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/
# https://wiki.debian.org/Nginx/DirectoryStructure
#
# In most cases, administrators will remove this file from sites-enabled/ and
# leave it as reference inside of sites-available where it will continue to be
# updated by the nginx packaging team.
#
# This file will automatically load configuration files provided by other
# applications, such as Drupal or Wordpress. These applications will be made
# available underneath a path with that package name, such as /drupal8.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##

# Default server configuration
#
server {
        listen 80;
        listen [::]:80;

        # SSL configuration
        #
        # listen 443 ssl default_server;
        # listen [::]:443 ssl default_server;
        #
        # Note: You should disable gzip for SSL traffic.
        # See: https://bugs.debian.org/773332
        #
        # Read up on ssl_ciphers to ensure a secure configuration.
        # See: https://bugs.debian.org/765782
        #
        # Self signed certs generated by the ssl-cert package
        # Don't use them in a production server!
        #
        # include snippets/snakeoil.conf;

        root /var/www/html/;

        # Add index.php to the list if you are using PHP
        index index.html index.htm index.nginx-debian.html;

        server_name prometheus.agungsurya.my.id;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                #try_files $uri $uri/ =404;
                auth_basic "Prometheus server authentication";
                auth_basic_user_file /etc/nginx/.htpasswd;
                proxy_pass http://localhost:9090;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
                proxy_set_header Host $host;
                proxy_cache_bypass $http_upgrade;
        }

        # pass PHP scripts to FastCGI server
        #
        #location ~ \.php$ {
        #       include snippets/fastcgi-php.conf;
        #
        #       # With php-fpm (or other unix sockets):
        #       fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
        #       # With php-cgi (or other tcp sockets):
        #       fastcgi_pass 127.0.0.1:9000;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #       deny all;
        #}
}


# Virtual Host configuration for example.com
#
# You can move that to a different file under sites-available/ and symlink that
# to sites-enabled/ to enable it.
#
#server {
#       listen 80;
#       listen [::]:80;
#
#       server_name example.com;
#
#       root /var/www/example.com;
#       index index.html;
#
#       location / {
#               try_files $uri $uri/ =404;
#       }
#}

sudo rm /etc/nginx/sites-enabled/default
sudo ln -s /etc/nginx/sites-available/prometheus /etc/nginx/sites-enabled/

sudo nginx -t

Output of Nginx configuration tests
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

sudo systemctl reload nginx

sudo systemctl status nginx

Downloading Node Exporter

Node Exporter is used for exporting hardware and operating system metrics that are exposed by the Linux kernel to Prometheus. In this section, I am going to show you how to download the latest version of Node Exporter and install it on Ubuntu 20.04 LTS.

wget https://github.com/prometheus/node_exporter/releases/download/v1.1.2/node_exporter-1.1.2.linux-amd64.tar.gz

ls -lh

tar xzf node_exporter-1.1.2.linux-amd64.tar.gz

ls -lh

ls -lh node_exporter-1.1.2.linux-amd64

sudo mv -v node_exporter-1.1.2.linux-amd64/node_exporter /usr/local/bin/

sudo chown root:root /usr/local/bin/node_exporter

node_exporter --version

img

sudo nano /etc/systemd/system/node-exporter.service

[Unit]
Description=Prometheus exporter for machine metrics

[Service]
Restart=always
User=prometheus
ExecStart=/usr/local/bin/node_exporter
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload

sudo systemctl start node-exporter.service

sudo systemctl enable node-exporter.service

Adding Node Exporter to Prometheus

sudo nano /etc/prometheus/prometheus.yml

Add the following lines in the scrape_configs section of prometheus.yml file. Make sure to indent everything correctly to avoid syntax errors.

  - job_name: 'node_exporter'
    static_configs:
    - targets: ['192.168.20.131:9100']

sudo systemctl restart prometheus.service

http://192.168.20.131:9090/targets
  • setelah semua di konfigurasi

example full config prometherus

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
#scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
#  - job_name: "prometheus"

# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.

#    static_configs:
#      - targets: ["localhost:9090"]

#basic_auth_users:
#  alice: $2y$10$mDwo.lAisC94iLAyP81MCesa29IzH37oigHC/42V2pdJlUprsJPze
#  bob: $2y$10$hLqFl9jSjoAAy95Z/zw8Ye8wkdMBM8c5Bn1ptYqP/AXyV0.oy0S8m
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "node_exporter"
    static_configs:
      - targets: ["localhost:9100"]

  #############################
  #blackbox start

  #  - job_name: 'prometheus-black'
  #    static_configs:
  #    - targets: ['localhost:9090', 'localhost:9115']

  #  - job_name: 'blackbox'
  #    metrics_path: /probe
  #    params:
  #      module: [http_prometheus]
  #    static_configs:
  #      - targets: [ "https://google.com/", "https://youtube.com/", "https://500.agungsurya.my.id/" ]
  #        - https://google.com
  #       - https://youtube.com
  #    relabel_configs:
  #      - source_labels: [__address__]
  #        target_label: __param_target
  #      - source_labels: [__param_target]
  #        target_label: instance
  #      - target_label: __address__
  #        replacement: 127.0.0.1:9115  # The blackbox exporter's real hostname:port.

  #blackox end
  #######################

  - job_name: "node-exporter-tls"
    basic_auth:
      username: prometheus
      password: wadawdawdaw=
    static_configs:
      - targets: ["142.202.243.83:12012"]
        labels:
          instance: hostdata-usa

  - job_name: "node-exporter-tls2"
    basic_auth:
      username: prometheus
      password: awdawdawdawd=
    static_configs:
      - targets: ["103.155.250.23:11115"]
        labels:
          instance: whpuls-indo

  - job_name: "node-exporter-tls3"
    basic_auth:
      username: prometheus
      password: Uawdawawd=
    static_configs:
      - targets: ["47.250.47.226:3389"]
        labels:
          instance: alibaba-SG

  - job_name: "network-exporter"
    #basic_auth:
    #  username: prometheus
    #  password: awdawdawd=
    static_configs:
      - targets: ["localhost:9318"]
        labels:
          instance: hostdata-indo

  - job_name: "network-exporter2"
    #basic_auth:
    #  username: prometheus
    #  password: awdadawdawdaw=
    static_configs:
      - targets: ["142.202.243.83:12013"]
        labels:
          instance: hostdata-usa

  - job_name: "node-exporter-tls4"
    basic_auth:
      username: prometheus
      password: awdawdawdwd=
    static_configs:
      - targets: ["localhost:11104"]
        labels:
          instance: indo-lokal
#- job_name: "blackbox-http"
#    metrics_path: "/blackbox-exporter/probe"
#    params:
#      module: [ "http_2xx" ]
#    relabel_configs:
#      - source_labels: [ "__address__" ]
#        target_label: "__param_target"
#      - source_labels: [ "__param_target" ]
#        target_label: "instance"
#      - target_label: "__address__"
#        replacement: "localhost:9115"
#    static_configs:
#      - targets: [ "https://prometheus.io/", "https://grafana.com/

example full config node_exporter (systemmd)

[Unit]
Description=Prometheus exporter for machine metrics

[Service]
Restart=always
User=prometheus
ExecStart=/usr/local/bin/node_exporter --web.listen-address=192.168.20.111:11115 --web.config=/etc/prometheus_node_exporter/configuration.yml
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

example configuration.yml

basic_auth_users:
  prometheus: $2y$10$Fawdwadawdaw.xdawdawwdawd.aSFt.

commands catatan

sudo nano /etc/blackbox/blackbox.yml

sudo kill -HUP 651244

ps aux | grep blackbox

sudo systemctl restart prometheus

sudo systemctl status prometheus

password=`openssl rand -base64 32`
passwordHashed=`echo ${password} | htpasswd -inBC 10 "" | tr -d ':\n'`
echo "Clear password to keep for Prometheus Server: ${password}"

sudo cat << EOF >> /etc/prometheus_node_exporter/configuration.yml
basic_auth_users:
  prometheus: ${passwordHashed}

EOF

Restart node_exporter

sudo systemctl restart node_exporter

The command curl http://localhost:9100/metrics should reply "Unauthorized".

The command curl -u prometheus:${password} http://localhost:9100/metrics should return the list of metrics.

node exporter download wget https://github.com/prometheus/node_exporter/releases/download/v1.1.2/node_exporter-1.1.2.linux-amd64.tar.gz

tar xzf node_exporter-1.1.2.linux-amd64.tar.gz

sudo mv -v node_exporter-1.1.2.linux-amd64/node_exporter /usr/local/bin/

node_exporter --version

sudo nano /etc/systemd/system/node-exporter.service

ExecStart=/usr/local/bin/node_exporter --web.listen-address=172.18.17.59:3389 --web.config=/etc/prometheus_node_exporter/configuration.yml

sudo chown -R prometheus:prometheus /etc/prometheus_node_exporter

https node exporter config example

 - job_name: 'node-exporter-tls3'
    scheme: https
    tls_config:
      insecure_skip_verify: true
      ca_file: tls.crt
    basic_auth:
      username: prometheus
      password: e=
    static_configs:
    - targets: ['47.250.47.226:3389']
      labels:
        instance: alibaba-SG