Posts on ToPass

Pushing metrics to Prometheus using Pushgateway

Sun, 07 Aug 2022 00:00:00 +0000

Introduction

Prometheus

Prometheus is a metrics engine that ingest data and lets query them in order to get data aggregations and statistics. It stores metrics inside a time series database and persists them on local disk. Through the dashboard, we can get all the values collected by Prometheus and makes different aggregations.

The Prometheus server, which acts as the system’s “brain” by collecting various metrics and storing them in a time-series database. Unlike other systems monitoring tools that wait passively to receive information, Prometheus actively scrapes metrics from applications and services (e.g. Docker containers) at designated intervals using the Prometheus client libraries.

Grafana

Beside of Prometheus, for displaying metrics and data in a human-readable format. In addition to native Prometheus visualization, many Prometheus users integrate the tool with Grafana, an open-source web application for analytics and data visualization.

Grafana enables you to query, visualize, alert on, and explore your metrics, logs, and traces wherever they are stored. Grafana provides you with tools to turn your time-series database (Prometheus in this case) data into insightful graphs and visualizations.

Pushgateway

As discussed above, Prometheus scrapes data from your applications and services at regular intervals. but what if those intervals are still too slow to capture valuable information? Many short-lived and batch jobs complete so quickly that their lifespans are shorter than the Prometheus scrape interval itself, making it tricky or impossible to capture metrics from them at runtime.

Or maybe what if the source doesn’t have a valid address to scrape by prometheus, then we will need to send metrics from the source instead.

Prometheus Pushgateway sounds like a tremendously useful tool for capturing data from jobs that are difficult to scrape.

Setup using Docker Compose

💡 You can find the source code in the Github repository

Create the working directory

1
2


mkdir ./prometheus-stack
cd ./prometheus-stack

Create the docker-compose.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


version: '3.7'

volumes:
 prometheus-data:
 grafana-data:

services:
 ###########################################################
 ## PROMETHEUS
 ###########################################################
 prometheus:
 image: prom/prometheus:v2.30.3
 volumes:
 - ./prometheus:/etc/prometheus
 - prometheus-data:/prometheus
 command: --web.enable-lifecycle --config.file=/etc/prometheus/prometheus.yml
 ports:
 - 3000:9090
 ###########################################################
 ## Push Gateway
 ###########################################################
 pushgateway:
 image: prom/pushgateway:v1.4.3
 container_name: pushgateway
 restart: unless-stopped
 ports:
 - 3001:9091
 ###########################################################
 ## GRAFANA
 ###########################################################
 grafana:
 image: grafana/grafana-enterprise:8.2.0
 environment:
 - GF_SECURITY_ADMIN_PASSWORD=strong-password
 volumes:
 - grafana-data:/var/lib/grafana
 ports:
 - 3002:3000
 

Create the Promethus config file

1
2


mkdir promethus
cd promethus

And make this file within the directory:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# prometheus/prometheus.yml
global:
 scrape_interval: 10s
 scrape_timeout: 10s

scrape_configs:
 - job_name: services
 metrics_path: /metrics
 static_configs:
 - targets:
 - pushgateway:9091

Deploy the docker-compose.yml file

1
2


docker-compose pull
docker-compose up -d

Now you must have the enpoints up and running. Lets test and make sure:

Sending metrics to Pushgateway

We can send metrics to Pushgateway using http requests. for a simple one we can POST to the endpoin in this format:

1

http://localhost:3001/metrics/job/<JOB_NAME>{/<LABEL_NAME>/<LABEL_VALUE>}

Using cURL

1

echo "temp_celsius 24.5\n humidity_ratio 0.3" | curl --data-binary @- http://localhost:3001/3001/metrics/job/job_01/site/greenhouse_01

Using Axiso in Javascript

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


import axios from 'axios';

const res = await axios({
 url: 'http://localhost:3001/3001/metrics/job/job_01/site/greenhouse_01',
 method: 'POST',
 data: `
 # Temperature
 temp_celsius 12
 # Percent
 humidity_ratio 0.3
 `,
});

Done 🙂

Windows installation TODO list for developers

Mon, 04 Jul 2022 00:00:00 +0000

Fix Git “End Of Line” issue

The culprit is git, a configuration property of core.autocrlf

For historical reasons, the line breaks of the text file on windows and linux are different.

Windows At the time of line break, carriage return is used at the same time CR(carriage-return character) and line breaks LF(linefeed character)
Mac and Linux only use the newline character LF
Old version of Mac uses carriage return CR Therefore, there will be incompatibility problems when text files are created and used in different systems.

When I clone code on Windows, autocrlf is true as default and then each line of the file is automatically converted to CRLF. If you do not make any changes to the file, eslint delete CR by pre-commit since git automatically convert CRLF to LF.

So, after global configuration, you need to pull the code again.

1

git config --global core.autocrlf false

Done 🙂

Self hosted Sentry updating TODO list

Wed, 22 Jun 2022 00:00:00 +0000

In addition to making its source code available publicly, Sentry offers and maintains a minimal setup that works out-of-the-box for simple use cases.

Sentry encourage everyone to regularly update their Sentry installations to get the best and the most recent Sentry experience.

Create a backeup/snapshot

Make Sure you have a quickly revertable backup/snapshot.

Download the closer update

You have to update your Sentry one by one, so you can’t upgrade from 20.x to 22.x or some thing.

Also read the release notes carefully. some times there are more steps before updating to a release:

https://github.com/getsentry/self-hosted/releases

Supposing the next one is 21.9.0:

1

wget -qO- https://github.com/getsentry/self-hosted/archive/refs/tags/21.9.0.tar.gz | tar xvz

And go to the directory:

1

cd ./self-hosted-21.9.0

Copy the config/env files to the new version directory

These are the config files. It’s not bad to compare the old and new config files before moving, to make sure the formats are same.

sentry/config.yml
sentry/sentry.conf.py

Also you should update the values of the env file. Note that you shouldn’t copy entire values of the prev values, while it includes the docker image tags too.

.env

Run the installation script

1

sudo bash ./install.sh

Start the services

1

docker-compose up -d

Done 🙂

Dokku Installation TODO list

Sat, 18 Jun 2022 00:00:00 +0000

Prepare the requirements

At least 1 GB of system memory
A fresh installation of Ubuntu 18.04/20.04/22.04
A domain pointing to the server ip will allow to add git remotes dokku.my-domain.com.
A wildcard domain *.dokku.my-domain.com will allow access to apps via $APP.dokku.my-domain.com.

Installation

1
2


wget https://raw.githubusercontent.com/dokku/dokku/v0.27.5/bootstrap.sh;
sudo DOKKU_TAG=v0.27.5 bash bootstrap.sh

Post installation

Allow authorized ssh keys to access dokku too

1

cat ~/.ssh/authorized_keys | dokku ssh-keys:add admin

Set global base domain

1

dokku domains:set-global dokku.my-domain.com

Letsencrypt TLS certificates

dokku-letsencrypt is the official plugin for dokku that gives the ability to automatically retrieve and install TLS certificates from [letsencrypt.org](https://letsencrypt.or During ACME validation, your app will stay available at any time.

1

sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git

Set global email

1

dokku config:set --global DOKKU_LETSENCRYPT_EMAIL=your@email.tld

Enable automatic certificate renewal

1

dokku letsencrypt:cron-job --add

Postgres database setup

Installation

1

sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git postgres

Creating the service

1

dokku postgres:create my-database

Expose the port

Expose a postgres service on custom host:port to have access from out world:

1

dokku postgres:expose my-database 5432

You can now find the user/pass/post of the datbase this way:

1

dokku postgres:info my-database

Deployment

Creating an app

1

dokku apps:create my-app

Set env variables

1

dokku config:set my-app JWT_SECRET=jwt-secret JWT_EXPIRES_IN=1d NODE_ENV=production ...

Link postgres database to the app

1

dokku postgres:link my-database my-app

This will add the proper DATABASE_URL to the env variables of my-app.

Add a git remote

1

git remote add dokku dokku@dokku.may-domain.com:my-app

Push to Dokku

1

git push dokku

Set the proxy port (for the Dockerised apps)

1

dokku proxy:ports-add my-app http:80:3000

Enable Letsencrypt

1

dokku letsencrypt:enable my-app

Done 🙂

How to update Gitlab Runners?

Thu, 26 May 2022 00:00:00 +0000

Introduction

GitLab Runner is an application that works with GitLab CI/CD to run jobs in a pipeline. GitLab Runner is open-source and written in Go. It can be run as a single binary; no language-specific requirements are needed.

How to update?

Stop the service

1

sudo gitlab-runner stop

Download the binary to replace the GitLab Runner executable

We can see the list of all releases in the Official Gitlab Runner repository.

In this case I had an Ubuntu machine and wanted to update to the latest v14.x machine. so the proer version is v14.8.3.

1

sudo curl -L --output /usr/local/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/v14.8.3/binaries/gitlab-runner-linux-amd64"

Setting permissions to execute

1

sudo chmod +x /usr/local/bin/gitlab-runner

Starting the service

1

sudo gitlab-runner start

Done 🙂