Basic monitoring configuration
The common ReportPortal instance consists of two main parts – the application server and the database server – both of which must be covered by basic system-level and application-level metrics. Basic system-level monitoring should include metrics tracking the main application and database servers' VM and cluster resources, such as:
- Disk Operations,
- Storage Usage.
Extended system-level monitoring for the application instance should include:
- CPU and RAM utilization per each service,
- MinIO storage usage trend,
- Restarts count per each service,
- Disk Bytes,
- CPU usage in cores per each service,
- Network IO per each service,
- RabbitMQ runtime metrics.
In the case of rollouts of the ReportPortal PostgreSQL database as an Amazon RDS, the AWS RDS Performance Insights is used for monitoring purposes and CloudWatch metrics are used to measure basic system-level database workload.
For regular monitoring, we've chosen one of the most popular, lightweight, and easy-to-maintain tools stack - Telegraf+InfluxDB+Grafana. Telegraf is an open-source server agent which enables us to use a lot of different monitoring protocols out of the box. InfluxDB is used to collect these metrics from Telegraf agents. Grafana is used for metrics visualization, building charts, and custom dashboards to reach a "single pane of glass" monitoring principle. The PostgreSQL database in all cases is monitored by AWS RDS Performance Insights, which gives high visibility into the Database workload in runtime, meets all monitoring requirements, and covers OS system-level and database-specific metrics. In addition to that, for the basic database monitoring is used PGHero tool - useful monitoring dashboard for PostgreSQL.
Not recommended mixing the ReportPortal services and monitoring services on the same machine, especially in docker installation. To avoid competition for resources with the services of the ReportPortal, deploy a separate virtual machine for monitoring (for our instances we using m5.large shape for the monitoring node) and install the following services:
1) InfluxDB database: https://docs.influxdata.com/influxdb/v2.0/install/?t=Docker;
2) Grafana: Dashboard examples(Grafana IDs): 5955, 3056.
- Kubernetes deployment: telegraf agent needs to be installed at each cluster node.
- Docker deployment: telegraf agent should be installed at the ReportPortal VM. In case of rollouting the Database server on the separate VM, telegraf agent should be installed also at that VM.
Telegraf installation guide:
Update your system.
sudo yum -y update
Add Influxdata RPM repository.
cat <<EOF | sudo tee /etc/yum.repos.d/influxdb.repo
name = InfluxDB Repository - RHEL
baseurl = https://repos.influxdata.com/rhel/7/x86_64/stable/
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
Install Telegraf on RHEL 8 / CentOS 8. Once the repository has been added, install Telegraf on RHEL 8 using the command below.
sudo dnf -y install telegraf
Open "telegraf.conf" file for the monitoring configuration. In the case of Kubernetes deployment need to configure telegraf on each cluster node separately.
sudo nano /etc/telegraf/telegraf.conf
Change following configs(press Ctrl+W to search for the particular configs):
hostname = "api_node_1"
Search for the "outputs.influxdb" and update URL and database name for the InfluxDB:
urls = ["http://<influxdb_host>:8086"]
database = "telegraf"
Search for the "inputs.docker" and update next configs(should be uncommented each value which you need to add to the monitoring):
endpoint = "unix://var/run/docker.sock"
perdevice_include = ["cpu"]
total_include = ["cpu", "blkio", "network"]
Search for the "inputs.net" for adding the network metrics to the monitoring. Uncomment only the plugin name:
Save changes, close "telegraf.conf" and start telegraf service:
sudo systemctl enable --now telegraf
Check the status(should be green and in active running status):
sudo systemctl status telegraf
In case of errors "E! [inputs.docker] Error in plugin: Got permission denied while trying to connect to the Docker daemon socket at unix://var/run/docker.sock Permission denied"
Need to add permissions to the /var/run/docker.sock:
sudo chmod 666 /var/run/docker.sock
PGHero - simple monitoring dashboard for PostgreSQL
- requests statistic: number of calls, average and total execution time (with the ability to store history);
- currently active queries;
- information about tables: disk space occupied, dates of the last VACUUM and ANALYSE launches;
- information about indexes: disk space occupied, presence of duplicated / unused indexes;
- recommendations regarding adding an index on complex queries with Seq Scan;
- statistics on open connections to the database;
- displaying basic database settings that affect performance (shared_buffers, work_mem, maintenance_work_mem, etc.).
How to enable query stats
In the database settings(for RDS database - in parameter group) add/change the following parameters:
shared_preload_libraries = 'pg_stat_statements'
pg_stat_statements.track = all
pg_stat_statements.max = 10000
track_activity_query_size = 2048
Restart the database or reboot the RDS instance. As a superuser from the psql console, run:
CREATE extension pg_stat_statements;
How to configure historical query stats
To track query stats over time, create a table to store them.
Execute the following query for table creation:
CREATE TABLE "pghero_query_stats" (
"id" bigserial primary key,
Build index on the created table:
CREATE INDEX ON "pghero_query_stats" ("database", "captured_at");
Include the following to the installation string to schedule the task to run every 5 minutes:
The query stats table can grow large over time. Remove old stats with: