Skip to content

LiteSpeed Ingress Controller Metrics

The LiteSpeed Ingress Controller contains a built-in Prometheus-compatible exporter which can be made available by just activating it. The exporter not only scrapes the /tmp/lslbd/.rtreports file for useful information, but also adds additional value by resolving the backend IP addresses into their exported host names.

Since the exporter requires a program, like Prometheus itself to request these parameters, Prometheus can also be installed by activating that as well.

Activating Metrics

Metrics control is activated when the LiteSpeed Ingress Controller starts with command line arguments. If you're using Helm, as mentioned in the Controller Configuration, you enable these during the Helm startup by at least specifying --enable-metrics=true:

helm install ls-k8s-webadc ls-k8s-webadc/ls-k8s-webadc -n NAMESPACE --set extraArgs="{enable-metrics=true,install-prometheus=true}"

The full list of metrics specific parameters are here. Most users will not change most of the port parameters, however, some parameters that might be changed more frequently include:

  • metrics-evaluation-interval and metrics-scrape-interval both default to 1m (1 minutes). Note that the load balancer writes the file every 10 seconds and this is the most often you can expect modified metrics to be available, so setting these values to 10s will get you the most current data.
  • prometheus-port. The default of 9090 makes the Prometheus program available outside the Kubernetes cluster. If you want Prometheus run, but the statistics only collected to be consumed within the cluster, set this value to 0.

Using Prometheus' User Interface

Many of the metrics require active connectivity to display any value. Keep that in mind when testing or demonstrating the feature and either choose counter type features or run with gauge features when activity is happening.

To access the Prometheus panel enter the current LiteSpeed Ingress Controller IP Address at the Prometheus port (for example, 123.456.789.123:9090) which will bring up the Prometheus entry screen:

Prometheus Entry Screen

To see the list of options, press the icon between the Expression text box and the Execute button:

Prometheus LiteSpeed Metrics Options List

Select one option, for example, on a busy system select litespeed_requests_per_second_per_backend and press the Execute button:

Prometheus Statistics

Metrics Exported

Overall Metrics

The LiteSpeed metrics export includes the following overall metrics. In the .rtreport file, these metrics are at the top and don't repeat. For example:

VERSION: LiteSpeed Load Balancer//3.1.6
UPTIME: 02:52:56
BPS_IN: 0, BPS_OUT: 0, SSL_BPS_IN: 0, SSL_BPS_OUT: 2, L4_BPS_IN: 0, L4_BPS_OUT: 0
PLAINCONN: 1, SSLCONN: 1, L4CONN: 0, IDLECONN: 0

The titles to the table mean:

  • Name is the Prometheus name for the metric. Each name will have a litespeed_ prefix.
  • Scraped Value is the source from the .rtreport file the value originates from
  • Description is a simple description of the meaning of the parameter.
  • Type is either Gauge for values which can go up or down or Counter for values which can only go up.
Name Scraped Value Description Type
litespeed_current_http_connections PLAINCONN Current number of http connections Gauge
litespeed_current_idle_connections IDLECONN Current number of idle connections Gauge
litespeed_current_l4_connections L4CONN Current number of layer 4 connections Gauge
litespeed_current_ssl_connections SSLCONN Current number of https connections Gauge
litespeed_exporter_scrapes_failures_total - The number of failed scrapes. Counter
litespeed_exporter_scrapes_total - The total number of scrapes. Counter
litespeed_incoming_http_bytes_per_second BPS_IN Incoming number of bytes per second over HTTP Gauge
litespeed_incoming_l4_bytes_per_second L4_BPS_IN Incoming number of bytes per second over layer 4 Gauge
litespeed_incoming_ssl_bytes_per_second SSL_BPS_IN Incoming number of bytes per second over HTTPS Gauge
litespeed_outgoing_http_bytes_per_second BPS_OUT Outgoing number of bytes per second over HTTP Gauge
litespeed_outgoing_l4_bytes_per_second L4_BPS_OUT Outgoing number of bytes per second over layer 4 Gauge
litespeed_outgoing_ssl_bytes_per_second SSL_BPS_OUT Outgoing number of bytes per second over HTTPS Gauge
litespeed_up - Whether LiteSpeed is up or down (1 or 0) Gauge
litespeed_version VERSION Returns whether LiteSpeed is up or down and the version field returns the text "LiteSpeed Load Balancer//3.1.6" Gauge

VHost (REQRATE) Metrics

The LiteSpeed metrics exported include the following VHost (virtual host) metrics. In the .rtreport file, these metrics repeat and have a REQ_RATE prefix with the first line representing the total and subsequent lines for VHosts which are defined and accessed in the conventional (non-ZeroConf) way; the only VHost which is included in the initial definition is named Example and is generally sparsely used. For example:

REQ_RATE []: REQ_PROCESSING: 1, REQ_PER_SEC: 3.1, TOT_REQS: 7079, PUB_CACHE_HITS_PER_SEC: 0.0, TOTAL_PUB_CACHE_HITS: 0, PRIVATE_CACHE_HITS_PER_SEC: 0.0, TOTAL_PRIVATE_CACHE_HITS: 0, STATIC_HITS_PER_SEC: 0.0, TOTAL_STATIC_HITS: 0
REQ_RATE [Example]: REQ_PROCESSING: 0, REQ_PER_SEC: 0.0, TOT_REQS: 8, PUB_CACHE_HITS_PER_SEC: 0.0, TOTAL_PUB_CACHE_HITS: 0, PRIVATE_CACHE_HITS_PER_SEC: 0.0, TOTAL_PRIVATE_CACHE_HITS: 0, STATIC_HITS_PER_SEC: 0.0, TOTAL_STATIC_HITS: 0, BPS_OUT: 0

Note that in the Prometheus table each VHost, including the overall one will be assigned a separate line; in the graph, each VHost will be assigned a separate color.

Each Prometheus Name will include, besides the litespeed_ prefix, a _per_vhost suffix.

Name Scraped Value Description Type
litespeed_current_requests_per_vhost REQ_PROCESSING Current number of requests in flight Gauge
litespeed_outgoing_bytes_per_second_per_vhost BPS_OUT Current number of bytes per second outgoing. Only available for configured VHosts Gauge
litespeed_private_cache_hits_per_second_per_vhost PRIVATE_CACHE_HITS_PER_SEC Private cache hits per second Gauge
litespeed_private_cache_hits_per_vhost TOTAL_PRIVATE_CACHE_HITS Total private cache hits Counter
litespeed_public_cache_hits_per_second_per_vhost PUB_CACHE_HITS_PER_SEC Public cache hits per second Gauge
litespeed_public_cache_hits_per_vhost TOTAL_PUB_CACHE_HITS Total public cache hits Counter
litespeed_requests_per_second_per_vhost REQ_PER_SEC Requests per second Gauge
litespeed_static_hits_per_second_per_vhost STATIC_HITS_PER_SEC Static file requests per second Gauge
litespeed_static_hits_per_vhost TOTAL_STATIC_HITS Total number of static file hits Counter
litespeed_total_requests_per_vhost TOT_REQS Total number of requests Counter

Backend Metrics (EXTAPP)

LiteSpeed exports what is prefixed as external application metrics, but which are for Kubernetes actually backends. In Kubernetes, you use kubectl describe ingress INGRESS_NAME -n NAMESPACE to see the host names and the backends for each one. The .rtreport file continually updates statistics for these backends. For example:

EXTAPP [Proxy] [] [10.244.1.137:8080]: CMAXCONN: 100, EMAXCONN: 100, POOL_SIZE: 1, INUSE_CONN: 0, IDLE_CONN: 1, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 51, SESSIONS: 1
EXTAPP [Proxy] [] [10.244.0.114:8080]: CMAXCONN: 100, EMAXCONN: 100, POOL_SIZE: 1, INUSE_CONN: 0, IDLE_CONN: 1, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 67, SESSIONS: 1
EXTAPP [Proxy] [] [127.0.0.1:8088]: CMAXCONN: 200, EMAXCONN: 200, POOL_SIZE: 0, INUSE_CONN: 0, IDLE_CONN: 0, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 414, SESSIONS: 1
EXTAPP [Proxy] [] [10.244.1.134:8080]: CMAXCONN: 100, EMAXCONN: 100, POOL_SIZE: 1, INUSE_CONN: 0, IDLE_CONN: 1, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 43, SESSIONS: 2
EXTAPP [Proxy] [] [10.244.1.247:8080]: CMAXCONN: 200, EMAXCONN: 200, POOL_SIZE: 2, INUSE_CONN: 1, IDLE_CONN: 1, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 51, SESSIONS: 2
EXTAPP [Proxy] [] [10.244.0.19:8080]: CMAXCONN: 100, EMAXCONN: 100, POOL_SIZE: 1, INUSE_CONN: 0, IDLE_CONN: 1, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 31, SESSIONS: 2
EXTAPP [Proxy] [] [10.244.1.223:8080]: CMAXCONN: 0, EMAXCONN: 0, POOL_SIZE: 0, INUSE_CONN: 0, IDLE_CONN: 0, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 118, SESSIONS: 2
EXTAPP [Proxy] [] [10.244.2.112:8080]: CMAXCONN: 100, EMAXCONN: 100, POOL_SIZE: 1, INUSE_CONN: 0, IDLE_CONN: 1, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 39, SESSIONS: 1

The backend IP address is not terribly useful so in the exporter, while handler still displays the backend IP address, hostname displays the exported DNS name:

Prometheus Statistics

Note that in the graph, when you mouse-over a line, it displays the details for that particular backend.

Each Prometheus Name will include, besides the litespeed_ prefix, a _per_backend suffix.

Name Scraped Value Description Type
litespeed_config_max_connections_per_backend CMAXCONN Configured maximum number of connections Gauge
litespeed_connections_idle_per_backend IDLE_CONN Number of idle connections Gauge
litespeed_connections_in_use_per_backend INUSE_CONN Number of connections in use Gauge
litespeed_current_sessions_per_backend SESSIONS Current number of sessions Gauge
litespeed_pool_count_per_backend POOL_SIZE Total number of pools Gauge
litespeed_pool_max_connections_per_backend EMAXCONN Maximum number of connections for the pool Gauge
litespeed_requests_per_second_per_backend REQ_PER_SEC Number of requests per second Gauge
litespeed_total_requests_per_backend TOT_REQS Total number of requests Counter
litespeed_wait_queue_depth_per_backend WAITQUE_DEPTH Depth of the waiting queue Gauge

Using Prometheus to push to Grafana Cloud

One of the reasons to install Prometheus is to use it to export the scraped metrics to Grafana Cloud. You will need an account in Grafana Cloud and a free one works just fine. To get the details:

  • Login to grafana.com
  • In the Prometheus box, select the Details button
  • It will give you the Remote Write Endpoint, Username and give you the opportunity to create an API Key with at least Metrics-Publisher permission. You should note the key as well (it's very long and generally needs to be saved at creation time).
  • Add the following metrics specific arguments to your helm --set extraArgs="{}" command which will include:
Name Description Value
--prometheus-remote-password The prometheus remote_write password. Often your Grafana Prometheus Metrics API Key. none
--prometheus-remote-url The prometheus remote_write url. Often your Grafana Prometheus Metrics service. none
--prometheus-remote-user The prometheus remote_write username. Often your Grafana Prometheus Metrics username (a number). none

Troubleshooting Prometheus

Prometheus writes it's errors to the program's stdout. At startup, this is redirected to /tmp/prom.err in the LiteSpeed Ingress Controller pod. All troubleshooting should start by examining that log. Like any other file in the Kubernetes system, use kubectl cp to copy the file into your environment. For example to copy it to your current directory:

kubectl cp -n NAMESPACE POD:tmp/prom.err prom.err

Last update: June 21, 2022