LiteSpeed Ingress Controller Metrics¶

The LiteSpeed Ingress Controller contains a built-in Prometheus-compatible exporter which can be made available by just activating it. The exporter not only scrapes the /tmp/lslbd/.rtreports file for useful information, but also adds additional value by resolving the backend IP addresses into their exported host names.

Activating Metrics¶

Metrics control is activated when the LiteSpeed Ingress Controller starts with command line arguments. If you're using Helm, as mentioned in the Controller Configuration, you enable these during the Helm startup by at least specifying --service.metrics.enabled=true:

helm install ls-k8s-webadc ls-k8s-webadc/ls-k8s-webadc -n NAMESPACE --set service.metrics.enabled="true"

This will install the LiteSpeed Ingress Controller with the Prometheus metrics exporter available and its Kubernetes service Annotation set correctly so that an interanl Prometheus node exporter can read the metrics.

The full list of the helm metrics specific parameters are here and the application specific parameters are here.

Using Prometheus' User Interface¶

Many of the metrics require active connectivity to display any value. Keep that in mind when testing or demonstrating the feature and either choose counter type features or run with gauge features when activity is happening.

The best way to install Prometheus is with helm, into the cluster itself.

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm install prometheus prometheus-community/prometheus --set server.service.type=NodePort --set server.service.nodePort=30010

Then you can access the Prometheus server from a web browser from the exposed node, at the exposed node port address of 30010.

Accessing Prometheus in a Kubernetes cluster¶

To then access Prometheus from a web browser, you'll need to find the IP address exposed by the node itself. First get the node name of the running Prometheus server pod:

$ kubectl get pods -o=wide|grep prometheus-server

For example:

prometheus-server-57f4946cdb-x54rh                2/2     Running   0          40h     10.244.3.17      pool-e85y5p9p2-7jkc5   <none>           <none>

In column 7, it reports the node that it is running in pool-e85y5p9p2-7jkc5. Remember that name. Then get the list of external IP addresses for your nodes:

$ kubectl get nodes -o=wide

For example:

NAME                   STATUS   ROLES    AGE   VERSION    INTERNAL-IP      EXTERNAL-IP       OS-IMAGE                       KERNEL-VERSION          CONTAINER-RUNTIME
pool-e85y5p9p2-7jkc5   Ready    <none>   5d    v1.21.14   10.136.121.238   159.223.189.158   Debian GNU/Linux 10 (buster)   5.10.0-0.bpo.15-amd64   containerd://1.4.13
pool-e85y5p9p2-7jkck   Ready    <none>   5d    v1.21.14   10.136.121.235   147.182.135.113   Debian GNU/Linux 10 (buster)   5.10.0-0.bpo.15-amd64   containerd://1.4.13
pool-e85y5p9p2-7jkcs   Ready    <none>   5d    v1.21.14   10.136.121.237   137.184.105.17    Debian GNU/Linux 10 (buster)   5.10.0-0.bpo.15-amd64   containerd://1.4.13

In this example see that the pool pool-e85y5p9p2-7jkc5 has external IP address of 159.223.189.158. Since the helm install command line exports the application on port 30010 on that IP, you can bring up a browser and enter in the url: http://[YOUR IP ADDRESS]:30010, which in this example is: http://159.223.189.158:30010:

To see the list of options, press the icon between the Expression text box and the Execute button, look for entries that begin with litespeed_:

Select one option, for example, on a busy system select litespeed_requests_per_second_per_backend and press the Execute button:

Metrics Exported¶

Overall Metrics¶

The LiteSpeed metrics export includes the following overall metrics. In the .rtreport file, these metrics are at the top and don't repeat. For example:

VERSION: LiteSpeed Load Balancer//3.1.6
UPTIME: 02:52:56
BPS_IN: 0, BPS_OUT: 0, SSL_BPS_IN: 0, SSL_BPS_OUT: 2, L4_BPS_IN: 0, L4_BPS_OUT: 0
PLAINCONN: 1, SSLCONN: 1, L4CONN: 0, IDLECONN: 0

The titles to the table mean:

• Name is the Prometheus name for the metric. Each name will have a litespeed_ prefix.
• Scraped Value is the source from the .rtreport file the value originates from
• Description is a simple description of the meaning of the parameter.
• Type is either Gauge for values which can go up or down or Counter for values which can only go up.

VHost (REQRATE) Metrics¶

The LiteSpeed metrics exported include the following VHost (virtual host) metrics. In the .rtreport file, these metrics repeat and have a REQ_RATE prefix with the first line representing the total and subsequent lines for VHosts which are defined and accessed in the conventional (non-ZeroConf) way; the only VHost which is included in the initial definition is named Example and is generally sparsely used. For example:

REQ_RATE []: REQ_PROCESSING: 1, REQ_PER_SEC: 3.1, TOT_REQS: 7079, PUB_CACHE_HITS_PER_SEC: 0.0, TOTAL_PUB_CACHE_HITS: 0, PRIVATE_CACHE_HITS_PER_SEC: 0.0, TOTAL_PRIVATE_CACHE_HITS: 0, STATIC_HITS_PER_SEC: 0.0, TOTAL_STATIC_HITS: 0
REQ_RATE [Example]: REQ_PROCESSING: 0, REQ_PER_SEC: 0.0, TOT_REQS: 8, PUB_CACHE_HITS_PER_SEC: 0.0, TOTAL_PUB_CACHE_HITS: 0, PRIVATE_CACHE_HITS_PER_SEC: 0.0, TOTAL_PRIVATE_CACHE_HITS: 0, STATIC_HITS_PER_SEC: 0.0, TOTAL_STATIC_HITS: 0, BPS_OUT: 0

Note that in the Prometheus table each VHost, including the overall one will be assigned a separate line; in the graph, each VHost will be assigned a separate color.

Each Prometheus Name will include, besides the litespeed_ prefix, a _per_vhost suffix.

Backend Metrics (EXTAPP)¶

LiteSpeed exports what is prefixed as external application metrics, but which are for Kubernetes actually backends. In Kubernetes, you use kubectl describe ingress INGRESS_NAME -n NAMESPACE to see the host names and the backends for each one. The .rtreport file continually updates statistics for these backends. For example:

EXTAPP [Proxy] [] [10.244.1.137:8080]: CMAXCONN: 100, EMAXCONN: 100, POOL_SIZE: 1, INUSE_CONN: 0, IDLE_CONN: 1, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 51, SESSIONS: 1
EXTAPP [Proxy] [] [10.244.0.114:8080]: CMAXCONN: 100, EMAXCONN: 100, POOL_SIZE: 1, INUSE_CONN: 0, IDLE_CONN: 1, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 67, SESSIONS: 1
EXTAPP [Proxy] [] [127.0.0.1:8088]: CMAXCONN: 200, EMAXCONN: 200, POOL_SIZE: 0, INUSE_CONN: 0, IDLE_CONN: 0, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 414, SESSIONS: 1
EXTAPP [Proxy] [] [10.244.1.134:8080]: CMAXCONN: 100, EMAXCONN: 100, POOL_SIZE: 1, INUSE_CONN: 0, IDLE_CONN: 1, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 43, SESSIONS: 2
EXTAPP [Proxy] [] [10.244.1.247:8080]: CMAXCONN: 200, EMAXCONN: 200, POOL_SIZE: 2, INUSE_CONN: 1, IDLE_CONN: 1, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 51, SESSIONS: 2
EXTAPP [Proxy] [] [10.244.0.19:8080]: CMAXCONN: 100, EMAXCONN: 100, POOL_SIZE: 1, INUSE_CONN: 0, IDLE_CONN: 1, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 31, SESSIONS: 2
EXTAPP [Proxy] [] [10.244.1.223:8080]: CMAXCONN: 0, EMAXCONN: 0, POOL_SIZE: 0, INUSE_CONN: 0, IDLE_CONN: 0, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 118, SESSIONS: 2
EXTAPP [Proxy] [] [10.244.2.112:8080]: CMAXCONN: 100, EMAXCONN: 100, POOL_SIZE: 1, INUSE_CONN: 0, IDLE_CONN: 1, WAITQUE_DEPTH: 0, REQ_PER_SEC: 0.0, TOT_REQS: 39, SESSIONS: 1

The backend IP address is not terribly useful so in the exporter, while handler still displays the backend IP address, hostname displays the exported DNS name:

Note that in the graph, when you mouse-over a line, it displays the details for that particular backend.

Each Prometheus Name will include, besides the litespeed_ prefix, a _per_backend suffix.