cgroups

If you are a web server administrator and your users are creating CGI services that are chewing up your server's CPU, memory, or tasks, you have a powerful tool to deal with it in LiteSpeed Web Server: cgroups.

"cgroup" is short for "control group". cgroups is a Linux kernel feature that, according to Wikipedia, "limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes."

The feature is part of the Linux Operating System and can be activated and utilized with version 6.0 or higher of LiteSpeed Web Server.  Previous versions used an updated v1 cgroups which will no longer be supported in the future. cgroups v2 requires a kernel 5.2 or higher and some additional configuration discussed below.

cgroups v2 has a much richer set of features supported for control than cgroups v1 as discussed here. The features that have been certified to work with cgroups are:

• CPU utilization
• real memory
• maximum tasks

Things to Know About cgroups¶

cgroups is only available for true CGI applications at this time.  Below we discuss the details for enabling it, but it works on the operating system resource definitions for the owner of the CGI script file.

Containers (like Docker containers) are operating system facilities that run within the base OS.  Containers may use cgroups to avoid completely dominating the machine, and they can only do this if using cgroups. This is because cgroups is one of the few methods that can be used to address overuse of resources without killing the processes (unlike ulimit and other limiting schemes).  cgroups helps you better balance the use of your system, keeping it active and functional for all of your users.

cgroups affects more than just LiteSpeed Web Server, since it is a kernel feature. It also affects tasks running as specific users that start their own user sessions, like SSH or SFTP sessions.  Thus, if you set a CPUQuota of 10% for a user, all accesses by the user must not exceed 10%. This includes those in LiteSpeed Web Server and user login sessions. All will be shared fairly by the operating system.  If you start more processes, the total percentage for the user does not change, but each process runs a bit slower so as to avoid monopolizing the CPU.

Supported cgroup Features¶

The following are cgroup facilities supported by the operating system, tested, and certified to work with LiteSpeed Web Server.  The names specified below are the ones that you will use in systemctl to modify them:

1. CPUQuota
2. TasksMax
3. MemoryMax

CPUQuota¶

This is specified as a percentage of a single processor to use.  By default, the CPUQuota of all tasks is infinite, and explains why a single user's task can pretty much monopolize the CPU resources of your system.  This is by far the most useful of the supported features as CPU affects not only processor I/O but also access to buffers in buffered I/O.  ToLSWS like top help you monitor CPU utilization for a task and display the user and process that are running.  Note that values above 100% are valid and indicate that multiple CPUs are to be allowed.

TasksMax¶

TasksMax refers to the number of tasks that can be started by a user. This ensures that the number of tasks accounted for the user stays below a specific limit. This takes an absolute number of tasks for the user on the system.  A sample default used by the system is 12000 and can result in a very large number of tasks, each of which is managed by the OS.  A reasonable value might be 2500.

MemoryMax¶

This is the maximum amount of real memory allocated to a process, specified in bytes.  It is not the amount of virtual memory that a process uses.  Specify the limit on maximum memory usage of the executed processes. The limit specifies how much process and kernel memory can be used by tasks in this unit. If the value is suffixed with K, M, G or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively.  Remember that this is not virtual memory (which is stored both on disk and in real memory), but real memory.  Note that real memory affects virtual memory and that tasks that take too much of it can weigh a system down.  A reasonable value might be 10M.

Using cgroups with LiteSpeed Web Server¶

Step 1. System Configuration¶

To verify that you are running cgroups v2 check to see if you have the cgroups v2 controllers file:

ls /sys/fs/cgroup/cgroup.controllers

If this file is not there then you are not running v2 cgroups.

By default only a limited number of commands can be controlled through cgroups. You can see this list:

cat /sys/fs/cgroup/cgroup.subtree_control

If this list does not include the features you wish to control: cpuset and cpu for CPUQuota, memory for MemoryMax and tasks for MaxTasks, then you will need to add them. If these features are already included, you can skip this step and proceed to User Configuration below.

Extend the set of commands by creating a drop-in file:

    mkdir -p /etc/systemd/system/user@.service.d
    cat > /etc/systemd/system/user@.service.d/delegate.conf << EOF
    [Service]
    Delegate=cpu cpuset io memory pids
    EOF
    systemctl daemon-reload

Step 2. User Configuration¶

User configuration is within the operating system.  What's nice about the systemd programs loginctl and systemctl is that configuration is applied immediately and preserved on reboot.

You will need to identify the users you want to limit and obtain their userids.  The /etc/passwd file contains a list of users and passwords, though there are many ways of identifying IDs for a user.  Note that all configuration will need to be done either running as root, or by using the sudo command as a prefix for your command.  The notes below will show the entries as if you were running as root, add the sudo prefix if you need to on your system.

Using systemctl¶

systemctl is the primary tool for accessing services, slices and units on a machine.  It is accessed from a terminal session.  You can see its options by using:

systemctl --help

You may have noted that this is the same command used to start and stop services on your system.  This is not an accident; these facilities were initially intended for services and have been extended by kernel developers for users.

systemctl requires that you specify the user using the userid (uid) in the form user-UID.slice, where UID is replaced by the userid. 

systemctl show user-1001.slice

The titles and values for the properties are mentioned above (CPUQuota=xx%, TasksMax=xx, and MemoryMax=xx).  To set a property like CPUQuota to 10% for user2 (uid=1001), enter:

systemctl set-property user-1001.slice CPUQuota=10%

Properties are set in the system and take effect immediately for future processes.  Current ones are not affected, as mentioned above, they are also saved and may work on reboot.  If you don't want them saved, you can avoid this by adding the modifier --runtime after systemctl and before the command.  For example:

systemctl --runtime set-property user-1001.slice CPUQuota=10%

To see its current value, enter (as you did above):

systemctl show user-1001.slice

Its current value will be buried within all of the other parameters, but it will be there.  Except CPUQuota, which is listed in a systemctl show command as CPUQuotaPerSecUSec (USec means microseconds, but it's displayed in milliseconds).  10% is displayed as 100ms.

You can see where the information you update gets stored by using systemctl with the status command and the user slice.  For example:

$ systemctl status user-1001.slice
● user-1001.slice 
  Loaded: loaded 
 Drop-In: /etc/systemd/system/user-1001.slice.d 
          └─50-CPUQuota.conf, 50-TasksMax.conf 
  Active: active since Wed 2019-01-23 15:56:52 EST; 1 day 22h ago 
   Tasks: 0 (limit: 1000)

You will need to repeat the steps above for all of the users you wish to control.

Step 3. LiteSpeed Web Server Setup.¶

In the WebAdmin Console, configure native LiteSpeed Web Server so that CGI apps use the user ID of the owner of the file.  Navigate to Configuration > Virtual Hosts and select the virtual host you wish to manage.  In the Basic tab, modify the Security entry and set the ExtApp Set UID Mode to CGI File UID.  Save the entry.

To enable cgroups at the server level, in LiteSpeed Web Server WebAdmin, navigate to Server Configuration > Security and edit the CGI Settings group.  The parameter cgroup contrLSWS whether cgroups can be used and if so, sets the default value:

• Off (the same as not set):  Allows cgroups to be used by LiteSpeed Web Server, but does not enable cgroups by default. cgroups must be explicitly enabled at the virtual host level (see below).
• On:  Allows cgroups to be used by LiteSpeed Web Server and enables cgroups by default. cgroups can be disabled at the virtual host level.
• Disabled:  No matter the setting at the virtual host level, the cgroups feature is disabled.

Save the group to proceed.  If you do not want to set cgroups at the virtual host level, do a graceful restart of LiteSpeed Web Server to activate your configuration and proceed to Step 3.

Optional Virtual Host Configuration¶

If you have not disabled cgroups at the server level and want to further control them at the vhost level, navigate to Configuration > Virtual Hosts and select the virtual host you wish to manage.  In the General tab, modify the General entry and set cgroups:

• Off: Disables cgroups for this vhost, regardless of the server setting.
• On: Enables cgroups for this vhost if not disabled at the server level.

Save the group to proceed.  Do a graceful restart of LiteSpeed Web Server to activate your configuration and proceed to Step 3.

Modifying the Configuration in Files¶

This is only for reference if you're using the WebAdmin Console.

In $SERVER_ROOT/conf/httpd.config.conf there is a single parameter in the CGIRLimit section with parameters like procHardLimit:

• cgroups:  The possible values of 0, 1, and 2 correspond to Off, On, and Disabled, respectively, as described above.

You can specify specific virtual hosts to enable/disable at the virtual host level unless you have disabled it entirely.  That is done in the $SERVER_ROOT/conf/vhosts/<virtual host>/vhconf.conf file. 

• cgroups:  Set to 1 to override the default and turn cgroups on for the virtual host, and 0 to override the default and turn cgroups off for the virtual host.

You will need to gracefully restart LiteSpeed Web Server to activate the configuration.

Configuring Apache Virtual Hosts¶

The previous instructions refer to native LiteSpeed Web Server. To enable or disable cgroups for an Apache virtual host, use LsCgroups with a value of either off or on in the vhost's Apache configuration file.

Step 4. Validation¶

After configuring users and LiteSpeed Web Server to use cgroups, you want to make sure that they are applied correctly.  Configure a user to use a small amount of CPU (like 10%).  Then create a CGI file in your virtual host directory which is owned by your configured user (with a command like chown).  Below is an example.

Name the file $SERVER_ROOT/<virtual host>/cgi-bin/pound and change it's owner to be user2:

#!/bin/sh

date=date -u '+%a, %d %b %Y %H:%M:%S %Z'
results=/usr/bin/dd if=/dev/zero of=/dev/null 
cat << EOF
Content-type: text/plain
Expires: $date

$results

EOF

chown user2:users pound

When you specify the site in your browser: http://127.0.0.1/cgi-bin/pound the dd command should be owned by user2 and it should take only 10%.  You can view this with the top program and kill it there.  If there are problems you should check the $SERVER_ROOT/logs/error.log file to see if there are any cgroup messages.

Other Useful Tools¶

There are other useful tools which can help you understand how configuration is implemented in your system:

• systemd-cgls -a:  Enter this command to display a tree-like structure of the processes and under which slice and scope they are running.  The dd command in the pound example above should be running under user-1001.slice.
• The /proc/<process-id>/cgroup file:  In the 1:name line you should see the slice that the process is running in.

Problems With cgroups in LiteSpeed Web Server¶

Since enabling and disabling cgroups is so easy in LiteSpeed Web Server, almost all problems will be in the systemctl configuration. We recommend that you create a simple program like the pound program above and that you use top to verify the results. If they show 100% CPU after setup you should verify the following:

• Use systemctl show user-$UID.slice where $UID is your user's UID. Look for the following properties:
  • If you are controlling CPU: CPUAccounting=yes and CPUQuotaPerSecUSec set to a non-infinite value.
  • If you are controlling Tasks: TasksAccounting=yes and TasksMax set to the value you specified. This can be a bit difficult to test as you need to create a program that generates a lot of tasks and we do not provide one.
  • If you are controlling Memory: MemoryAccounting=yes and MemoryMax set to the value you specified, but in bytes. Note that this is for real memory and can't be identified easily in testing.
• Use systemctl status user-$UID.slice where $UID is your user's UID. Look to verify that the status is loaded and that in the log under the status there aren't any errors that refer to the drop in files. Note that it may show an Active state of inactive and there may be red colored messages in the log entries below the status that are not drop in file errors and can generally be ignored.
• If the user information is not taking effect after verifying the above, make sure that you are running LSWS v6.2.2 or above (OLS 1.8.1 or above) as a linger requirement was removed with that release.