Troubleshooting¶

CloudLinux/cPanel Problems¶

If you have experienced an error while using LSMCD Secure User Data in a CloudLinux/cPanel environment, you will probably require administrator intervention. This section assumes that you are either discussing the issue with your administrator now, or are collecting information for later discussion.

Errors Getting Stats¶

There are several types of errors which can occur while getting stats:

• RC: 121 Output: Stats server exception:
• RC: 122 Output: Stats server access error

RC: 121 Output: Stats server exception: (message)¶

The message should be examined in detail to attempt to understand the problem. You may need to check the LSMCD log, usually in /tmp/lsmcd.log. If it is not obvious from examination, contact LiteSpeed Technical Support.

RC: 122 Output: Stats server access error¶

Most of the time this is caused by the LSMCD server not being active. Verify the process lsmcd is running. If it is, you should check the LSMCD log, usually found in /tmp/lsmcd.log. If the service is not running, it can be started from a root console with the following command:

sudo /usr/local/lsmcd/bin/lsmcdctrl start

Other Tools¶

There are other things you can do to validate accessibility to the LSMCD server. All involve running in a console (ssh session) as root:

• Run telnet (TCP) or nc (UDS)
• Check the log
• Get stats manually

Run telnet (TCP)¶

To run telnet to lsmcd, you must not have SASL enabled. You must know the IP address and port configured for LSMCD. For example if you are running on the local server with the default port of 11211:

telnet 127.0.0.1 11211

sudo yum install telnet

Once you have connected, you can run run the stats command to validate that you have proper connectivity. It should report the current stats for the anonymous user.

Run nc (UDS)¶

If you have configured in your node.conf, a Cached.Addr which uses a UDS: prefix then you should use nc to connect to lsmcd rather than telnet. As for telnet this only works if you are not using SASL. You must know the UDS socket name. For example, if you have in your node.conf Cached.Addr=UDS:///tmp/lsmcd.sock you would specify a connection request of:

nc -U /tmp/lsmcd.sock -C

Check the log¶

This should be done for any and all problems. The log is /tmp/lsmcd.log by default. If there is insufficient logging to help you find the problem, you can update node.conf (in /usr/local/lsmcd/conf/node.conf) to set LogLevel=dbg_medium. You will need to restart the LSMCD server.

Manually perform the stats function¶

The stats are obtained by running a script. You can do this manually from the command line. In a default install:

cd /usr/local/cpanel/base/frontend/paper_lantern/lsmcd_usermgr
./lsmcdsasl.py 127.0.0.1:11211 user

Where user is the user you are verifying.

SASL Authentication Errors in PHP including in LSCache for WordPress Plugin¶

If you enable SASL in LSMCD, PHP programs will need to follow the configuration and programming instructions documented in the SASL section.

If you are seeing errors in authentication using the correct user and password you should use the tools described above for problem solving including checking the logs, turning on debugging and the like.

PHP can be a black box but it has an ideal tool to help you figure out problems: phpinfo. You can write a simple phpinfo display by creating the file phpinfo.php in your default docs directory:

<?php
phpinfo();
?>

If you continue to have authentication errors, these can be also caused by missing system sasl plugins, particularly sasl2/libplain.so. In every location where libsasl2.so is installed there will be a directory named sasl2 which will contain the plugins used by SASL. This must contain libplain.so. In cPanel installations, we have seen cases where /opt/alt/cyrus-sasl/lib64 will be missing this plugin which will result in authentication failures. Follow the cPanel instructions for installing cyrus-sasl* libraries which will include libplain.so.

If you have random crashes in LSCache it may make sense to disable SASL altogether. There is a known problem with the libmemcached released with most distros (version 1.0.18) and there is no easy way to upgrade it. For now the best method is to avoid SASL if you are running into this problem.

Other Errors (LSMCD related)¶

Some other errors which may pop up:

• Unrelated messages in the system journal when lsmcd starts up
• Determining if LSMCD is running
• node.conf not found in expected location
• cached.addr not found in
• cached.addr not given a value in
• Error messages in the log indicating a shortage of memory, split memory or other issues.

If any of these errors occur, you should check the LSMCD log. The log is usually in /tmp/lsmcd.log, as specified in the LogFile parameter of the default node.conf file (usually found in /usr/local/lsmcd/conf/node.conf).

It is recommended that you also check the Other Tools in the section above for other troubleshooting ideas.

Unrelated messages in the system journal when lsmcd starts up¶

Your system journal may show messages like this when lsmcd starts:

lsmcd[3327105]: ldapdb_canonuser_plug_init() failed in sasl_canonuser_add_plugin(): invalid parameter su
lsmcd[3327105]: _sasl_plugin_load failed on sasl_canonuser_init for plugin: ldapdb
lsmcd[3327105]: sql_select option missing
lsmcd[3327105]: auxpropfunc error no mechanism available
lsmcd[3327105]: _sasl_plugin_load failed on sasl_auxprop_plug_init for plugin: sql

You can remove those messages by uninstalling cyrus-sasl plugins for ldap and sql (if they are not needed by other applications). Run as root:

yum remove cyrus-sasl-sql
yum remove cyrus-sasl-ldap

Determining if LSMCD is running¶

systemctl is the utility used to access many of systemd service features in modern Linux implementations, and it's a great tool for accessing services.

However, due to problems like the one described above, systemctl start lsmcd may show an error, even if the service is indeed up. The correct way to determine if the LSMCD process is running is to use ps -ef|grep lsmcd.

Similarly, systemctl status lsmcd may show that the service is down when the LSMCD processes is indeed running. This is caused by LSMCD restarting itself and systemd not properly detecting it. Again ps -ef|grep lsmcd should be used to determine if LSMCD is running.

node.conf not found in expected location¶

This message will be followed by the expected fully qualified file name location. It is the conf directory in the startup location for LSMCD, usually /usr/local/lsmcd/conf/node.conf. Verify that node.conf is in this location, as this is the configuration file for LSMCD. If the file does exist in that location or the location is something unexpected, contact LiteSpeed Technical Support.

cached.addr not found in (node.conf file name)¶

This message indicates that in the node.conf file (which should be displayed fully qualified) the parameter title Cached.Addr was not found. This parameter, which is not case-sensitive, is the server address used by external applications to get to LSMCD and is required. Cached.Addr is a title, and all parameters in node.conf are in the form title=value. The form for the line is (for TCP/IP listening)

Cached.Addr=(IP address):(port number)

Cached.Addr=UDS:///(file name)

Cached.Addr=127.0.0.1:11211

cached.addr not given a value in¶

This message is followed by the complete line the parameter is found in, and is then followed by the file name. Verify that Cached.Addr has a value as described above. The default value in the default file is

Cached.Addr=127.0.0.1:11211

Memory errors in the log¶

If you are running into messages indicating a shortage of memory or split memory in the log, the problem may be corrupt shared memory files. Your primary symptom will be failure of applications to correctly use lsmcd. To regenerate the shared memory files:

• Stop the service: sudo /usr/local/lsmcd/bin/lsmcdctrl stop
• Remove the shared memory files: sudo rm -rf /dev/shm/lsmcd
• You should be able to restart the service and all should work correctly: sudo /usr/local/lsmcd/bin/lsmcdctrl start

If you continue to have problems about memory logged you should contact LiteSpeed Technical Support.

Updating¶

One approach to troubleshooting is to update to the latest version of the code. The procedure is:

• Log in as the user who did the original compile and cd to the directory. Download the latest version and extract it as you did in Getting Started. If you used git it is as simple as git pull.
• Build the executable: make.
• Stop the server: sudo /usr/local/lsmcd/bin/lsmcdctrl stop
• Copy the files: sudo make install
• Restart the server: sudo /usr/local/lsmcd/bin/lsmcdctrl start

If you have a problem compiling on a system where the software has been built before, try the following process, logged on as the compiling user and in the directory where you wish to compile:

autoreconf --install --force

Compiling a debug or sanitized version¶

The program can be compiled with full debugging information or with the address sanitizer. The Debugging information is particularly useful if you are going to be actively debugging the code or analyzing a core file with the debugger.

The Google Address Sanitizer build is a release that actively looks for memory issues that can happen in C and C++ programs like lsmcd. It writes these to stderr which is by default, when started with lsmcdctrl written to the lsmcd.log file. In most cases it is difficult to have both core dumps and the address sanitizer.

To compile, you will need to install cmake, libasan and llvm. In RedHat-type systems you install them with the command:

yum install cmake libasan llvm

cmake .
make
sudo make install

The above will build the debug version by default. If you wish to build the Address Sanitizer version, you will need to edit the CMakeLists.txt file and change: set(SANITIZE false) to set(SANITIZE true).

In most cases if you allow a LiteSpeed technician access to your system, you would want the Debug version without the Address Sanitizer. If you can't allow access to your system, then you would want both the Debug and Address Sanitizer enabled and send the /tmp/lsmcd.log file to LiteSpeed support.