503 Error¶
503 errors are often caused by a malfunction in PHP. It is rare for the web server to be the cause of a 503 error, but just to be safe, you should rule that out first thing by checking whether the error occurs with Apache as well.
Does it Occur with Apache?¶
If your server is running on an Apache control panel, such as cPanel or Plesk, there is an easy way to determine whether an issue is caused by your LiteSpeed server:
- Temporarily switch to Apache
- Repeat the steps that originally led to the issue.
Are you able to reproduce the error under Apache?
NO¶
If the error could not be triggered under Apache, then the problem is likely to be a LiteSpeed server issue. Please open a ticket from your client area or email support@litespeedtech.com
and provide as many details as possible, so that we may assist you.
YES¶
If Apache experiences the same problems, then the issue is not a LiteSpeed Web Server issue.
Switch back to LiteSpeed Web Server, and keep reading! Even though it is not a server problem, we have provided the following troubleshooting documentation that we hope will help you to find the solution.
Tip
Overwhelmed? Don't have the time or interest to deal with these steps? We can help. Engage our team through Hourly Support, and we'll do the troubleshooting for you!
Warning
If you are running LiteSpeed Web Server v5.3 or newer, you will need to restart PHP seperatly from LSWS.
$ sudo touch /usr/local/lsws/admin/tmp/.lsphp_restart.txt
$ sudo systemctl restart lsws
Quick Troubleshooting Guide¶
Before engaging us for Premium Support service, you may like to try a few of these simple steps to see if you can fix the error yourself:
- Check the
phpinfo
page of the problematic user account. (Create aphpinfo.php
file if nophpinfo
page exists.) - Generally, the server's error_log or
stderr.log
should provide some hints as to the problem. For example, it may be a memory issue. - Disable opcode cache for the PHP version in use and verify from the
phpinfo
page. - Disable unsafe PHP extensions such as ZendGuardLoader, Suhosin, ionCube, etc.
- Check disk space.
- Try increasing the PHP memory limit (you can verify from
phpinfo.php
) and lsphp external app memory limit. - If you're using CloudLinux:
- Check the LVE memory limit to see if it needs to be increased.
- Check the LVE process limit to see if it needs to be increased.
- Try to force an update of CageFS.
These are just a few simple quick fixes for you to try first. If they don't help, you can refer to the following log files for detailed steps/reasons, or engage our support as mentioned earlier.
Check Log Files¶
There are many different log files that may help you determine the cause of the 503 error. In this section we will discuss the different log files, their locations, and what information they provide.
When troubleshooting 503 errors it is ideal to check these logs in the following order:
Standard Error Log stderr.log
¶
The stderr.log
file contains standard errors generated while PHP is running. It can usually be found in either the /var/log/apache2/
or/usr/local/lsws/logs/
directory. This log is usually the most helpful when determining why PHP is throwing 503s.
Example
Example output from a standard error log: vi /var/log/apache2/stderr.log
[STDERR] fork() failed, please increase process limit: Cannot allocate memory
Web Server Error Log¶
This error log is generated by the web server. It can usually be found in either the /var/log/apache2/
or /usr/local/lsws/logs/
directory. The web server error log may provide helpful hints as to whether the web server caused PHP to fail.
Example
Example output from a web server error log:
[INFO] [319934] [1.1.1.1:49873-1#APVH:lsapi] connection to [uds://tmp/lshttpd/APVH-php56.sock] on request #0, confirmed, 0, associated process: 0, running: 0, error: Too many open files!
The following very important command should give you ideas on how many 503 errors have been generated and wheather there are any new 503 errors since your last attempted fix by checking the timestamp. However, most of the time, the error log doesn't give you the reason why the 503 error happened. It only shows you when it happened and with which domain it happened.
grep oops /etc/apache2/logs/error_log
System Log¶
This error log is generated by the operating system. It can usually be found as either /var/log/messages
or /var/log/syslog
. This log is helpful if PHP is being affected by the operating system.
Example
Example output from a system log: grep lsphp /var/log/messages
lfd[18304]: *User Processing* PID:18264 Kill:1 User:xxxxx VM:538(MB) EXE:/usr/local/lsws/fcgi-bin/lsphp-5.4.42 CMD:lsphp5
PHP Error Log¶
This error log is defined in php.ini and you can check phpinfo.php for the php error_log defined. By default, cpanel will set it to error_log
. It is generated by PHP when it detects an error, warning, etc., depending on your log level in PHP. It is usually generated inside of the same folder that the PHP script runs in.
Example
Example output from a PHP error log: vi home/mysite/public_html/error_log
Fatal error: Call to undefined function my_function() in /home/mysite/public_html/test.php on line 2
These types of errors are usually found inside of the PHP application you are trying to run. It is best to forward these logs to the developer of the PHP application.
Troubleshooting¶
Basic¶
Display Errors¶
In a production environment it is common to disable PHP application errors from displaying in order to secure the server in the event of a issue. This however is counter productive when trying to diagnose 503 errors.
If you are experiencing 503 errors for a specific site/domain, try adding the following right after the opening <?php
tags, to the same script with the 503 error:
<?php # The opening PHP tag
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
When you visit the web page now, it should show an error like this:
Fatal error: Call to undefined function my_function() in /home/mysite/public_html/test.php on line 2
Normally this has nothing to do with the web server and is actually an issue with the code. We suggest checking Stack Overflow or contacting the developer of the software for a fix.
Note
After you record the error, be sure to revert the file back to its previous state in order to keep your server secure.
PHP Info Page¶
The PHP info page gives us a lot of detail about your PHP installation. This page is a very basic file that should not throw a 503 error.
<?php
echo phpinfo();
?>
Note
If you do get a 503 error with a PHP info page you should skip this section.
Here is a list of some of the useful information the PHP info page gives us:
- Extensions
- Memory Limit
- Execution Time
- Other Settings
With this information, you can see if OPCache is enabled, what the memory limit is, which extensions are loaded, etc; all useful information when looking for the cause of a 503 error.
This page can also be used to compare differences between Apache's and LiteSpeed's PHP implementations.
Note
If the Apache and LSWS PHP info pages are different, please open a ticket with us and we will take a look.
Extensions¶
These are a handful of PHP extensions that have been known to cause issues, especially when they are outdated:
- Imunify360
- ZendGuardLoader
- Suhosin
- ionCube
We recommended you regularly check for updates for these extensions. If you are already using the latest versions, try to disable them and see if the problem persists.
If you disable these plugins and the 503 error is still being shown, disable all other extensions that were not enabled by default, and check again.
OPCache¶
OPCache is the process of storing bytecode in shared memory so PHP does not need to process the same calls over and over, but it can occasionally cause issues. When troubleshooting 503 errors, it is best to have OPCache disabled.
Common OPCache Extensions:
- OPCache
- APC/u
- XCache
- eAccelerator
To disable the OPCache extensions follow the extensions guide.
Error
Here is an example of an error that points to OPCache:
[STDERR] zend_mm_heap corrupted
Disk Space¶
PHP will sometimes store files on your server in order to function properly. If your disks become full, especially the partition that houses /tmp/
, it could be the cause of your 503 error.
To check your disk space usage you can run $ sudo df -h
in a terminal.
Example
Here is an example output of the $ sudo df -h
command:
$ sudo df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 901M 0 901M 0% /dev
tmpfs 915M 96K 915M 1% /dev/shm
tmpfs 915M 89M 827M 10% /run
tmpfs 915M 0 915M 0% /sys/fs/cgroup
/dev/mapper/cl_centos 29G 4.2G 25G 15% /
/dev/vda1 976M 184M 726M 21% /boot
tmpfs 183M 0 183M 0% /run/user/0
If you have full disks, clean up unused and old files. This should solve the 503 error.
Memory Limit¶
When PHP reaches its allocated memory limit, it will crash. There are 2 different memory limits that PHP must adhere to: its internal memory limit, set by php.ini
, and LiteSpeed's memory limit, set inside of the web server for when the PHP process spawns.
Example
These are examples of errors correlating to memory issues:
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 12864 bytes)
[STDERR] fork() failed, please increase process limit: Cannot allocate memory
PHP Memory¶
If you get a memory error like the one above, the first step in troubleshooting it is to increase the memory limit. Adjust the following setting inside of php.ini
:
memory_limit = 100M
To figure out where your php.ini
file is located, and find out your current memory limit, you can create a PHP info page.
<?php
echo phpinfo();
?>
Sometimes scripts may append lines to .htaccess
to override the PHP settings. This can override what is set in your php.ini
file.
Here is an example of a memory_limit
entry inside of .htaccess
:
php_value memory_limit xxxM
You can add/delete/modify this line inside of your .htaccess
file to suit your needs.
CloudLinux Memory Limit¶
If you are running CloudLinux and Lightweight Virtual Environment (LVE) it is possible that you are hitting the limits set by that account.
Here shows accounts that have hit a limit, depending on what limit the account has hit you should adjust the limit. To adjust the limits follow CloudLinux's guide.
litespeed PHP Process Memory¶
If you have increased the memory limit in php.ini
and confirmed you are not hitting a LVE limit then it is possible LSPHP needs to increase its virtual memory.
- Go to your Web Admin console (
https://YOUR_SERVER_IP:7080
) and log in. - Next, navigate to Configuration > Server > PHP.
- Now click Edit for PHP Handler Defaults.
- Change the following settings, then click Save.
- Memory Soft Limit:
4097M
- Memory Hard Limit:
4098M
- Memory Soft Limit:
- Now that everything has been set up, navigate to Actions and press the icons next to Restart Detached PHP Processes. Then Apply Changes / Graceful Restart.
Max Execution Time¶
If you have a max execution time that is not high enough to handle long running requests then you may get the following error in your web server error log: error: Connection reset by peer!
and have a value of 0
, meaning that the process never finished the request. These errors will often be explained in your stderr.log. If the stderr.log does not provide any insight then you can try to increase the max execution time by following the steps in the Control guide.
PHP suEXEC Max Conn¶
In stderr.log, you may see something like the following:
[30117] Reached max children process limit: 10, extra: 3, current: 13, busy: 13, please increase LSAPI_CHILDREN.
PHP SuEXEC Max Conn
is 10
, but the domain needs 13
PHP processes. To increase LSAPI_CHILDREN, go to LSWS WebAdmin > Server > General > PHP suEXEC Max Conn, and increase it to a higher number. In this case, let's say 20
. Restart LSWS and pkill lsphp
so the new setting takes effect. If the errors persist you can continue increasing PHP suEXEC Max Conn until you have enough to stop the errors. CSF/LFD¶
If you have CSF/LFD installed they may be causing the 503 error. These programs may be killing the process before it finishes. CSF/LFD has two settings that can kill a LSPHP process: PT_USERTIME
and PT_FORKBOMB
.
Kill¶
If a LSPHP process takes too long to complete it is possible that the PT_USERTIME
feature will be called to kill the process. If it does kill the process you may receive an email or an entry in /var/log/lfd.log
with the following message:
lfd[18304]: *User Processing* PID:18264 Kill:1 User:xxxxx VM:538(MB) EXE:/usr/local/lsws/fcgi-bin/lsphp-5.4.42 CMD:lsphp5
This means that we need to whitelist the LSPHP process from CSF/LFD to prvent this from happening.
To do this, run the following commands in a terminal:
$ sudo echo "pexe:/usr/local/lsws/fcgi-bin/lsphp.*" >> /etc/csf/csf.pignore
$ sudo csf -r; sudo service lfd restart
Forkbomb¶
In rare cases, LSPHP will be killed by PT_FORKBOMB
for having too many processes spawned at any given time. This will, in return, end up sending the signal 9/SIGKILL to the main Litespeed process. To stop csf/lfd from killing the LSPHP and LiteSpeed processes, the following settings inside of /etc/csf/csf.conf
should be adjusted:
PT_LIMIT - Default: 10
PT_USERPROC - Default: 10
PT_FORKBOMB - Default: 0
PT_INTERVAL - Default 60
These settings should be adjusted on a per case basis as each scenario is different. They should be increased slightly higher than the current values until the issue stops occurring. For the changes to take effect, both csf and lfd need to be restarted. This can either be done via the WHM plugin or through a shell terminal by using the following command:
$ sudo csf -r; sudo service lfd restart
Extension Mismatch¶
Running LSPHP extensions that are not built for the version of LSPHP you are using may cause a 503 error. If this is the case, you will see the following messagin inside of a stderr.log
:
Warning: PHP Startup: imap: Unable to initialize module
Module compiled with module API=20090626
PHP compiled with module API=20100525
These options need to match in Unknown on line 0
To fix this, you must rebuild the module and/or PHP, making sure to use a version of PHP that works with the module. Also, make sure that the correct extension path is used.
Extension Loading Order¶
With PHP, some extensions require other extensions be loaded first in order to work efficently. This is accomplished by changing the extensions's .ini
file from mysql.so
to 20-mysql.so
. The 20-
value is the loading priority. It can be changed from 01-
to 99-
, with lower values loading before the higher values. You can find more information on the PHP site.
If this is the cause of the issue, then you should see the following entry in your stderr.log
.
PHP Warning: PHP Startup: Unable to load dynamic library
Open Files Limit¶
If you see the following entry inside of your web server error long, then your system is maxed out of the amount of files it can open.
[INFO] [319934] [1.1.1.1:49873-1#APVH:lsapi] connection to [uds://tmp/lshttpd/APVH-php56.sock] on request #0, confirmed, 0, associated process: 0, running: 0, error: Too many open files!
To check your current limit, run the following command:
$ sudo ulimit -n
It will show you your system's current limit. To fix this issue, increase the limit by running:
$ sudo ulimit -n <Increased_Number>
Now you should no longer see a 503 error because LSPHP can open more files on your system.
Resource temporarily unavailable¶
If you are using WordPress, you may see this error generated when working with images or graphics:
libgomp: Thread creation failed: Resource temporarily unavailable
memory_limit
in php.ini
or the lsphp external app memory limit to a higher value such as 8G. Or try both. These measures may fix the issue. PHP code issue¶
Bad PHP code or a buggy script may create too many PHP connections or a loop event, which can cause 503 errors. You can strace the PHP process to find out more, and then optimize your PHP code or script.
Advanced¶
The comprehensive guide above lists the most likely reasons you'd receive a 503 error, but in some circumstances, it may be difficult to locate the root cause. If PHP crashes, you can enable a core dump and use GDB to debug the core file. Even if you find an internal PHP bug causing the crash, there are limited actions you can take. You can disable the module or change to another PHP version without the bug. Reporting the bug to PHP is an option, but a solution may not be provided in a timely manner.
If there is a core file generated and you've tried all of the above methods without finding a solution, try to strace the PHP process for a hint. Just be prepared that the strace log may be extremely long and overwhelming. You likely need some knowledge and experience to understand what the log entries mean; it can even be a challenge locating the relevant sections of the log.
Strace¶
Use the following command to strace the PHP process to find out what happened:
strace -tt -T -f -p <pid>
This command will show you what PHP is doing and where the issues is.
Example
For example, inside the web server log you may see the following log entry:
[INFO] [1.1.1.1:48506] connection to [/tmp/lshttpd/APVH_vhost_Suphp.sock] on request #0, confirmed, 1, associated process: 791228, running: 1, error: Connection reset by peer!
Here we can take the process ID from associated process: 791228
and then run strace using the following command:
strace -tt -T -f -p 791228
Tip
Sometimes, it may not be easy to get the pid since PHP may be running quickly. In this case, try to strace the parent pid. The disadvantage of stracing the parent pid is you may get extra unrelated information in the strace log, which will make it difficult to locate the problem pattern.
One solution is using a script to catch the pid. It depends on which SuEXEC user your PHP will be running and which script will be running (such as index.php etc).
while true; do if mypid=`ps aux | grep $USERNAME | grep lsphp | grep $SCRIPTNAME | grep -v grep | awk '{print $2; }' | tail -1`; then strace -tt -T -f -p $mypid; fi ; done
Note: $USERNAME
should be replaced with a real username; $SCRIPTNAME
should be replaced by the name of the script that is running, such as index.php.
Core Files¶
If you find that PHP has crashed (as demonstrated by the process being killed by signal
other than value 15
), a core dump will allow you to look further into the cause of the crash. These core files are usually generated inside of the folder that PHP crashed with.
Enable Core Files¶
This section assumes you have core dumps enabled for centOS or Ubuntu already.
- Go to your Web Admin console (
https://YOUR_SERVER_IP:7080
) and log in. - Next, navigate to Configuration > Server > PHP.
- Now click Edit for PHP Handler Defaults.
- Change the following settings then click Save.
- Environment:
LSAPI_ALLOW_CORE_DUMP=1
- Environment:
- Now that everything has been set up, navigate to Actions and press the icons next to Restart Detached PHP Processes. Then Apply Changes / Graceful Restart.
Analayze Core Files¶
GNU Debugger (GDB) uses the syntax gdb <path/to/lsphp/binary> <path/to/core/file>
. The <path/to/lsphp/binary>
is the path to the PHP binary that the PHP application was using. The path to the core file is the location of the core file.
After running the above command, you will see a prompt with (gdb)
then you will want to type bt
to view the backtrace then hit enter
.
This will generate output like the following:
gdb <path/to/lsphp/binary> <path/to/core/file>
Program received signal SIGSEGV, Segmentation fault.
0x000000000061c91b in ?? ()
(gdb) bt
#0 0x000000000061c91b in ?? ()
#1 0x0000000000641ac3 in zend_stack_push ()
#2 0x000000000060bbf9 in ?? ()
#3 0x0000000000611515 in lex_scan ()
#4 0x000000000061fb60 in ?? ()
#5 0x0000000000608223 in ?? ()
#6 0x0000000000614965 in compile_file ()
#7 0x00007fd3c99eda21 in ?? () from /opt/alt/php55/usr/lib64/php/modules/phar.so
#8 0x00007fd3d0993359 in ?? () from /opt/alt/php55/usr/lib64/php/modules/opcache.so
#9 0x00007fd3d0994187 in ?? () from /opt/alt/php55/usr/lib64/php/modules/opcache.so
#10 0x00000000006b54da in ?? ()
#11 0x00000000006b68e8 in execute_ex ()
#12 0x000000000064239c in zend_execute_scripts ()
#13 0x00000000005e2fb0 in php_execute_script ()
#14 0x00000000006f29ff in ?? ()
#15 0x00000000006f2c5c in ?? ()
#16 0x00000000006f2f85 in ?? ()
#17 0x00007fd3d37cdd1d in __libc_start_main () from /lib64/libc.so.6
#18 0x0000000000424bc9 in _start ()
Here we can see that the some of the modules that loaded when it crashed were phar.so
and opcache.so
. To try to troubleshoot this, you would want to disable phar.so
and opcache.so
to figure out which module is causing the issue.
Advanced
It is possible to dig deeper into the core file and figure out which function is exactly causing the issue.
To figure this out run the following:
(gdb) frame 13
(gdb) print (char *)(executor_globals.function_state_ptr->function)->common.function_name
(gdb) print (char *)executor_globals.active_op_array->function_name
(gdb) print (char *)executor_globals.active_op_array->filename
It if fine for these to return blank, just continue running them and it should help you to track down which file/function is causing the issue.
For more information on debugging PHP core files check the document here.
Auto Fix 503¶
Warning
This feature does not FIX 503 errors. It is a mask for 503 errors. Please follow the following steps to figure out the issue!
In LiteSpeed Web Server's WebAdmin Console, there is a feature to Auto Fix 503 Error. It specifies whether to try to fix the 503 Service Unavailable
error by restarting the server gracefully. A 503 error is usually caused by malfunctioning external applications and a web server restart can often fix the error temporarily. If Auto Fix 503 Error is set to Yes
, the server will restart automatically whenever there are more than thirty 503 errors within a 30 second span. This feature is disabled by default, but you can enable it by navigating to LSWS Admin > Server > General and setting Auto Fix 503 Error to Yes
.
Auto Fix 503 Error is not a permanent fix. You should not rely on this feature to fix recurring 503 errors. Going through the above steps to find a permanent fix is always recommended.