Skip to content

Magento 2 Crawler changelog

Note

We do our best to update this changelog in a timely manner. However, if a version is missing, you can always find the release announcement in our LiteSpeed Edge Google Group.

Version 1.9.3

Released: August 15, 2023

  • [Feature] Added support for the -e, --escape option.
  • [Feature] Added support for the -cc, --custom-cookie option.
  • [Improvement] Optimized the response-header filter function.

Version 1.8.0

Released: January 27, 2021

  • [Feature] Added support for the -w, --webp option.
  • [Improvement] Added the -g, --general-ua option for mobile crawling.
  • [Improvement] Added support for sitemap links with private SSL certificates.

Version 1.7.4

Released: July 14, 2020

  • [Improvement] Added support for caching the final URL after a 30x redirect.
  • [Improvement] Displayed the help message more clearly.

Version 1.7.3

Released: October 23, 2019

  • [Feature] Added support for a crawler result report.
  • [Feature] The script now automatically aborts after 10 consecutive page errors or no-cache-detected issues.
  • [Improvement] Added more status codes to error-code detection.
  • [Improvement] Updated the blacklist format to match the entire line.

Version 1.7

Released: June 25, 2019

  • [Feature] Added the ability to automatically add invalid pages to the blacklist and bypass them in future crawls.
  • [Feature] Added debugging methods for URLs with crawling issues.
  • [Improvement] Excluded the Cloudflare cookie from crawl output.
  • [Improvement] Improved compatibility with a server behind LSADC and QUIC.cloud.
  • [Improvement] Retrieved cookie information for the -c, --with-cookie option by using the first valid URL with a random query string.
  • [Improvement] The cURL command now includes HTTP response headers in the output.