Here are some answers to common questions about the Serpstatbot in a form of a quick note. You can learn more about Serpstat crawlers in the article or by contacting technical support.
FAQ: Frequently asked questions about Serpstatbot
What is the purpose of Serpstatbot?
Serpstatbot is a crawler that powers the Serpstat Link Analysis module. It constantly scans websites, adding new links to the database and tracking changes on them.
The continuous work of the bot allowed us to collect one of the largest link bases in the niche of SEO tools: 952 billion links across 168 million domains.
Every day the database grows by 2 billion new links. Based on the database, you can control your own link profile or study the link building strategies of competitors.
Every day the database grows by 2 billion new links. Based on the database, you can control your own link profile or study the link building strategies of competitors.
Information about submitted pages is added to the Serpstat link index, this data is available to all tool users to analyze the link profile of the site or competitive sites in the niche.
What happens to submitted pages?
To ensure that the link profile of user sites is not affected by temporary changes, Serpstatbot also captures historical data and rechecks pages with 404 and 301 codes to see if they still exist. Since such pages negatively affect the ranking of the site, it is important to detect and remove them in time. This can be done with a Serpstat website audit. You can also check out Google's 404 page policy.
Are pages with 404 and 301 server response codes fixed?
Yes, the bot scans those links even though they don't affect the page's ranking.
Does Serpstatbot scan links with the rel=nofollow attribute?
How to block Serpstatbot?
Make sure the file is available to the robot, otherwise Serpstatbot will scan the site by default. To learn how to correctly set guidance for robots, read the article.
If you have additional questions or think that the bot does not follow the guidance, please email serpstatbot@serpstatbot.com or contact technical support via chat.
If you have additional questions or think that the bot does not follow the guidance, please email serpstatbot@serpstatbot.com or contact technical support via chat.
To prevent Serpstatbot from scanning your website pages, use the robots.txt file by adding the following text:
User agent: serpstatbot
disallow: /
disallow: /
Why are there issues with Serpstatbot being blocked in robots.txt?
If you are unable to block the bot, it may indicate one of the following errors:
When requesting robots.txt, the bot is redirected to another site: Serpstatbot only follows a redirect on the same domain.
If there are several domains on one server, then some servers can write access logs to these domains in one file without specifying the domain. Add domain information to logs or separate logs for different domains.
What robots.txt commands does Serpstatbot support?
Serpstatbot supports the following additions to robots.txt:
Crawl-delay up to 20 seconds (higher values equal 20 seconds)
Redirect within the same site when trying to get robots.txt
Simple wildcards in Disallow directives, following the Yahoo specification for the use of the "*" character
Allow directives take advantage over Disallow if they are longer
Errors when receiving a robots.txt file, such as 403 Forbidden, are treated as if there were no prohibitions. In this case, the bot will scan all physically accessible pages
Is it possible to slow down Serpstatbot?
The Crawl-Delay value (the length of time to wait between requests in seconds) must be an integer. Serpstatbot takes into account values up to and including 20 seconds. Large Crawl-Delay values will reduce the load on your site. Crawl-Delay will also be taken into account when using User-Agent: *.
If Serpstatbot detects that you have used the Crawl-Delay directive for any other bot, it will automatically slow down the crawl process.
If Serpstatbot detects that you have used the Crawl-Delay directive for any other bot, it will automatically slow down the crawl process.
You can slow down the bot by adding the following directives to the robots.txt file:
User Agent: serpstatbot
Crawl Delay: 5
Crawl Delay: 5
Serpstatbot does not have a set of static IP addresses, but if it is important for you to understand that requests come from our bot, complete the following:
How to check if requests are coming from Serpstat?
After confirming your request, along with bot requests to your site, a special string with identifiers (HTTP headers CRAWLER-IDENT or User-Agent) will come, which will be used only for your domain
Write a request to serpstatbot@serpstatbot.com with information about your site and the identifiers you want to send. If necessary, we can generate a random ID for you
Learn more about the Serpstat link index and its possibilities for building your website promotion strategies. Book a 30-minute demo with our expert.
Use the base of 952 billion links to promote your projects
Personal demonstration
Leave a request, and we will conduct a personal demonstration of the service for you, provide you with a trial period, and offer comfortable conditions for starting exploring the tool