|How-to||– 9 min read –||September 24, 2019|
What pages should be closed from indexing
Reasons to close pages from indexing
Let's consider the reasons why you should remove indexing from the website or individual pages:
Such content can include technical and administrative website pages, as well as personal data information. Besides, some pages can give the illusion of duplicate content, which is a violation and may lead to penalties on the entire resource.
A crawling budget is a certain number of website pages that a search engine can crawl. We are interested in spending server resources only on valuable and high-quality pages. In order to quickly and efficiently index important resource content, you need to close unnecessary content from crawling.
What pages should be removed from indexing
Website pages under development
In fact, the printed page is a copy of its main version. If this page is open for indexing, the search robot can choose it as a priority and consider it more relevant. To properly optimize a website with a large number of pages, you should remove indexed pages for printing.
To close the link to the document, you can use the content output using AJAX, close the pages using the meta tag <meta name = "robots" content = "noindex, follow" />, or close all indexed pages from indexing in robots.txt.
Perhaps the content of these files doesn't meet the needs of the target audience of the website. Or documents appear in the search results above the html pages of the website. In this case, indexing documents is undesirable, and it's better to close them from crawling in the robots.txt file.
User forms and elements
Website technical data
Personal customer information
How to close pages from indexing
Robots meta tag with noindex value in html file
When you are using this method, the page will be closed for crawling even if there are external links to it.
To close the text from indexing (or a separate piece of text), and not the entire page, use the html tag: <noindex> text </noindex>.
You can limit the indexing of pages through the robots.txt file in the following way:
.htaccess configuration file
Removing URLs through Webmaster Services
Restricting access to certain pages and documents will save search engine resources and speed up indexing of the entire website.
This article is a part of Serpstat's Checklist tool
|Try Checklist now|
Learn how to get the most out of Serpstat
Want to get a personal demo, trial period or bunch of successful use cases?
Send a request and our expert will contact you ;)
Cases, lifehacks, researches and useful articles
Don’t you have time to follow the news? No worries!
Our editor Stacy will choose articles that will definitely help you with your work. Join our cozy community :)