What is Deindexing?

Deindexing

What is Deindexing?

Deindexing is the practice of removing a webpage or website from search engine result pages (SERPs). This process can be initiated by either the website owner or the search engine.

Reasons for Deindexing

Initiated by Bloggers:
- Bloggers or website owners may deindex content they do not want to appear on search engine result pages, such as low-quality or private pages like thank you pages, landing pages, author pages, and login pages.
Initiated by Search Engines:
- Search engines generally deindex content that violates their guidelines. Google, for instance, deindexes sites engaging in activities against its spam policies after a manual review confirms the violation.

Why Will Google Deindex My Site?

Google will deindex your site if you engage in practices that go against its spam policies. These policies are guidelines that site owners must follow to ensure their content appears on Google Search result pages.

How to Deindex Content from Search Results Pages

Using the Noindex Meta Tag

The noindex meta tag prevents webpages from appearing in search results. To deindex a webpage, add the following noindex meta tag to the head tag of the page’s HTML code:

<meta name="robots" content="noindex">

This is how the code looks when added to the head tag of the site’s HTML code:

<!DOCTYPE html>
<html>
  <head>
    <meta name="robots" content="noindex">
    (…)
  </head>
  <body>
    (…)
  </body>
</html>

You can also instruct specific search engine crawlers not to index the webpage by replacing robots with the name of the crawler bot:

<meta name="googlebot" content="noindex">

Using the X-Robots-Meta Tag

The X-Robots-Meta tag is useful for blocking webpages and non-HTML elements like images, videos, and PDF files from appearing in search results. You can block a webpage or resource from appearing in search results using the following X-Robots-Meta tag:

X-Robots-Tag: noindex

This tag is configured on the server side, either through server configuration files or using a server-side scripting language like PHP:

<?php 
header('X-Robots-Tag: noindex'); 
?>

Deindexing in WordPress

WordPress allows you to discourage search engines from indexing your site. To do this, navigate to WordPress Dashboard → Settings → Reading, enable Discourage search engines from indexing this site, and click Save Changes.

To request Google to recrawl and deindex your site, go to Google Search Console, enter your site URL into the URL Inspection field, and click REQUEST INDEXING.

Frequently Asked Questions

Can I Add the Noindex Meta Tag to the robots.txt File?
- The robots.txt file does not support the noindex meta tag. Use the noindex meta tag in the head tag of the HTML code instead.
Should I Block the Webpage in the robots.txt File?
- Do not block the webpage in the robots.txt file. Google should be able to access and crawl the webpage to properly process the noindex directive.
What Happens if I Add the Noindex Rule on a Webpage Over a Long Period?
- Google will stop crawling any webpage that has been declared noindex over a long period of time, even if a follow attribute is added.
Can I Deindex Duplicate Pages?
- Do not deindex duplicate pages. Instead, use the rel="canonical" link element to specify the original version of the webpage.

Conclusion

Deindexing is a valuable practice for managing the visibility of your webpages in search engine result pages. By understanding the reasons and methods for deindexing, you can ensure that only the most relevant and high-quality content from your site is indexed and displayed by search engines.

Deindexing