Fixing common crawl errors in Google Search Console

This report shows you when Google is having problems crawling a URL on your site that they previously indexed.

Example of an error report in Google Search Console.

Crawl errors can be broken down into a number of different types.

Server Error

This means Google experienced an issue while crawling your site because of your server. Google will usually tell you what error code they received from the server. These are usually transient errors and go away, but if you get a lot of these errors, then this could indicate a server problem. In this case, you should contact your hosting company and let them know about the error codes Google is reporting.

Noindex Error

If you’re seeing this error, then Google encountered a URL in your sitemap which is tagged as “noindex”. This value tells Google and other search engines that you do not want the page to be indexed.

If this is your intention, then you can safely ignore the error. Navigate to the relevant page and make sure to check the “Exclude From Sitemap” option in the All in One SEO Pack meta box so that the URL is no longer included in your sitemap.

If you however do want this page to be indexed, then you need to remove the “noindex” meta tag in either the General Settings or meta box (depending on whether you’ve enabled noindexing on a global scale for the relevant post type or only for that specific post). In a future release, we’ll exclude noindexed pages from the sitemap automatically.

Access Denied

This means Google couldn’t crawl your website because their user-agent was blocked. Make sure you’ve configured your robots.txt file in the Robots.txt module and make sure no firewall is blocking Google or server authentication.

404/Not Found Errors

The bulk of the errors you may see, and we get asked about, are 404 or so called “Not Found” errors. This means that Google has crawled the URL in the past but now when they crawl it they get a 404/Not Found error page because the URL no longer exists. Check each error and determine why Google cannot crawl the URL. Some common reasons are:

  • You removed that content and it’s not coming back. Use the “Remove URLs” tool in the older version of Google Search Console (the new version does not include this tool yet) to remove the URL from their index. You also might want to verify that it is no longer included in your sitemap; otherwise Google will continue throwing errors.
  • You removed that content and replaced it with other content. Create a 301 redirect from the old URL to the new URL using the Redirection tool under Tools > Redirection.
  • The URL or your permalink structure changed. Create a 301 redirect from the old URL to the new URL using the Redirection tool under Tools > Redirection.
  • It’s an incorrect link from someone else’s site. You cannot control how others link to your site, so if someone links to a page that doesn’t exist or typed the URL in wrong then you should create a 301 redirect from the bad URL to the correct URL (using the Redirection tool under Tools > Redirection)
  • There’s an incorrect link on your own site.

Make sure you fix crawl errors as soon as they are reported as they may have a significant influence on organic traffic to your site. If you use All in One SEO Pack to generate your XML Sitemap and you’re using the default “Dynamically Generate Sitemap” setting, then you can be assured that Google is always getting an updated version of your sitemap with all the correct URLs from your site.