Crawl Site

To analyze a site, run Screaming Frog’s Free SEO Spider. The first 500 URIs are free. There’s an annual paid plan if you need more.

Webmaster Tools

Register your site with Google’s Webmaster Tools. You can find a good Overview of Google Webmaster Tools at Search Engine Watch. Register your site with Bing’s Webmaster Tools. You can find a good checklist for getting started at Bing.


If user and search engines cannot find your site, you might as well not have one. In the beginning, it’s all about being found. And not just being found, but being found for right keywords. Your site is a easy-to-access view determined by the information architecture that your content and keywords have defined. If you get the Keyword Research right (including seasonal trends and well-formed groups), your information architecture will describe the structure and organization of your website and how it fits together.

Information architecture is the structure that affects user experience. It is this structure that informs the basic website design. You then incorporate effective navigation of the content and incorporate the functionality of the meta-data.


Manually check your robots.txt file to make sure it’s not restricting crawlers. Check the robots meta tag to make sure it’s not telling crawlers not to follow a page’s links. You can check this by using Filezilla. Open your site’s robots.txt file and see what’s there. If it’s a WordPress site, go to the WordPress Support page and enter robots.txt. There you’ll find tons of information on what a well-formed robots.txt file should look like. Or, you can install a plugin that will let you manage it from the Dashboard.

404s and Soft 404s

Find and fix any URLs in your site that return a 404 (Page Not Found), or Soft 404s (the content of the Page is entirely unrelated to the HTTP response returned by the server). Google states that 404s do not affect your ranking, but they do have an affect on User Experience, which indirectly affects your ranking. For example, running into 404s increases the liklihood that a visitor will bail and go elsewhere.

It is very important that you build a custom page to send user to if the server returns a 404. There you should say something that is helpful, light, and redirects them to the mostly like place on the site to help with the category information they were looking for when the 404 was received from the server. It’s very difficult to avoid any possibility of a 404, of either type, but being helpful when it happens is a much better strategy than just ignoring it because it doesn’t directly impact ranking.

XML Sitemap

Submit well-formed XML to both Google and Bing.

Site Architecture

Your site architecture should be “flat”, meaning the fewer number of click to get to your content the better for both User Experience and Crawlers.

Site Performance

Run Google PageSpeed Insights to get helpful suggestions on improving the page-load speed of your site’s pages.