How To Get Google To Index Your New Website & Blog Quickly

How To Get Google To Index Your New Website & Blog Quickly

Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index.We use a huge set of computers to fetch billions of pages on the web. The program that does the fetching is called Googlebot (also known as a robot, bot, or spider). Googlebot uses an algorithmic process: computer programs determine which sites to crawl, how often, and how many pages to fetch from each site.

Google's crawl process begins with a list of web page URLs, generated from previous crawl processes, and augmented with Sitemap data provided by webmasters. As Googlebot visits each of these websites it detects links on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are noted and used to update the Google index.

Google doesn't accept payment to crawl a site more frequently, and we keep the search side of our business separate from our revenue-generating AdWords service.
Crawling

The first thing we need to look at is to make sure that all of our target pages can be crawled by the search engines. I say “target pages” because there will be occasions when you may want to actively stop certain pages being crawled, which I’ll cover shortly.

Before that, let’s look at how we make our website crawlable and how to look for potential problems.

Good site architecture

A good website architecture is not only good for search engines, it is good for users too. Put simply, you want to make sure that your most important pages are easy to find, ideally within a few clicks of the homepage. This works well for a couple of reasons:

Usually, your homepage is the most linked to and therefore can flow a lot of PageRank throughout the rest of the site
Users will be able to find your key pages quickly – increasing the likelihood of them finding what they want and converting into customers
In terms of what this actually looks like, a simple structure will be like this:

site-architecture-simple

 If you own an ecommerce website, the detail pages in this example would be your product pages. This is quite a logical structure and one that is recommended for small to medium size websites. The only other major consideration here is to make sure that you map keyword research to this structure which you can read about in this post by Stephanie Chang and this post by Richard Baxter.

But what if you have a website with millions of pages? Even with good category structure, your key products may end up being quite far away from the homepage. In this case, you may want to consider implementing a faceted navigation which can help with this. Faceted navigation adapts itself to what the user is looking for and eliminates a lot of the noise by letting them easily filter to find what they want. The best thing to do here is to show you an example of what I mean, luckily the guys at Madgex wrote this good article and created this example which visualises things very well.