In the simplest terms, you could think of searching the web as looking in a very large book with an impressive index telling you exactly where everything is located. When you perform a Google search, our programs check our index to determine the most relevant search results to be returned ( “served”) to you.

Crawling:
Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index.
We use a huge set of computers to fetch (or “crawl”) billions of pages on the web. The program that does the fetching is called Googlebot (also known as a robot, bot, or spider). Googlebot uses an Algorithmic process: computer programs determine which sites to crawl, how often, and how many pages to fetch from each site.

Google’s crawl process begins with a list of web page URLs, generated from previous crawl processes, and augmented with Sitemap data provided by webmasters. As Googlebot visits of each these websites it detects links on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are noted and used to update the Google index.

Google does not accept payment to crawl a site more frequently, and we keep the search side of our business separate from our revenue-generating AdWords service.

Indexing:
Googlebot processes each of the pages it crawls in order to compile a massive index of all the words it sees and their location on each page. In addition, we process information included in key content tags and attributes, such as Title tags and ALT attributes. Googlebot can process many, but not all, content types. For example, we can not process the content of some rich media files or dynamic pages.

Serving results:
When a user enters a query, our machines search the index for matching pages and return the results we believe are the most relevant to the user. Relevancy is determined by over 200 factors, one of which is the PageRank for a given page. PageRank is the measure of the importance of a page based on the incoming links from other pages. In simple terms, each link to a page on your site from another site adds to your site’s PageRank. Not all links are equal: Google works hard to improve the user experience by identifying spam links and other practices that negatively impact search results. The best types of links are those that are given based on the quality of your content.

In order for your site to rank well in search results pages, it’s important to make sure that Google can crawl and index your site correctly. Our Webmaster Guidelines outline some best practices that can help you avoid common pitfalls and improve your site’s ranking.

Google’s Related Searches, Spelling Suggestions, and Google Suggest features are designed to help users save time by displaying related terms, common misspellings, and people’s queries. Like our google.com search results, the keywords used by these features are automatically generated by our web crawlers and search algorithms. We only display these suggestions when we think they might save the user time. If a site ranks well for a keyword, it’s because we’ve algorithmically determined that its content is more relevant to the user’s query.

Is the whole domain, or just a page gets sandboxed?

Due to my research in this field whole domain and subdomains are sanboxed too, unless this is a “hilltop” domain, when single subdomain can get sandboxed, while other subdomains, domain, and domain’s children pages experience no affection of this filter.

Please, give your opinions concerning sandbox. I think that it is affecting pages, which gain a lot of backlinks in short period of time, but some of my friends have different opinions concerning this question. Well, what do you think?

I have a couple of downloadable archives with software of side publisher, but I had to put support documentation of this side publisher on download and support pages. Of course it is nonsense to rewrite long documents, which sometimes contain over 100 pages. I wrote short preview passages for each document with a link to the original document. Then I forbid indexing of full documents. That’s how it works on my site over a year and I have no problems with duplicate content penalty.

Sincerely

yours

Max.

Some webmasters are building sites with over 100k pages with several levels of linking. If you have troubles with indexing of such sites, then use these tips to get your huge site indexed quicker:

- You need some links to force GoogleBot read your site, so buy, or exchange links.
- Don’t point all links to the index page of your site, but put several of them to the index page and put at least one link per each subcategory. This means that if, for instance, you have four levels of linking – each category of each level must have an inbound link.
- If you haven’t got enough links to feed all your subcategories, put them to second level only and when it will get properly indexed, change links to feed thrid level. However, this way is not recommended.
- In case if your site stays unindexed, refer to some site analyzing software that will spot broken links, inaccessible areas of your site and other mistakes related to linking.
- Check your .htaccess and robots.txt files for consistency with current project.

It is also possible to create site’s categories and subcategories on subdomains of general domain and link there. It’s up to you what strategy to choose, but remember – there’s nothing impossible in SEO, so even very huge site can be forced to be indexed.

Usually I give a name that corresponds to general site title and specific item of pic. For instance in the site is about cars and there is a Lamborghini Diablo on a picture, I’d call it lamborghini_diablo_cars.jpg, or cars_lamborghini_diablo.jpg – depending on which keyword it must be stressed.

What do you think about optimization of pictures? Do you use pictures optimization on your sites?

- Google dislikes shared IPs; one should park domains on dedicated IPs.
- Google don’t care if you have shared or dedicated IP for each domain
- You need do have IPs from different networks to make safe linking between such domains
- If you have your domain on shared IP and there is a cheater or spammer having his domain on the same IP, you can get banned along with this cheating site
- If such cheating site will be banned, you won’t be banned too, but lowered in SERPs
- You will experience no changes, when site from same IP will be banned

You can see that there are much opinions and guessing about this problem I have concluded that:

If you have a couple of domains with sites, that are sometimes similar in code, names of images and affiliate links – you must avoid putting these sites on domains parked on same IP, or IPs from same network. In case if sites are completely unlike and you don’t link between them – it is not necessary to host them on different IPs.

IPs from the same network means that IPs vary in last digit(s): for instance xxx.xxx.xxx.1 and xxx.xxx.xxx.2 – Google will definitely ban or penalize you for cross linking of sites hosted on domains parked on close IPs. IPs from various networks must vary in at least third group of digits.

How do you find out whether your domain is parked on dedicated or shared IP and what sites are hosted on this IP along with yours?