Selectively Excluding Pages from Being Indexed
There are many times you may with to exclude certain pages from being indexed by certain engines. One way to do this is by utilizing a robots.txt file and uploading it to the root directory of your Web site.
Basically, you just create a text file with Window's NotePad or any other editor that can save ASCII .txt files.
Use the following syntax:
User-Agent: {SpiderNameHere}
Disallow: {FilenameHere}
For example, to tell Inktomi's spider, called Slurp, to not index files called orderform.html and junk.html, create a robots.txt file as follows:
User-Agent: ArchitextSpider
Disallow: orderform.html
Disallow: junk.html
You would then upload this robots.txt file to the root directory of your Web site. Although this is a voluntary protocol, most major search engines will honor it.
You can add more lines to exclude pages from other engines by specifying the User-Agent parameter again in the same file, followed by more Disallow lines. Each disallow statement will be applied to the last User-Agent that was specified. If you want to exclude an entire directory, use this syntax:
User-Agent: ArchitextSpider
Disallow: /mydirectory/
Other options are to exclude the page from all spiders with:
User-Agent: *
Disallow: /mydirectory/
Do NOT use the wildcard (*) character in the Disallow line since that's not supported.
Make sure you use the proper syntax. If you misspell something, it's not going to work.
This article is copyrighted and has been reprinted with permission from FirstPlace Software, the makers of WebPosition Gold. FirstPlace Software helped define the SEO industry with the introduction of the first product to track your rankings on the major search engines and to help you improve those rankings. A free trial of WebPosition Gold is available from their Web site.
Search Engine optimization articles
What is "organic" search and how can it help your company?
How to increase your sales by optimizing for local markets.
Seven Steps of Search Engine Optimization
Why You Should Validate Your HTML
22 Reasons Why Your Page Did Not Get Indexed
Make Your Dynamic Web site Search Engine Friendly
Is Your WebSite Guilty By Association?
Improving Table Prominence for Higher Rankings
The Top Five Strategies for SEO
Search Engine Marketing 102: Boosting Prominence
Finally a Cost-Effective Way to Track Your Sales
Using Miva Merchant and not Ranking Well in Google? Learn Why!
Watch Out for Fancy Menu Systems
Ten Ways to Gain More SEO Clients: Beginning with your local market
Search Engine Marketing 101: What Search Engines See When They Visit Your Web Site
Advanced Tip: Improving Rankings via Server Side Includes (SSI)
Are You Losing Visibility by Duplicating Titles?
Increase your Click-Throughs With Killer Title Tags
Future Outlook: Will all Submissions Soon Become Paid?
Thou Shalt Not Spam! The 12 Commandments of Search Engine Marketing
Top Reasons Why You May Not Be Indexed
TIP: Expand Your Traffic Through Misspellings - Free Service!
The Top 5 Tips and the Top 5 Mistakes of Search Engine Marketing
How Often Should I Submit?
Benefits of Organizing Your Content Into Separate Domains
How and Why to Build a Robots.txt
Which is Better: Manual Submission or Automated?
Could my competitors be spamming me?
Should I include dashes in my domain name?
How to Host Multiple Domains to Maximize your Rankings
How to Get Your Pages Indexed and Then Keep Them That Way
A Marketing Technique You Should Avoid
How to Make Better Use of Images
Selectively Excluding Pages from Being Indexed
Why some pages rank high for no apparent reason!
Avoid Pitfalls with Frames
How to Avoid Trouble with the Engines
Link Tags: Overlooked Way to Improve your Score
How to Create Effective Page Descriptions
Check our complete range of services: SEO Optimization India, Web Design India, Web Hosting India, Internet Marketing &
Offshore Software Web Development Company India
