Home | Toolshop | Guides | FAQ's | Inside |
Submit Corner Logo
Browse Submit Corner

Submit Corner : Web Tools : Server-Side Robots Generator


Assessment Tools
Keyword Thesaurus
Link Popularity Tracker
META Tag Generator
META Tag Scanner
Robot Generator
Submit Engine
Top Keywords
Word Tracker


Description Tag
Keywords Tag
Revisit Tag
See Complete List

Search Engines
See Complete List

Site Improvement
Image Tags
Title Optimization
Effective META Tags
See Complete List

Software Packages
Ultimate Demon
SENuke XCr
See Complete List

Website Assessment
Editor's Viewpoint of Directories
Assess Your Site Indexing
Who's Linking To You
Competitor Identification

Bandwidth Conservation
Background Images
External Javascript
Optimize HTML
See Complete List


Our Recommendations
Read FAQ

Webposition Gold - Download Now!

Inside Submit Corner

Linking Info
Privacy Policy
Rave Reviews
Terms of Use
Local SEO

Server-Side Robots Generator
Tell a Friend About This Page

Tool Overview

Server side Robots are controlled by a plain text file located on your root web at http://www.yoursite.com/robots.txt. The robots.txt file is a set of standardized rules that govern to search engines and their user-agents (spiders), which webpages and directories to allow indexing and which to deny.

Server-Side Robots Generator

Who Will You Allow
There are over 200 user agents (spiders) that can read through your webpages when indexing. Since new spiders get added/removed daily, we have limited the choices to the major search engines (which make their User-Agent names public). The default setting is to apply your preferences to All Agents.

Allowed User Agents:

Imposing Restrictions
You may impose restrictions on which webpages to disallow indexing. By default, most users will want to allow all directories except their /cgi-bin directory which commonly holds scripts. To enable all webpages, check the "Enable All Webpages" checkbox. Otherwise, enter each webpage or directory path in the exclusion box, one per line (all paths must end with a "/"). If you checked off Enable All Webpages, this box will not be read.
Example: "http://www.sample.com/cgi-bin/" (Excludes /cgi-bin/ directory)
Example: "http://www.sample.com/images/" (Excludes /images/ directory)
Example: "http://www.sample.com/hello.html" (Excludes /hello.html webpage)

Enable All Webpages
Exclude These URL's:

Sponsored Links

Win Great Prizes Just for Using Our Services

Latest Headlines

All links open in a new window
View All Headlines
(Mon, Jan 30 12:59:01)

Searches for Holidays to Africa Overtake North America and Close Gap With Europe
Source: Film Imaging
Date: Jan 30 2012 4:51PM

Google-backed DMARC system aims to block phishers
Source: ZDNet UK
Date: Jan 30 2012 4:51PM

Book reviews: Google books
Source: ZDNet UK
Date: Jan 30 2012 4:51PM

Searches for Holidays to Africa Overtake North America and Close Gap With Europe
Source: DMN Final Cut Pro
Date: Jan 30 2012 4:50PM

Earthquakes, Demi Moore and ACTA drive Google searches
Source: Yahoo Philippines News
Date: Jan 30 2012 4:49PM

Stop poaching staff, Jobs told Google
Source: Cyber India Online
Date: Jan 30 2012 4:49PM

Google, Facebook, Twitter Execs Grilled By UK MPs On Privacy
Source: Yahoo! Finance
Date: Jan 30 2012 4:48PM

There are 23 additional news headlines. Click to View All Headlines

Copyright ©2000 - 2009 Wired 2000 Corporation
All Rights Reserved
Privacy | Terms of Use