With the Internet developing at an incredible range, it's coming more and more difficult to get a high page ranking. There are 100s of dynamics that serve engines decide how to rank a page. Learned how some of these dynamics work, will definitely help you increase your sites exposure on the Internet. One of the dynamics that you need to take into condition is "meta robots". Trust it or not, but you can really control the robots that crawl your sites in a number of directions, which will undoubtedly help your ranking.
Meta robots are unique to each page on a website, and are used by search engines to direct crawlers on a single page. For the most part you want as many crawlers as possible to access your site; however, there are certain instances where you may not want pages or specific areas of your site being crawled. Robot tags will allow you to keep control of what is being scanned by search engines, and preventing them from returning search query results, while still being able to transfer their link value to a different pages.
Why would you prefer to block a page from being scanned, you are wondering? Here are a some reasons: for security, you may want to protect private information such as contact information, and addresses. Block spam crawlers from trying to collect sensitive information.
Mostly, you'll use the default values of meta name="robots" content="index, follow", which is the same thing as having no meta robots tag in the least. For duplicate pages you will want to use meta name="robots" content="noindex, follow", basically you do not want the duplicate page indexed, but you want the links on your page to hold value. With user-generated content, meta name="robots" content="index, nofollow", would be appropriate. The page will still be indexed, but the links will hold no value. When trying to protect private information, meta name="robots" content="noindex, nofollow", would be used to block everything.
To get started you'll need to create a robots.txt file, that lists all your pages you want to use the meta robot tag for. Than insert the appropriate tags mentioned above on each page. Every spider or bot that enters your site should read the robot.txt file which will tell them what not to index and where not to go on your website.