Before i addressing you about Robots.txt,firstly you have to know why it
needs to your website ?
The simple and smartest answer is some what tricky but easy, to secure your
website pages by notifying to search engine that, you don't want to it crawl
your important website pages or blog pages .
cautious or note :
Just use this feature for getting
your website pages secure other wise your website or blog ignored by
search engine's with out indexing your websites .If it happens, obviously you
may miss a lot of traffic to your website from user to access their useful
information.
For example:
Your website pages having some parts of valuable information like log-in and password pages or word-press admin log in pages.so you are not supposed to allow these pages to all visitors and these are secured pages.we need to protect it
Your website pages having some parts of valuable information like log-in and password pages or word-press admin log in pages.so you are not supposed to allow these pages to all visitors and these are secured pages.we need to protect it
by adding Robots.txt to your pages,
from this you are not allow search engines to crawl or index your pages. and
make important pages getting secured.See below image, from this message you
could get some indeed idea about what exactly about the robots.txt.
I want to secure above website or blog's word press admin's page.so that
Google not crawl or index to your admin pages information by adding custom
robots.txt to above page Then you can
easily secure your website .
What happen if your website, not being with
robots.txt :
Simply, by yourself you can providing your valuable information not only to
search engine but also to all the visitors. so you need to secure your website
by using custom robots.txt to your website pages.
What happen, if your website, not being with robots.txt :
simply, by yourself, you providing
your valuable information to search engine and all the visitors. so you need to
secure website by using custom
robots.txt to your website pages.
How to add customs robots.txt to your website or
blog :
The best practice and wonderful way of using custom robots.txt by webmaster
tools.Here i will show the navigation to all of you about, how to use webmaster
tools for activate robots.txt to your website pages.
Navigation to add Robots.txt:
step 1 :
log on to Google webmaster tool. If
not create a webmaster account by using your g-mail and then add your website
in webmaster.
step 2:
click on your website in Google
webmaster tool. If you are not submitted ,just go ahead and submit .After
submitting your website ,verify your website by clicking
verify your site .As shown below.
Step 3:
click on tools after that ,under tools navigation you can find
"Generate Robots.txt". click on generate that option.
step 4:
Add your website or blog on the fields ,which are under Generate Robots.txt
as per your concern.
step 5:
After finishing above steps. Just
check your website is being with robots.txt or not.
How to check robots.txt on your website as shown below.
suppose your website name is www.example.com
now you want to check restricted pages by using robots.txt in webmaster
tools.
so type as www.example.com/robots.txt
on browser.
Obviouslly the pages displaying as shown below
For example: I restricted word press admin pages against www.eample.com. So by using above url www.example.com/robots.txt . I can get
confirmation ,whether that pages restricted or not. If restricted it display as
below.
user-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
If you get above message nothing but, you succeeded with above steps and
successfully securing your website or blog pages by adding Robots.txt .