Im­pact Of The Robots.Txt, Meta-Ro­bots & Rel=Nofollow On Seo And Their As­pects To Avoid Seo Mistakes

Most of the peo­ple don’t know the ex­act use of robots.txt, the META-ro­bots and the rel=nofollow an­chor at­trib­utes and be­cause of this they make mis­use of them.

This in turn af­fects the crawl­ing, page rank dis­tri­b­u­tion and in­dex­ing of a site. As a re­sult the search en­gine op­ti­miza­tion of their site is affected.

This ar­ti­cle will tell you the im­pact of robots.txt, the META-ro­bots and the rel=nofollow on SEO.

Robots.txt

Robots.txt is a text file which is placed in the top level di­rec­tory of a web­site (for ex­am­ple www.myexample.com/robots.txt). It is used to hide or re­move any in­for­ma­tion which you do not want to ap­pear in search re­sults. Web­mas­ters use it to in­struct crawlers / web spi­ders which pages to spi­der. The robots.txt file lists which web pages should not be crawled by ro­bots, what should be the crawl rate, and the lo­ca­tion of XML sitemap.

Fol­low­ing are some of as­pects of robots.txt to help you avoid­ing the ma­jor SEO blunders:

• If you block a web­site, page or any folder by us­ing robots.txt file then it will not be crawled by those search en­gines which fol­low the Ro­bot Ex­clu­sion Stan­dard. How­ever they will be crawled by spam bots de­spite your ro­bot­s’txt file.
• If you block any site, web­page or folder by us­ing robots.txt then it is not crawled by ma­jor search en­gines. But it might ap­pear in search en­gine re­sults if they find suf­fi­cient in­for­ma­tion about a site on DMOZ or if it has links on other pages. The search en­gine might show the page in search re­sults, ex­cept that it won’t show a description.
• Cal­cu­la­tion of page rank does not de­pend upon whether the page is blocked or not by robots.txt. There­fore those in­bound links which are point­ing these pages do sur­pass link juice. Since blocked pages can­not be crawled by search en­gines there will be no out­bound links and there­fore these pages are con­sid­ered as dan­gling pages. This means these pages fade out the page rank of other pages of a site and lead to the loss of page rank of a site.

META-ro­bots

Meta-ro­bots is a Meta tag that spec­i­fies whether the par­tic­u­lar page should be in­dexed or not and whether the out­bound links should be fol­lowed or not.

De­fault tag in Meta-ro­bots is “index,follow” which means this par­tic­u­lar page should be in­dexed and the out­bound links should be fol­lowed if there is no use of rel=nofollow attribute.

The valid val­ues of these Meta tags are in­dex, fol­low, noin­dex, no­fol­low and none.

Fol­low­ing are some of the as­pects of meta-ro­bots that help you in avoid­ing the ma­jor SEO blunders:

• If you block a page by us­ing noin­dex Meta-ro­bots value then it doesn’t mean it will not crawl by search en­gines. It will crawl by all ma­jor search en­gines where all links of a page will be extracted.
• If the blocked page is crawled by search en­gines then it doesn’t mean that it will ap­pear in search re­sults. To rea­son this out, crawl­ing and in­dex­ing have to be un­der­stood as dif­fer­ent terms. As the web­page is crawled, the search en­gine iden­ti­fies the meta-ro­bots noin­dex value. The page will then never show up in the re­sults of search engine.

Ex­am­ples:

The page will not be in­dexed, but the search en­gine will fol­low all links on the page:
<META NAME=„ROBOTS” CONTENT=„NOINDEX, FOLLOW”>
The page will be in­dexed, but the spi­der will NOT fol­low any link on the page:
<META NAME=„ROBOTS” CONTENT=„INDEX, NOFOLLOW”>
The page won’t be in­dexed, and the links won’t be followed.
<META NAME=„ROBOTS” CONTENT=„NOINDEX, NOFOLLOW”>

If you want your pages in­dexed and all links fol­lowed, you do not need the ro­bots meta tag at all.

Rel=Nofollow

The rel=”nofollow” is an at­tribute that is used in an­chor links which we wish to block so that they do not end up pass­ing any link juice to any of the tar­get pages.

Fol­low­ing are some of the as­pects of rel=”nofollow” that help you in avoid­ing the ma­jor SEO blunders:

• If you block a link by us­ing rel=”nofollow” then it doesn’t mean the search en­gine will not in­dex the page. It will only not count your link for page rank. There­fore, rel=nofollow is not rec­om­mended for pre­vent­ing the in­dex­ing of a page.
• In prin­ci­ple, the search en­gines don’t crawl the “no­fol­low” links. How­ever, prac­ti­cally, the con­verse is true. Hav­ing a link no-fol­lowed does not mean the spi­der will not find the page. But it does mean that the links don’t pass any link juice to the linked page.

Google web­mas­ter cen­tral can give you a brief idea on this.

About the au­thor: Matthew An­ton is an on-page op­ti­miza­tion ex­pert. He has done ex­ten­sive re­search on the im­pact of Robots.Txt, Meta-Ro­bots and Rel=Nofollow on seo.