Blog

Commissioning of a completed Bitrix-based site is a small loss. As a rule, the most interesting begins when Google and Yandex web crawlers first index the site, when a search engine results page includes information, unnecessary for users: from “technical trash” to that photo from a New Year's office party.


 

Hold on, the unknown SEO-specialist, hold on, the unfortunate programmer, after all, the only thing to make was the correct robots.txt for Bitrix.


 

For reference: robots.txt is a file located in a website’s root, which limits access to web crawlers of its certain sections and pages.


 

Robots.txt for corporate sites and business card websites


 

The favourite phrase of the beginning copywriters “every project is unique” is the most suitable for our situation. The exception is only standard directives for robots.txt: User-agent, Disallow, Host and Sitemap. As you like it, it is a mandatory minimum.


 

The remaining issues of disallowing and overlapping are up to you. Despite the fact that Bitrix is an off-the-shelf product, the directives of the projects based on it can differ significantly from each other. The thing is in the structure and functionality of a separately taken site.


 

Imagine that you have a Bitrix-based corporate site with a standard set of sections: “About”, “Services”, “Projects”, “Contacts”, “News”. If the content of this site is unique, it is necessary to work on the disallowance of technical part of the project.




 

1. Disallow indexing for the folders /bitrix and /cgi-bin. It’s purely technical information (CSS, templates, captchas), which no one needs but GoogleBot, swearing in the panel of webmasters. You can safely disallow it. The algorithm of actions is the following: Disallow: / example/ 


 

2. The folder /search is also interesting neither to searchers, nor to users. By disallowing it, you secure yourself against doubling pages, repeating tags and titles in the results page.


 

3. When making robots.txt for Bitrix, sometimes they forget to disallow forms of authorization and PHP-authentication on the site. It is about


/auth/ 
/auth.php


 

4.  If your site is capable of printout – whether it be a district map or the invoice for payment – don't forget to disallow the following directories in the file robots.txt:


 

/*?print= 
/*&print=


 

5. Bitrix carefully stores all history of your site: successful registration of users, records about successful change and recovery of passwords. However, the interest for web crawlers is doubtful.


 

/*register=yes 
/*forgot_password=yes 
/*change_password=yes 
/*login=yes 
/*logout=yes 
/*auth=yes


 

6. Imagine that you look through a photo album on the site, you open one, two, three photos, but on the fourth you decide to return. In the address bar there will be something like such curse: ?back_url_ = %2Fbitrix%2F%2F. It can be also disallowed by changing of the file robots.txt in the root of the 1C-Bitrix CMS.


 

/*BACKURL=* 
/*back_url=* 
/*BACK_URL=* 
/*back_url_admin=*


 

Thus we insure the open (visible to users) and the disallowed parts (visible to the Bitrix CMS administrators).


 

7. Folder /upload. There Bitrix stores pictures and videos from the site. If the content is unique, it isn't necessary to disallow the folder. After all, the indexed pictures and videos is an additional source of traffic data. Another matter when /upload contains confidential information or not unique content.


 

Download robots.txt for Bitrix-based website


 

Robots.txt for Bitrix-based online stores

 


The same base as for corporate sites, though there’re some corrections. 


 

1. Unlike a simple corporate site, online store has, as a rule, more than one hundred pages. The pages of pagination, responsible for transition of the user from one product page to another, clog search engines. The more pages, the more “trash”.


 

/*?PAGEN


 

2. Disallow indexing of actions of users and site administrators. Traces of filtering, comparing of products, adding of products in a basket also shall be hidden from the view of a web crawler.


 

/*?count 
/*?action 
/*?set_filter=*


 

3. Finally, UTM-marks. Deny access to them by proceeding as follows:


 

/*openstat= 
/*utm_source=

 

Download robots.txt for Bitrix-based online store


Back to the list


0
Guest
Name Quote 0
0
Guest
mcm backpack, purses and handbags, babyliss pro, washington wizards jersey, valentino outlet, giants jerseys, polo ralph, bills jerseys, ray ban zonnebril, burberry sale, north face outlet, browns jerseys, ralph lauren outlet online, prada shoes, skechers, purses and handbags, pandora, roshe run, thomas sabo, air max schoenen, converse chucks, converse shoes, mbt shoes outlet, toms shoes outlet, lunette ray ban, denver nuggets jerseys, adidas, indoor soccer shoes, timberland femme, nike outlet, michael kors, mizuno running, oklahoma city thunder jersey, toms shoes, pandora bracelet, portland trail blazers jersey, coach factory outlet, dolphins jerseys, hollister online, lacoste shoes, birkenstock shoes, oakley sunglasses, toms.com, reloj, tommy hilfiger outlet stores, burberry sale, longchamp taschen, michael kors purses, rayban, hollister kids, thomas sabo jewellery, coach factory outlet, lunette ray ban, ecco outlet, pandora schmuck, ralph lauren outlet, ferragamo shoes uk, christian louboutin shoes, nike air jordan, ray ban sunglasses, polo ralph lauren outlet online, redskins jerseys, coach outlet, adidas online, swarovski, fitflop outlet, rolex, new balance outlet, air max, air max, zapatillas nike, louboutin, the north face, utah jazz jersey, detroit pistons, adidas superstar, fendi handbags, ralph lauren outlet, boots on sale, michael kors bags, fred perry polos, asics, nike air max shoes, montblanc, pandora charms, ralph lauren outlet online, ferragamo, oakley, raiders jerseys, nike store, tommy hilfiger online, nike outlet, burberry outlet store, michael kors v?skor, cowboys jerseys, salvatore ferragamo, oakley canada, chargers jerseys, levis jeans, oakley outlet, ray bans, cheap ray ban, hollister clothing store, roshe run, ray ban outlet, mont blanc, toms outlet, north face, belstaff outlet, dsquared2 sale, baseball jerseys, red bottom shoes, philadelphia 76ers jerseys, puma online, ray ban wayfarer, mcm bags, nike air max 90, juicy couture handbags, eagles jerseys, dsquared2, swarovski crystal, longchamp outlet, jets jerseys, katespade, nike air max, polo ralph, hugo boss shop, givenchy outlet, montre homme, belstaff, burberry outlet online, guess factory outlet, fivefingers, long champ, ray bans outlet, atlanta hawks jerseys, replica watches, bengals jerseys, birkenstock outlet, nike tn, coach factory, barbour outlet, oakley sungalsses outlet, barbour factory, roshe run, north face, christian louboutin shoes, versace outlet online, michael kors outlet online, packers jerseys, polo ralph lauren, nike air max, oakley sunglasses outlet, jaguars jerseys, true religion, kate spade outlet online, air max 2015, oakley sunglasses outlet, michael kors outlet, mizuno wave, jerseys from china, fidget spinner outlet, plein shoes, ray ban occhiali, pandora, soccer shoes, glasses frames, fred perry outlet, ralph lauren, prada outlet, hilfiger online shop, adidas.se, jordan retro, gucci shoes outlet, coach purses outlet online, veneta, coach outlet, bcbg max, baseball jerseys, supra footwear, dansko outlet, timberland outlet, burberry outlet, marc by marc jacobs, charlotte hornets jerseys, tommy hilfiger, ralph lauren polos, rams jerseys, tods outlet, prada outlet, the north face, adidas canada, katespade, rolex watch, reebok, ray ban sonnenbrillen, ray ban sunglasses outlet, michael kors, true religion jeans outlet, calvin klein underwear, northface, hermes belt, Sac Hermes, titans jerseys, coach outlet online, nike.dk, cheap michael kors, ralph lauren outlet, louboutin shoes, ray bans, huarache, chi hair, lions jerseys, oakley, nike air max, nike free 5.0, nike air max, orlando magic jersey, long champ, oakley sunglasses cheap, coach outlet online, rolex watches, supra footwear, ralph lauren factory store, michael kors outlet, hermes bags, cheap oakley sunglasses, timberland shoes, oakley vault, ray ban, uggs outlet, adidas superstar, nfl jerseys, hollister clothing, coach outlet canada, abercrombie, burberry outlet, polo ralph lauren outlet, nike air force, tory burch outlet, golden state warriors jersey, san antonio spurs jerseys, toms shoes outlet, true religion jeans, cheap true religion, hollister kids, hoodies, nike huarache, toronto raptors jersey, nike roshe, baseball jerseys, ferragamo shoes, coach outlet, nike schuhe, barbour jackets outlet, giuseppe zanotti sneakers, steelers jerseys, michael jordan, nba jerseys, coach outlet online, nfl jerseys, woolrich outlet, burberry sale, jordans, baseball bats, nike.se, polo outlet store, mcm backpack, jordan release dates 2015, reebok shoes outlet, pandora charms, michael kors canada, ralph lauren, skechers outlet , soccer shoes outlet, asics outlet, vans schuhe, new orleans pelicans jersey, jimmy choo outlet, new balance canada, converse outlet, www.tommyhilfiger.nl, houston rockets jerseys, ralph lauren, ralph lauren outlet, salomon schuhe, nike air max thea, parajumpers, nike shop, glasses online, toms outlet online, prada, coach factory shop, nike free 5.0, tory burch, patriots jerseys, dallas mavericks jerseys, longchamp outlet, gucci shoes, tommy hilfiger, moncler outlet online, knockoff handbags, giuseppe shoes, mcm handbags, kate spade outlet online, chiefs jerseys, iphone 5s cases, ralph lauren, nike shoes, coach factory, ed hardy clothing, louboutin shoes, birkenstock deutschland, michael kors canada, timberland boots, ray ban wayfarer, omega watches, belstaff outlet online, jerseys from china, adidas superstar, red bottoms, nike air max, adidas online, falcons jerseys, cheap nba jerseys, juicy couture outlet, longchamp, tracksuits store, jack wolfskin outlet, the north face outlet, michael kors outlet online, burberry, cheap oakley, cleveland cavaliers, guess outlet, burberry outlet online, michael kors, iphone case, new balance, oakley sunglasses, vikings jerseys, azcardinals jerseys, swarovski jewelry, under armour shoes, fossil uhren, michael kors, chicago bulls jersey, texans jerseys, coach outlet online, michael kors outlet, omega watches, coach black friday, bcbg max azria, burberry online shop, cc beanies, jerseys from china, hermes, milwaukee bucks jersey, softball bats, air max, sacramento kings jersey, vans shoes, jimmy choo shoes, replica watches, longchamp handbags, oakley vault, cheap oakley, bears jerseys, ralph lauren polos, boston celtics jersey, hogan, barbour outlet, moncler mens jackets, prada sunglasses, chrome hearts, christian louboutin shoes, lakers jerseys, under armour, minnesota timberwolves jersey, phoenix suns jersey, true religion jeans women, swarovski online, buccaneers jerseys, michael kors taschen, new balance, rolex watches for sale, nike schoenen, burberry outlet, thomas sabo uk, cheap basketball shoes, air jordan shoes, bottega veneta, air yeezy shoes, north face backpacks, pandora canada, gafas oakley, ferragamo shoes, michael kors outlet online, nike free run, michael kors, air huarache, celine bags, jerseys from china, true religion outlet, philipp plein outlet, north face jackets, links of london outlet, skechers, barbour jackets, burberry handbags, new york knicks, converse sneakers, saints jerseys, memphis grizzlies jersey, michael kors uk, chi flat iron, swarovski australia, coach store, ugg boots, coach outlet online, mcm backpack outlet, new balance, stephen curry shoes, ravens jerseys, nike roshe, brooklyn nets jerseys, ray-ban sunglasses, air jordan, colts jerseys, michael kors bags, hollisterco, armani outlet, burberry outlet online, hogan outlet, oakley sunglasses, nike roshe, seahawks jerseys, air jordan retro, tommy hilfiger outlet, watches canada, indiana pacers jersey, michael kors outlet online, swarovski jewelry, ralph lauren polo, nike shoes, nike free 5.0, the north face, stone island polos, air max, michael kors outlet online sale, los angeles clippers jersey, guciheaven shoes, jack wolfskin jackets, lacoste polos, michael kors, cheap nfl jerseys, rayban, 49ers jerseys, free running, hogan outlet, panthers jerseys, nike mercurial vapor, versace outlet online, vans outlet, marc by marc jacobs, oakley frogskins, cheap michael kors, asics gel, cheap nba jerseys, ralph lauren outlet, handbags outlet, heat jerseys, broncos jerseys, puma outlet
Name Quote 0
0
Guest
pandora charms sale clearancecheap uggsred wing outletdiscount oakley sunglassesultra boostfitflops sale clearancenike shoescoach outlet storebirkenstock sandalskate spade outlet storecanada goose jacketschch handbagsoakey sunglassesnike air max 87pandora jewelrynike zoomtimberland bootsbirkenstockkevin durant shoestiffany outletpolo outletadidas shoescoach outletsaint laurent handbagspandora charmsfitflops salered bottom shoesuggs outletcartier outletlouboutin shoesvalentino shoesralph laurenair jordan shoesugg outletgucci outletteva shoessupreme clothingcheap jordansvans storelouboutin shoesralph lauren outletoakley sunglassescanada goose jackets ukadidas nmdnike shoxberluti shoescanada goosepandora jewelryunderarmour shoesnike huarachemichael kors handbagslouis vuitton outlet storecanada goose jacketsred bottomscheap nfl jerseyscanada goose jacketscoach outletadidas shoespolo ralph laurencheap jordanscoach factory outlet onlinenike outlet onlinenike free 4.0mbtadidas yeezynike outletmoncler outletkeen sandalscoach outlet storetimberland boots outletnike huarachecheap oakley sunglassesmichael kors outletallen edmonds shoescanada goose outletfitflopskate spade outletmichael kors factory outletcoach factory outletpandora jewelry official sitehermes handbagscoach outletkate spade outletfred perrynike air max axisjohnston and murphy outletcaterpillar bootsmont blancgarmont bootsdior handbagsnike air prestonike storecanada goose outletcheap jordan shoesmoncler jacketsmoncler outletnike shoesnike outlet onlinekate spade outlet storecoach outlet storecheap ray bans180523yueqin
Name Quote 0
0
Guest
chanyuan2018.06.12christian louboutin outletcoach outlet onlineadidas outletconverse shoesdior outletmichael kors outletpandora jewelrymichael kors outletbarcelona jerseyoakley sunglasseshuf clothingecco shoesuggscoach outlet onlineyeezy boostlongchamp handbagsair jordan 4ralph lauren outletmanolo blahnik outletugg boots clearancepandora charmswizards jerseysmichael kors outletstussy clothingpandora outletfitflops sale clearancenfl jerseys wholesalecheap jordanstory burch outletbrequet wathesugg bootslakers jerseysfreshjive clothingcoach outlet onlinemac makeupchloe sunglassesscheap jerseysnuggets jerseysvalentino outletindianapolis colts jerseysjordan shoesharden shoesalexander mcqueen shoesoakley sunglassesdsquared2 jeanspolo ralph laurenkate spade outletadidas soccer shoesmichael kors outlettrue religion jeansair jordan shoesfitflops salecoach factory outletoakley sunglassespolo ralph laurentory burch outletstuart weitzman shoesjordan shoescheap jordanschristian louboutin outletoakley sunglasses wholesaleuggs outletprada outletmarc jacobs outletprada shoesfive fingers shoeskeen shoescoach outletcarrera sunglasseskappa clothingle coq sportif shoesoakley sunglasseschristian louboutin outletlacoste polo shirtsburberry outletcoach outletcoach outletugg outletswarovski outletadidas crazybulls jerseysarmani exchangediesel jeanspolo outletsimulation shoescoach outletcoach outletfitflops sale clearanceray ban sunglassesgucci outletcanada goose outletbally shoesrockets jerseyspolo ralph laurensupreme uked hardy clothingugg bootsbalenciaga sneakerstory burch outletkyrie 4uggs outletclarks shoesjack wolfskina bathing apemichael kors uktrue religion outletcheap jordanschristian louboutin outletmichael kors handbagsferragamo outletray ban sunglassesair jordan shoesbelstaff jacketsair max 95polo ralph laurenburberry outletoakley sunglassespandora charmsugg outletbucks jerseystrue religion jeansg-star jeanspandora charmscanada goose jacketsjazz jerseyscalvin klein jeansugg outletstoms shoesugg bootsraptors jerseyscanada goose outletcoach outletmontblanc penscanada goosemichael kors outletcoach outlet onlineoakley sunglassesgucci outletbaltimore ravens jerseysnike shoeskate spade outletair max 90yeezy boost 350new balance shoesray ban sunglassescheap snapbacksy3 shoesfitflops sale clearanceworld cup jerseyscanada goose outlet
Name Quote 0
Message Text*
Spam bot protection (CAPTCHA)
 
777en