How To Prevent Crawling Shopping Carts, Wishlists, Login Pages
-
What's the best way to prevent engines from crawling your websites shopping cart, wishlist, log in pags, ect...
Obviously have it in robots.txt but is their any other form of action that should be done?
-
Hi Shawn,
It is inadvisable to use more than one method of blocking bots from specific pages on your site.
There is a very good explanation of this in this blog post from Lindsay - Robot Access and Indexation Restriction Techniques: Avoiding Conflicts. Definitely worth reading!
Hope that helps,
Sha
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Keyword ranking for different page than the page optimized
I have optimized "equipment trailer for rent" on this page: http://www.bigtrailerrentals.com/flatbed-trailer-rentals/equipment-deckover. I'm wondering if anyone can tell me why Google has chosen to rank the keyword phrase for this page: http://www.bigtrailerrentals.com/flatbed-trailer-rentals/equipment-24 This is just one example. It has happened on several of my pages / keywords.
On-Page Optimization | | BigTrailerRentals0 -
Why are http and https pages showing different domain/page authorities?
My website www.aquatell.com was recently moved to the Shopify platform. We chose to use the http domain, because we didn't want to change too much, too quickly by moving to https. Only our shopping cart is using https protocol. We noticed however, that https versions of our non-cart pages were being indexed, so we created canonical tags to point the https version of a page to the http version. What's got me puzzled though, is when I use open site explorer to look at domain/page authority values, I get different scores for the http vs. https version. And the https version is always better. Example: http://www.aquatell.com DA = 21 and https://www.aquatell.com DA = 27. Can somebody please help me make sense of this? Thanks,
On-Page Optimization | | Aquatell1 -
Imiges on own page???
Can somebody help me to explain why when I click on an image in a post it goes to its own page?? And why if I click on other image it does not go anywhere?? this is one of the posts: http://villasdiani.com/beach-wedding-at-a-luxury-boutique-hotel/ what is good or bad?? On most of the images SEOmoz is reporting that I am missing meta description!
On-Page Optimization | | VillasDiani0 -
Is reported duplication on the pages or their canonical pages?
There are several sections getting flagged for duplication on one of our sites: http://mysite.com/section-1/?something=X&confirmed=true
On-Page Optimization | | Safelincs
http://mysite.com/section-2/?something=X&confirmed=true
http://mysite.com/section-3/?something=X&confirmed=true Each of the above are showing as having duplicates of the other sections. Indeed, these pages are exactly the same (it's just an SMS confirmation page you enter your code in), however, they all have canonical links back to the section (without the query string), i.e. section-1, section-2 and section-3 respectively. These three sections have unique content and aren't flagged up for duplications themselves, so my questions are: Are the pages with the query strings the duplicates, and if so why are the canonical links being ignored? or Are the canonical pages without the query strings the duplicates, and if so why don't they appear as URLs in their own right in the duplicate content report? I am guessing it's the former, but I can't figure out why it would ignore the canonical links. Any ideas? Thanks0 -
Page Title - What is better?
Hi SEO Heads, I have another question if someone would be so kind in answering What page title of the 2 below is better for SEO (i) Chocolate Cake|Chocolate Cake Recipe|Xmas Cake or (ii) Chocolate Cake | Chocolate Cake Recipe | Xmas Cake As you can see (ii) Page Title has a space before and after the | (vertical bar) I know the second page title looks better to human eyes but on some pages I had to forego the space so i could fit my keywords in the page title. is this a good idea? Can anyone help me? Cheers Aidan
On-Page Optimization | | aidanlawlor0 -
Source page leading to a 404 pages in reports
Hi everybody, I wonder how to find and quickly correct 404 errors in my crawl reports : SeoMoz says me "http://domain.com/this-page-is-dead" is 404, but I can't figure out a source page where a link to that url appears. I tried a google link:http://domain.com/this-page-is-dead request, with no more luck. I imagine the trick is trivial, but I need it 🙂 Moreover, why do not show a list of pages referring to this 404 page on reports ? Thanks, Loïc
On-Page Optimization | | mandinga0 -
Too many on-page links
Hi, My website - www.thepartyhouse.com.au is showing as having too many on-page links for over 4,000 pages. Take for example the homepage which is showing as 188 links, but I don't understand this because I've used SEO tools to display the links and I am showing around 90 links on this page. How can I see what all the links are? Thanks
On-Page Optimization | | Spyre0 -
Why home page ranks higher than keyword-optimized page
We have a page that is optimized for the keyword "job scheduling". A search on the keyword "job scheduling" results in this page not ranking at all, while our home page (uc4.com) ranks third. Could you provide some ideas/suggestions as to why this would be the case and how to make our job scheduling page rank higher? Thanks, claudia
On-Page Optimization | | claudmar0