How To Prevent Crawling Shopping Carts, Wishlists, Login Pages
-
What's the best way to prevent engines from crawling your websites shopping cart, wishlist, log in pags, ect...
Obviously have it in robots.txt but is their any other form of action that should be done?
-
Hi Shawn,
It is inadvisable to use more than one method of blocking bots from specific pages on your site.
There is a very good explanation of this in this blog post from Lindsay - Robot Access and Indexation Restriction Techniques: Avoiding Conflicts. Definitely worth reading!
Hope that helps,
Sha
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Your tips for this landing page
http://www.visalietuva.lt/imones/advokatai These are the yellow pages in Lithuanian 🙂 What would be your tips for this of page, to increase rankings? Every single tip is appreciated: The same in English: http://www.visalietuva.lt/en/companies/lawyers (does not have meta descriptions still) THANKS
On-Page Optimization | | FCRMediaLietuva0 -
Links to Paywall from Content Pages
Hi, My site is funded by subscriptions. We offer lengthy excerpts, and then direct people to a single paywall page, something like domain.com/subscribe/ This means that most pages on the site links to /subscribe, including all of the high value pages that bring people in from Google. This is a page with an understandably high bounce rate, as most users are not interested in paying for content on the web. My question is are we being penalized in Google for having so many internal links to a page with a very high bounce rate? If anyone has worked with paywall sites before and knows the best practices for this, I'd be really grateful to learn more.
On-Page Optimization | | enotes0 -
Page Rank drop from 4 to ?
Our site (ecommerce) has been around since 1998. The pagerank has gone from a 4 to a ? The Moz score is still good, but traffic is way down. Never got a warning from Google, and were never part of a BH linking scheme? I'm puzzled? There are some duplicate content issues and missing meta tags, but they were on customer login pages that we have since blocked? Thanks
On-Page Optimization | | rglaubinger0 -
Help with the indexation of my page
Hi all, I have a problem with my website. When writing site:www.pinesapiensa.com there're no pages indexed although the webmaster tools tells me that the sitemap file has been processed in 13 May and the number of indexed paged are 21. ¿What could be happening? I have to mention that there are two domains "www.piensapiensa.es" and "www.piensapiensa.com" addressing the same website and there's a redirection from piensapiensa.com to piensapiensa.com but it doesn't work properly. Thanks
On-Page Optimization | | juanmiguelcr0 -
On-page report tool
This is a question regarding the advise this tool offer to increase the ranking of a webpage with focusing with a particularly keyword we choose. I give an example: On-page Report Card am checking my keywords and I use. "cleanse london" my surprise is Report card give an "F" for my target landing page http://www.purifyne.com , but the issue is I am already in first place first position. I know SEOmoz know have the algorithm from Google to know how to rank better but my issue is should be a little more accurate! at least. I don't want to be misunderstood here, I just want more guidance, to rank much better using this tool that I am paying for. Any thoughts?
On-Page Optimization | | teksyte0 -
Homepage ranking before optimized page
We finally managed to obtain a spot in the first 10 positions of the serps for our main keyword. Since a week, our homepage started ranking in the top 15 as well, so we we're pretty excited about that. On our way to dominate the top 10! Since 2 days we started to rank with our homepage before our optimized page, which sucks because the metadescription (made up by Google) isn't helping our CTR. Is there a way that we can show Google that the other page is more relevant than the homepage? Or do we have to wait until we have build up enough PA to switch places with the homepage (seems unlikely to me).
On-Page Optimization | | duoweb0 -
Page Rank
I had just made a 301 re-direct on one of our product pages which had a PR of 4, now that Google has indexed the new page, it's now got a PR of 0, i'm struggling to understand why this could be, i know that you may see a drop of 1, which has happened in the past, but this drop just does not make sense. Any ideas of why this could be? Kind Regards
On-Page Optimization | | Paul780 -
Google is indexing spam pages from my site. What is the most effective way to get ride of the search results? Pages are deleted now but should I do something more?
A long time ago I created a forum (Invision Power Board) and it got full of spam. Massive amounts! /forum/ I've now deleted the forum but the spam pages are still indexed on Google. Can I do something else to hurry up the process to get ride of them?
On-Page Optimization | | ocarlsson0