How To Prevent Crawling Shopping Carts, Wishlists, Login Pages
-
What's the best way to prevent engines from crawling your websites shopping cart, wishlist, log in pags, ect...
Obviously have it in robots.txt but is their any other form of action that should be done?
-
Hi Shawn,
It is inadvisable to use more than one method of blocking bots from specific pages on your site.
There is a very good explanation of this in this blog post from Lindsay - Robot Access and Indexation Restriction Techniques: Avoiding Conflicts. Definitely worth reading!
Hope that helps,
Sha
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sub-pages have no pa
I took over a website a few months ago which is not performing well at all for chosen keywords. When I first inspected it, I found a rel canonical tag pointing to the homepage on every page. This was quickly deleted and all the pages were fetched in webmaster tools. 3 months later and the website is still performing badly. When I use the mozbar, it shows that all of the sub-pages have a pa of 1. It is only a small site and all of the pages are linked to on the navbar in a simple way. The links are not made using javascript and all the pages are on the sitemap which is submitted to wmt. I have checked that all of the changes that have been made have been indexed as well. Could it be possible that google still sees the canonical tag even though its not there? I can't think of any other reason why the pages have no pa or why it is so far behind the competitors despite having better content and links. Also, the site is appropriate for adults, but I found (among the mess left for me) a meta ratings tag set to "general". This has now been deleted, could it negatively affect rankings?
On-Page Optimization | | maxweb0 -
Home Page Keywords not Ranking and Assigned to Inside Pages
Hi, thank you for taking the time to read this. We have a few websites with the same problem. I will use http://www.prepared-meals.com as an example: The home page was ranking on page one for keyword "Prepared Meals". The site is about 6 months old. We use the Moz page optimizer on all pages of our websites to score an A rating. Recently we found the home page is no longer showing up in search results and the keyword "prepared meals" now points to an inside page that is not relevant: http://www.prepared-meals.com/Senior-Meals/Moms-Meals-Reviews.html this page shows up for Prepared Meals around page 15 in Google results. We have read keywords in the URL might be the issue, even though the page optimizer in MOZ says to do that. We are wondering if this is the issue or there is some other problem we are not aware of. Again, thank you for you for your time. -Craig
On-Page Optimization | | CraigSWD0 -
Optimizing web page
Hi I have implemented most of the suggestions, which are offered threw SEOMOZ on our web site www.zaposlitev.net. But rankings are not improving. Could anyone help please? Thanks Tomaz
On-Page Optimization | | tomaz770 -
To Many Links On Page
I'm having a problem on a crawl warning for our main site. The warning is that every one of my pages has to many links, a little over 1,000 on almost all of them. I think this is because our category list on our left hand sidebar has so many categories, and that sidebar appears on every last one of our pages even all the way into our products. Can anyone take a look and tell me if this is the reason why and what I could possibly do about this? Thanks in advance! www.Ocelco.com
On-Page Optimization | | Mike.Bean0 -
Limiting On Page Links
Right now, we have about 160 or so links on the home page. It's been recommended that we keep it to under 100, though that's not as big of a deal as it once was. Is it helpful to make a bunch of those links "nofollow" in order to preserve link juice? Is it going to make a difference, or be at all helpful? I assume it won't be harmful, especially as a bunch of them are to the same page but on different sections of the page. Would live your advice and thoughts! Thanks!
On-Page Optimization | | DeliaAssociates0 -
URL for location pages
Hello all We would like to create clean, easy URLs for our large list of Location pages. If there are a few URLs for each of the pages, am I right when I'm saying we would like this to be the canonical? Right now we would like the URL to be: For example
On-Page Optimization | | Ferguson
Domain.com/locations/Columbus I have found some instances where there might be 2,3 or more locations in the same city,zip. My conclusion for these would be: adding their Branch id's on to the URL
Domain.com/locations/Columbus/0304 Is this an okay approach? We are unsure if the URL should have city,State,zip for SEO purposes?
The pages will have all of this info in it's content
BUT what would be best for SEO and ranking for a given location? Thank you for any info!0 -
Landing page too long
In my first seomoz test I have many tilte pages for products that are over 70 charachers long. The part #'s are long like 10-782-10-10-10-PPxxxx etc. All these part #'s are not my key products and I could delete or truncate. My question is if the part numbers are not that important, is it OK to leave them as is or is ranking being damaged because they exist?
On-Page Optimization | | Wales0 -
Some of my pages are ranking for terms which I want other pages to rank for. What can I do to effectively switch the ranking?
Some of the pages are ranking for terms I have optimised other pages for. The pages which are ranking are quite rightly falling, because they aren't optimised for the terms they're showing for. However, I have pages which are optomised for those terms. How do I switch the SERPS to the page I want?
On-Page Optimization | | GlobalLingo1