How To Prevent Crawling Shopping Carts, Wishlists, Login Pages
-
What's the best way to prevent engines from crawling your websites shopping cart, wishlist, log in pags, ect...
Obviously have it in robots.txt but is their any other form of action that should be done?
-
Hi Shawn,
It is inadvisable to use more than one method of blocking bots from specific pages on your site.
There is a very good explanation of this in this blog post from Lindsay - Robot Access and Indexation Restriction Techniques: Avoiding Conflicts. Definitely worth reading!
Hope that helps,
Sha
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
When making content pages to a specific page; should you index it straight away in GSC or let Google crawl it naturally?
On-Page Optimization | | Jacksons_Fencing0 -
Why will my page description not update??!!
I updated my page title and description along time ago, the title updated but not the description. I updated it in google search console, but it keeps pulling the first few lines from the page content instead, which isn't really an attention grabbing sentence?? Any ideas? Here's the page http://thumannagency.com/personal-insurance/home-insurance This is what my Title/ description comes up as... Homeowners Insurance Quotes Dallas TX | Thumann Agency <cite class="_Rm">thumannagency.com/personal-insurance/home-insurance</cite>When shopping for homeowners insurance in Dallas, TX our local agents will provide you with home insurance quotes from highly rated home insurance ...
On-Page Optimization | | MissThumann0 -
How to improve page authority?
Hello guys, I am in need of some helpful tips to improve my page authority for this site www.servicii-funerare-iuliana.ro . I have page authority 0, and I don't know why... Thanks.
On-Page Optimization | | iulianafun0 -
Why are my pages de-indexed?
<form id="form-t3_37nfib9dz" class="usertext" action="http://www.reddit.com/r/SEO/comments/37nfib/why_were_my_pages_deindexed/#"> Hello all, I am very new to SEO. For some reason many of the pages on my site were de-indexed. Specifically the ones linked from this page: However other pages, like the ones linked from this page and this page were not de-indexed. http://www.lawyerconnection.ca/practice-areas/car-accident-injury-lawyers/[1] However the pages linked from this page were not de-indexed: http://www.lawyerconnection.ca/practice-areas/slip-and-fall-lawyers/[2] http://www.lawyerconnection.ca/podcastresources/[3] That first page itself was not de-indexed, just the site that it links to. It just happened today, so maybe I am jumping the gun but I doubt it. When I enter the page into google webmaster tools again and press fetch, one of the child pages, it re-indexes. What could be the problem here? I had someone re-write the content for every city but I have a feeling that there is less differences in the car accidents pages? Is this considered duplicated content do you think? Am I making some other mistake I can't think of? Is it just a one day blip (I doubt it) Let me know, thanks. </form>
On-Page Optimization | | RafeTLouis0 -
Will pushing a visitor to a conversion page hosted on a 3rd-party domain hurt the landing page ranking
Had an interesting question from a client. The client has a page that is optimized for a specific term. The goal of the page is to push users to sign-up for a trial. The trial registration (conversion) page is hosted by a third-party. Will pushing users to the conversion page cannibalize the SEO authority of the landing page. My reflexive answer is to say no, but now am not so sure.
On-Page Optimization | | infoblue0 -
Issue: Duplicate Page Content
Hello SEO experts, I'm facing duplicate page content issue on my website. My website is a apartments rental website when client search apartment for availability. Automatic generate same url's. I've already block these url's in robots.txt file but facing same issue. Kindly guide me what can I do. Here are some example links. http://availability.website.com/booking.php?id=17&bid=220
On-Page Optimization | | KLLC
http://availability.website.com/booking.php?id=17&bid=242
http://availability.website.com/booking.php?id=18&bid=214
http://availability.website.com/booking.php?id=18&bid=215
http://availability.website.com/booking.php?id=18&bid=256
http://availability.website.com/details.php?id=17&bid=220
http://availability.website.com/details.php?id=17&bid=242
http://availability.website.com/details.php?id=17&pid=220&bid=220
http://availability.website.com/details.php?id=17&pid=242&bid=242
http://availability.website.com/details.php?id=18&bid=214
http://availability.website.com/details.php?id=18&bid=215
http://availability.website.com/details.php?id=18&bid=256
http://availability.website.com/details.php?id=18&pid=214&bid=214
http://availability.website.com/details.php?id=18&pid=215&bid=215
http://availability.website.com/details.php?id=18&pid=256&bid=256
http://availability.website.com/details.php?id=3&bid=340
http://availability.website.com/details.php?id=3&pid=340&bid=340
http://availability.website.com/details.php?id=4&bid=363
http://availability.website.com/details.php?id=4&pid=363&bid=363
http://availability.website.com/details.php?id=6&bid=367
http://availability.website.com/details.php?id=6&pid=367&bid=367
http://availability.website.com/details.php?id=8&bid=168
http://availability.website.com/details.php?id=8&pid=168&bid=168 Thanks and waiting for your response | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |0 -
Home Page Keyword Retargeting
Hi Guys! I have a question regarding the risk of **re-targeting **your homepage to another **keyword. ** To give a little background: I am working on a clients site that targets a keyword we initially thought during the keyword research phase, would be an appropriate target. What we found out is that it is not. Everything in analytics points to this keyword not performing well. We now want to re target the homepage to another keyword target. My questions are: How risky is it to do this given we have already done a fair amount of link building to the home page with anchor text from the initial keyword target? Will this look suspicious to google? What sort of things should we consider before moving forward with this change?
On-Page Optimization | | gravityseo0 -
On page internal link text
Hi, I'm in the process of rebuilding/designing an existing well ranking niche bespoke software site and have the following question - In the footer, I'm planning on linking to the main landing pages (blue widget software, red widget software etc theres about 7 in total). In these links I want to know if its best to have the word "software" in each link as I'm scared of it looking spamy. We sell custom software, and a lot of the keywords that currently attract traffic (as reported in analytics) end in the word software, for example - blue widget software red widget software In the footer would you end each link with software or not? How much effect would this have on rankings? Thanks in advance.
On-Page Optimization | | JamesJacobs0