I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
-
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
-
When you say "people," are you saying your own web team duplicates content to make their job easier? Or am I missing something?...
If that's the case, you really should create unique URL's with unique page titles, product info, etc. That's the correct way to avoid getting hit for duplicate content - don't create it. It seems like what you're doing now is more of a band-aid solution to the problem.
I'd consider that even though creating unique content in situations like this can seem daunting and/or be more expensive, there's probably huge long-term gains to made if you do it right.
-
It is not bad, just not best practices because Google will still index the URL's if they are mentioned on other pages. Just to quote them:
"While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information..."
What I would do instead is either use rel="canonical" or 301 redirects. I hope that helps.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to replace the keywords of our Google Site https://www.opcfitness.com/ 's TITLE
How to replace the keywords of our Google Site https://www.opcfitness.com/ 's TITLE Our new google site https://www.opcfitness.com/ page https://www.opcfitness.com/commercial-fitness title: Gym Equipment for Sale - Buy Commercial Fitness The site name is Gym Equipment for Sale. But we need the title like this Buy Commercial Fitness - Gym Equipment for Sale How to fix it?
On-Page Optimization | | ahislop5740 -
Why does Google pick a low priority page on my site?
Hi Guys. One of my pages ranks quite well for "mid year diaries 14-15" on Google. The problem is it's a really specific product page (A4, Hardback, day-to-a-page diary I think). It would be much better for the user to land on our mid-year diaries category, not really deep into the site. Why is Google prioritizing this product page over our general 'mid year diaries' category? Especially when the category would relate to the search more accurately? I work for TOAD diaries and I think our page rank is 10 for this search. Eagerly awaiting some insight 🙂 Thanks in advance everyone! Isaac.
On-Page Optimization | | isaac6630 -
If I put 'keyword/url' combination to 'stop run weekly', will it dissapear from the summary page in the on-page grader?
The summary page of the on-page grader chooses the keyword and url combination itself. Now if I choose another combination, I would like the former to dissapear from the summary page. The only option is 'stop running weekly'. But will it disappear from the list also?
On-Page Optimization | | jongeneelbv0 -
Is it better to delete old job pages on a recruitment site?
My client (online recruitment) has over 1.5 million pages indexed, the majority of which are old job posts and listings since they began. I wanted to know if it would be better to keep all of the pages, or advise my client to delete some of the archive as these pages will no longer be attracting traffic.
On-Page Optimization | | AxonnMedia0 -
Home page keyword effecting internal page ranking
Hello, My client has a second keyword for the home page that is competitive. The home page is not being ranked for this keyword. Instead, an internal category page is ranking. This internal category page is more relevant than the home page - it shows the categories for the actual products that this term refers to. But everyone around us in Google's page results has far more backlinks than the internal page, and we're all heavily optimized for this term. My question is, is it safe to pull the second term off of the home page or is this internal page strong because it is somehow being strengthened by the home page optimization?
On-Page Optimization | | BobGW0 -
Main page deindexed by google.
3 days ago our main page('/') has stopped appear in google results. Rest of pages works fine. Even our main page from canadian version of site with similar content works fine. Some times canadian page appears for key word where we had our com version before. But I think this is just result of disappearing com version. Any suggestion were to look? Messages box in google webmastertools is empty. Could this be the question too long page title? PS: Does any way exist to check if we were punished by Google and reason of this? For Bing and Yahoo everything works fine. PPS: We have just found that "cache:our_site.com" and "info:our_site.com" returns in google our_site.ca. So this should be reason of problem. So now we are looking how to fix this.
On-Page Optimization | | ctam0 -
Blocked By Meta Robots
Hi I logged in the other day to find that over night I received 8347 notices saying certain pages are being kept out of the search engine indexes by meta-robots. I have not changed my robots.txt in years and I certainly didn't block Google from visiting those pages. Is this a fault on Roger Mozbot behalf? Or is there a bot preventing 8000+ pages being indexed? Is there a way to find out what meta-robot is doing this and where? And how I can get rid of it? I usually rank between #3 and #5 for the term 'sex toys' on google.com.au, but I now rank #7 to #9 so it would seem some of my pages/content is being blocked. My website is www.theloveshop.com THIS IS AN ADULT TOYS SITE. There is no porn videos or anything like that on it, but just in case you don't wish to look at sex toys or are around kids I thought I would mention it. Blake
On-Page Optimization | | wayne10 -
Home page ranking dropped below internal pages
The index page for a site I manage has dropped significantly - internal pages rank above it. It's a new site, 2 months old but was ranking at 1st. Any suggestions as to how I can debug this?
On-Page Optimization | | OptioPublishing0