I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
-
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
-
When you say "people," are you saying your own web team duplicates content to make their job easier? Or am I missing something?...
If that's the case, you really should create unique URL's with unique page titles, product info, etc. That's the correct way to avoid getting hit for duplicate content - don't create it. It seems like what you're doing now is more of a band-aid solution to the problem.
I'd consider that even though creating unique content in situations like this can seem daunting and/or be more expensive, there's probably huge long-term gains to made if you do it right.
-
It is not bad, just not best practices because Google will still index the URL's if they are mentioned on other pages. Just to quote them:
"While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information..."
What I would do instead is either use rel="canonical" or 301 redirects. I hope that helps.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hi, Does having orphan pages on my site negatively affect my seo? Thank you.
Hi, I have quite a few orphan pages on my site and we see that our rankings have fell significantly over the past 6 months. Can this have be negatively affecting our rankings? Thank you.
On-Page Optimization | | whiteonlySEO0 -
What's the best SEO tactics when you have a dedicated web address pointing to a page on a different site?
Hope someone can help with a question I've got about sorting out some duplicate content issues. To simplify the question, imagine there is a website a.com which has a page a.com/newslettersignup. In addition to the a.com domain, there is also a different web address, ashortcut.com, which points to a.com/newslettersignup. ashortcut.com is the web address that is advertised in marketing material etc. So what is the best way then to tell Google etc. that ashortcut.com is the preferred URL for the page which sits at a.com/newslettersignup? The advice I've read about the canonical tag, for example, doesn't cover this exact scenario so although it can support cross-domain information, I'm not sure if that's the best route to follow. Thanks!
On-Page Optimization | | Nobody15755058948220 -
Why is my contact us page ranking higher than my home page?
Hello, It doesn't matter what keyword I put into Google (when I'm not signed in and have cleaned down my browsing history) the contact us page ranks higher than the home page. I'm not sure why this is, the home page has a higher page authority, more links and more social media shares, the website is an established one. When I have checked Google Analytics my home page gets more people landing on it than the contact us page. It looks like people are ignoring the contact us page and scrolling down until they find the home page. I'd appreciate any help or advice you might have. Thank you.
On-Page Optimization | | mblsolutions2 -
Google crawler showing cache of another page
For the page http://www.thinkdigit.com/top-products/Laptops-and-PCs/top-10-laptops-124.php google is showing another page in cache (http://www.thinkdigit.com/top-products/Ultrabooks/top-10-ultrabooks-153.php). Please let me know how this happened and how to correct it.
On-Page Optimization | | 9dot90 -
Google's Page Layout Algorythm
It seems that Google have been or will penalizing websites with too many ads above the fold. Is it me or Google's search result layout is a perfect example of what NOT to do?
On-Page Optimization | | sbrault741 -
Why does Google no longer like our site?
Hey guys, I'm trying to figure out why the traffic and rankings have been plummeting on www.readprint.com. It's a collection of both public domain books and books on Amazon's store. If anyone can offer any pointers as to if it's duplicate content or ??? It used to get 300K visits/mo but has slowly been dropping over the last year. I appreciate anyone's expertise!
On-Page Optimization | | CoBraJones0 -
Faq page
We are redoing our faq page and we were trying to decide on the best format. 1. Create each question on a separate page 2. Create one page with all the question and have the questions expand 3. Create different faq category pages (like 4) and divide the questions between them From my perspective #1 seems the best ---. you can create hyper relevant content for the user and optimize each question really well Any experience with this?
On-Page Optimization | | Morris770