I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
-
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
-
When you say "people," are you saying your own web team duplicates content to make their job easier? Or am I missing something?...
If that's the case, you really should create unique URL's with unique page titles, product info, etc. That's the correct way to avoid getting hit for duplicate content - don't create it. It seems like what you're doing now is more of a band-aid solution to the problem.
I'd consider that even though creating unique content in situations like this can seem daunting and/or be more expensive, there's probably huge long-term gains to made if you do it right.
-
It is not bad, just not best practices because Google will still index the URL's if they are mentioned on other pages. Just to quote them:
"While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information..."
What I would do instead is either use rel="canonical" or 301 redirects. I hope that helps.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can service request pages be indexed for a service site?
I think there is no point in indexing service request pages for a service site. And it causes the indexing of the main pages to be done with a delay. Does anyone have experience with indexing service request pages and their results?
On-Page Optimization | | sora.ya04680 -
Robot.txt file issue on wordpress site.
I m facing the issue with robot.txt file on my blog. Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue. The search result shows that "A description for this result is not available because of this site's robots.txt – learn more." Any suggestion to over come with this issue
On-Page Optimization | | Mustansar0 -
How can my national ecommerce site geo target specific states for specific pages?
If I am selling sports related products for sports/teams across the country, how can I target specific states or cities through various pages on my website. I am using adwords but would like to use increase organic search results.
On-Page Optimization | | briggsb0 -
Google picking up old pages
I recently redesigned a site that had all the keywords it was ranking for going to the home page. Now I have specific pages for each of these keywords but I'm seeing the home page (not the page that, if I do an on page optimization by hand in MOZ gives me an A rating) showing up in the auto reports (assuming pages Google sees for these keywords related to the url) as F's. They're all pointing to the home page. I've redirected the old index.html home page to the new but I suspect the reason is actually these pages (were) ranking for these terms (though none too well - all but one were not in the top 50 and one was 45) because these rankings are all dropping as well. I'm at a loss, with the site replaced, as to how to correct this and tell Google these keyword phrases all have their own pages now. I've dug through this forum and the only applicable answer I can see would be to add these phases to the home page (where they all rank for now) with anchored links to their new (A rated by Moz for these terms when I hand enter them) singular pages? Or is it just a waiting game?
On-Page Optimization | | adworksofboca0 -
How does Google view frequent changes to the copy of an ecommerce product page?
Is there any reason to think that adding to or improving copy might harm seo?
On-Page Optimization | | Brocberry0 -
Issues with Product Pages Getting Index In Google
I just started working here the other week and one of the big issue is that a lot of the product pages are not getting index in google. We have an xml.gz site map they submitted a long time ago. My guess is it might be something with not enough content on the pages? Here are a few example of pages that are not getting index in google. http://www.rockymountainatvmc.com/p/43/-/439/716/-/33097/Alpinestars-Dual-Motorcycle-Gloves http://www.rockymountainatvmc.com/p/47/-/201/803/-/28948/Camelbak-Blowfish-2013 http://www.rockymountainatvmc.com/p/46/-/203/836/-/6996/MSR-Head-Case http://www.rockymountainatvmc.com/p/44/54/208/764/80/1220/Galfer-Brake-Pad-Sintered-Metal There are 100's that are not indexed just trying to figure out what we need to do! We are working on new content to them all but we have over 5000 products so it will take a long time. We also have the reviews on the pages and are looking at starting a Q&A on page to help get more unique content.
On-Page Optimization | | DoRM0 -
Google Sitemap
Does adding a Google Sitemap to webmaster tools REALLY help SEO? If so, are there any resources for help creating one? Here is my site: http://www.petmedicalcenter.com Thanks,
On-Page Optimization | | PMC-312087
Brant0 -
Page Indexing
Hello All Nice easy question! I've made some on page changes to page titles, content, H1s etc but wanted to know if there was a way to check if Google has reindexed the page since the changes were made? I appreciate the different factors that will help improve your crawl rate like new content, external links, domain authority etc. I made these changes around 2 weeks ago. Google has cached the pages since I made the changes but not picked up on the new page titles in the search results. Cheers Todd
On-Page Optimization | | todd75850