Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
References for Healthcare Blog Content?
Hey everyone, We have a couple B2C medical/healthcare clients we produce content for and I was wondering what the industry stance is when it comes to giving references at the end of a blog, assuming there were no statistics or direct quotes used in the content. A lot of our content is written via research on a specific condition/treatment and doesn't really dive deep into specific medical nuances. Things like risks, recovery timelines, questions to ask, etc. are written about mostly. Still, should we be providing general references at the end of blogs to sites like WebMD, Medscape, etc. Thanks for any input!
Content Development | | danielreyes0 -
How can we dynamically populate content on our website based on a visitor's web history?
Recently, I have tried looking into options that would allow us to dynamically populate content (specifically images to be used at CTAs on our blog in wordpress) to different users based on their web history on our website. We would want to be able to dynamically populate images based on the number of visits in the past 60 days and the inferred industry based on pages hit. Unfortunately, I have been unable to find anything as a standalone tool - I believe HubSpot may have something like this but it is rolled into their blogging platform. Does anyone know of one?
Content Development | | SMPoulton0 -
2 URLs pointing to exactly the same content
Hi guys As far as I know if you have 2 websites with exactly the same (100%) content with 2 URLs which are not pointing to any other URL should attract penalisation from google, right? well, there is such a case and it was online for long time but the bad guys are in top of organic search and it does not seem to bother google at all! I don't want to list them here; it is extremely annoying and frustrating as I worked hard to get in higher search but seeing this thing is extremely frustrating! any advice on this? thanks
Content Development | | photoion0 -
Is there a way to repost content (with permission) to another site without being penalized by Google?
I write a monthly Social Media Marketing column for a local Business Journal and the column is printed in their paper as well as posted on their website. Is there any way I can repost these articles on my website's blog without being penalized by Google for "duplicate content"?
Content Development | | vyki0 -
Is it advisable to have unique pages for different cities/states though there wouldnt be any actual differentiation in the actual content.
Is it advisable to have unique pages for different cities/states though there wouldnt be any actual differentiation in the content. For example should we have separate pages for "hammers in california" & "hammers in new york". The product is same and content more or less the same. The search volume for individual queries is low but collectively makes a large number. The unique title tag automatically will generate traffic. So does it make sense to make 50 such pages. Else is there any way to uniquely target 50 such queries/month/city
Content Development | | DYo0 -
Will using online forum reviews create duplicate content issue?
We are looking at having a text box of 'What customers say' on our product pages using reviews written about us online to remain factual and wanted to know if this will create duplicate content issues? Thanks in advance.
Content Development | | jannkuzel0 -
How to organize content for ecommerce site
Hello, We've decided to create 24 articles of content for our ecommerce site, everything from an FAQ to history of the products to 10 articles on the top 10 products. Really useful to the user. How do you suggest that we make our content visible to the users? We could put a nice button on our right banner that says "Extensive Help Session" or we could put a banner on our home page or we could make it a tab at the top of the screen. We could additionally make a well organized footer with links to the articles. Or we could do all of those but that might be overkill. What do you suggest?
Content Development | | BobGW0 -
Why is this store getting hurt in SERPs when they removed duplicate content?
I work with an e-commerce client who got hit hard by Panda. They are very cautious, and want small-scale tests to prove each hypothesis before committing to larger changes. Recently, we reworked content on 30 product detail pages. Before, these product pages featured some original content mixed with some manufacturer content. The change we made was to remove the manufacturer content completely from the product page, leaving about 300 words of high-quality, original content--all of which was written by subject matter experts. I assumed that Google viewed this manufacturer text as duplicate content. However, when these 30 modified pages were compared to the control, they performed significantly worse. Question 1: Does any have any idea why these pages would perform worse than the control?
Content Development | | merch_zzounds
Question 2: Do you have any tips for convincing this client to try another test or get the buy-in to make the larger changes that--in theory--need to happen? FWIW, this client has about 10,000 product detail pages--the vast majority of which contain just manufacturer content. I appreciate your thoughts.0