International Sites and Duplicate Content
-
Hello,
I am working on a project where have some doubts regarding the structure of international sites and multi languages.Website is in the fashion industry. I think is a common problem for this industry. Website is translated in 5 languages and sell in 21 countries.
As you can imagine this create a huge number of urls, so much that with ScreamingFrog I cant even complete the crawling.
Perhaps the UK site is visible in all those versions
http://www.MyDomain.com/en/GB/
http://www.MyDomain.com/it/GB/
http://www.MyDomain.com/fr/GB/
http://www.MyDomain.com/de/GB/
http://www.MyDomain.com/es/GB/
Obviously for SEO only the first version is important
One other example, the French site is available in 5 languages and again...
http://www.MyDomain.com/fr/FR/
http://www.MyDomain.com/en/FR/
http://www.MyDomain.com/it/FR/
http://www.MyDomain.com/de/FR/
http://www.MyDomain.com/es/FR/
And so on...this is creating 3 issues mainly:
-
Endless crawling - with crawlers not focusing on most important pages
-
Duplication of content
-
Wrong GEO urls ranking in Google
I have already implemented href lang but didn't noticed any improvements. Therefore my question is
Should I exclude with "robots.txt" and "no index" the non appropriate targeting?
Perhaps for UK leave crawable just English version i.e. http://www.MyDomain.com/en/GB/, for France just the French version http://www.MyDomain.com/fr/FR/ and so on
What I would like to get doing this is to have the crawlers more focused on the important SEO pages, avoid content duplication and wrong urls rankings on local Google
Please comment
-
-
Hey Guido, don't know if it's the best solution, but could be a temporary fix until the best solution is in place. I suggest to move forward with proper HREF LANG tagging or definitely delete those irrelevant languages. Try to do what I said before about validate each country/language and submit a sitemap.xml reflecting that folder to see crawl and index stats pero country/language. Add a sitemap index and obviously validate your entire domain. Just block in the robots.txt unnecessary folders, like images, js libraries, etc. to save crawl budget to your domain.
Let me know if you have another doubt
-
Thank you Antonio, insightful and clear.
There is really not a need of EN versions of localized sites, I think has been done more as was easier to implement (original site is EN-US).
Don't you think robots and noindex EN version of localized sites could be the best solution? for sure is the easier one to implement without affecting UX.
-
Don't know why you have a UK oriented site for German and Italian people, I think is not important those languages in a country mainly English speaking (not US for example, there you must have a Spanish version, or in Canada for English and French). The owner must have their reasons.
Besides this, about your questions:
- If those non-relevant languages must live there, it's correct to implement HREF LANG (may take some time to show results). Also, if the domain is gTLD, you can validate all the subfolders in Google Search Console and choose the proper International targeting. With the ammount of languages and countries I imagine this might be a pain in the ***.
- About the crawling, for large sitesI recommend to crawl per language. If neccesary, per language-country. In this instance I recommend to create a sitemap XML per language or language-country for just HTML pages (hopefully dynamically updated by the e-commerce), create a Sitemap Index in the root of the domain and submit them in Google Search Console (better if you validated the languages or language-country). With this you can answer the question if some language or country are being not indexed with the Submited/Indexed stadistics of GSC.
- Maybe the robots.txt might save your crawl budget, but I'm not a fan of de-index if those folders are truly not relevant (after all, there should be a italian living in UK. If you can't delete the irrelevant langauges for some countries, this can be an option
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Setting up international site subdirectories in GSC as separate properties for better geotargeting?
My client has an international website with a subdirectory structure for each country and language version - eg. /en-US. At present, there is a single property set up for the domain in Google Search Console but there are currently various geotargeting issues I’m trying to correct with hreflang tags. My question is, is it still recommended practise and helpful to add each international subdirectory to Google Search Console as an individual property to help with correct language and region tagging? I know there used to be properly sets for this but haven’t found any up to date guidance on whether setting up all the different versions as their own properties might help with targeting. Many thanks in advance!
International SEO | | MMcCalden0 -
E-commerce : 1 site per country or 1 site per language?
I'm working with an European e-commerce; they already have a French website with a .fr domain. They want to target the Belgium with a .be domain and the Nederland with a .nl domain. Belgium = 50% dutch, 50% French. Is it better to do 3 websites, one per country, or 2 websites, one per language ? Thinking to SEO, costs, VAT management, what is your opinion?
International SEO | | johnny1220 -
International SEO Subfolders / user journey etc
Hi According to all the resources i can find on Moz and elsewhere re int seo, say in the context of having duplicate versions of US & UK site, its best to have subfolders i.e. domain.com/en-gb/ & domain.com/en-us/ however when it comes to the user journey and promoting web address seems a bit weird to say visit us at: domain.com/en-us/ !? And what happens if someone just enters in domain.com from the US or UK ? My client wants to use an IP sniffer but i've read thats bad practice and should employ above style country/language code instead, but i'm confused about both the user journey and experience in the case of multiple sub folders. Any advice much appreciated ? Cheers Dan
International SEO | | Dan-Lawrence0 -
Why would a site lose rankings in U.S while maintaining rankings in other English locations (Canada & Australia)
What would cause a site to lose ranking in the U.S while maintaining top (1st page) positions in other English results countries such as Canada or Australia? Is this purely penguin related because of location of backlinks or are there other significant factors that could be in play? Would this rule out Panda as a cause because it's simply an "English language" targeted algo and not location dependent like backlinks (penguin)? Appreciate any insights
International SEO | | ResumeGenius0 -
International SEO - Setting up reporting
What would you consider to be a best practice for setting up international keyword tracking? Both in MOZ and in Google Analytics? Would you set them up by language? IE Spanish, French, German... Or by country? Mexico, Spain, US Spanish, French Canadian, French... ect... *Our website is set up with our countries in subdomains. We currently are in about 10 different countries and plan on expanding globally. Any advice helps! Much thanks!
International SEO | | ScentsySEO0 -
Rel alternate use for multi regional website with english language and same content
I have a website targeting multi language and multi regional users. For example, my main site is www.abcd.com which is in English and targeting US. I have the same content in English which is targeting UK and India with www.uk.abcd.com and www.india.abcd.com. I want to avoid content duplication and help search engines to show the right pages on the country level searches. I have researched a bit and have come to conclusion of using re alternate tag. Can someone help me with how to place the codes for the same. Many thanks Mozers!
International SEO | | HiteshBharucha0 -
Non US site pages indexed in US Google search
Hi, We are having a global site wide issue with non US site pages being indexed by Google and served up in US search results. Conversley, we have US en pages showing in the Japan Google search results. We currently us IP detect to direct users to the correct regional site but it isn't effective if the users are entering through an incorrect regional page. At the top of each or our pages we have a drop down menu to allow users to manually select their preferred region. Is it possible that Google Bot is crawling these links and indexing these other regional pages as US and not detecting it due to our URL structure? Below are examples of two of our URLs for reference - one from Canada, the other from the US /ca/en/prod4130078/2500058/catalog50008/ /us/en/prod4130078/2500058/catalog20038/ If that is, in fact, what is happening, would setting the links within the drop down to 'no follow' address the problem? Thank you. Angie
International SEO | | Corel0 -
SEO Audit "Hybrid Site"
Hi everyone! I'm trying to analyze a website which is regional in scope. The way the site for every market has been build out is like this : http://subdomain.rootdomain.com/market | http://asiapacific.thisismybrandname.com/ph OR http://subdomain.rootdomain.com/language | http://asiapacific.thisismybrandname.com/en Since this is the first time I'm trying to work on these kinds of sites, I would want to ask for any guidance / tips on how to do about SEO site and technical audit. FYI, the owner of the sites is not giving me access / data to their webmaster account nor their analytics tracking tool. Thanks everyone! Steve
International SEO | | sjcbayona-412180