Robots.txt issue for international websites
-
In Google.co.uk, our US based (abcd.com) is showing:
A description for this result is not available because of this site's robots.txt – learn more
But UK website (uk.abcd.com) is working properly. We would like to disappear .com result totally, if possible. How to fix it?
Thanks in advance.
-
Can you share any information about your robots.txt?
-
My main problem is in the homepage. Both host similar type of products and brands.
You may check the screenshot. Sorry, I had to blanked out the text.
Thanks in advance.
-
Is it showing that for every page, or only some pages? If so, which types of pages? What's the contents of your robots.txt file for the US site?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Restructuring Areas of a Website
Hi We are changing the structure of 2 areas of our site but the URL's won't be changing. We're effectively removing the first category level as it doesn't make much sense: Current structure Cat 1 - http://www.key.co.uk/en/key/cupboards-lockers Cat 2s - http://www.key.co.uk/en/key/lockers & http://www.key.co.uk/en/key/cupboards Cat 3s... New structure will look like Cat 1's http://www.key.co.uk/en/key/lockers & http://www.key.co.uk/en/key/cupboards Cat 2's.... etc The top category 1 doesn't rank for much & the level 2's perform better anyway. Will moving the structure change rankings even though the URLs don't change - just what is assigned to them in the back changes I know if the on-page content changes, things may be affected, but we're minimising this as much as possible. Thank you
Intermediate & Advanced SEO | | BeckyKey0 -
INTERNAL LINKS strategy on our website
Hi Moz-ers, Currently doing an audit of our website. I have two questions on links. How can I see the current state of my internal links? Also, how can I improve our internal links on the website? what is a good framework to follow what should I avoid Thanks, looking forward to learning more on Moz!
Intermediate & Advanced SEO | | Eric_S
Eric0 -
What is internal like structure best for website
I want to construct website internal like structure better, can you advise me what's model architecture to build menu, navigation, build link, hub content will good for audience and search engine. Thank your advise
Intermediate & Advanced SEO | | dunghv360 -
Robots.txt vs noindex
I recently started working on a site that has thousands of member pages that are currently robots.txt'd out. Most pages of the site have 1 to 6 links to these member pages, accumulating into what I regard as something of link juice cul-d-sac. The pages themselves have little to no unique content or other relevant search play and for other reasons still want them kept out of search. Wouldn't it be better to "noindex, follow" these pages and remove the robots.txt block from this url type? At least that way Google could crawl these pages and pass the link juice on to still other pages vs flushing it into a black hole. BTW, the site is currently dealing with a hit from Panda 4.0 last month. Thanks! Best... Darcy
Intermediate & Advanced SEO | | 945010 -
Sitemap Issue - vol 2
Hello everyone! I validated the sitemap with different tools (w3Schools, and so on..) and no errors were found. So I uploaded into my site, tested it through GWT and BANG! all of a sudden there is a parsing error, which correspond to the last, and I mean last piece of code of thousand of lines, . I don't know why it isn't reading the code and it's giving me this as there are no other errors and I haven't got a clue about what to do in order to fix it! Thanks
Intermediate & Advanced SEO | | PremioOscar0 -
International SEO and server hosting
I'd appreciate feedback on a situation. We're going through a major overhaul in how we globally manage our websites. Regional servers were part of our original plan (one in Chicago, UK, and APAC) but we've identified a number of issues with this approach. Although it's considered a best practice among many, the challenges we'd face doing it are considerable (added complexity, added steps and delays to updating sites, among others). So, we shifted our plan and how are looking at hosting here in the US but to use Akami to deliver images and other heavier data pieces from their local servers (in the UK, etc.). This is how many of the larger companies like Amazon, etc. delivery their global websites. We hope that using Akami will allow us to have good performance while simplifying our process. Any warning signs we should be aware of? Anyone doing it this way and has a good experience/bad experience?
Intermediate & Advanced SEO | | josh-riley0 -
Content Marketing: Should we build a separate website or built in site within the Website itself?
Hi Mozzers, Client: Big carpet cleaner player in the carpet cleaning industry Main Goal: Creating good content to Get more organic traffic to our main site Structure of the extra content: It will act like a blog but will be differentiated from the regular site by not selling anything but just creating good content. The look and design will be different from the client's site. SEO question: In terms of SEO, what would be the most beneficial for us to do, should we built in this new section/site outside or inside the client's site? I personally think that it should be separated from the main site because of the main reasons: A followed link to the main site Anchor texts implementation linking back to our service pages If we would to choose to build in this content, it would be highly beneficial for getting organic traffic within the main site but I am afraid this will not provide us any link juice since anchor texts won't be accounted the same since all of those would be located in the Nav bar of the main site. Can someone tell me what would be the best in terms of SEO? P.S: My boss doesn't agree with me and would rather go the second option (build in within the main site) that's why i am asking you guys what would be the most beneficial? Thank you Guys
Intermediate & Advanced SEO | | Ideas-Money-Art0 -
Using 2 wildcards in the robots.txt file
I have a URL string which I don't want to be indexed. it includes the characters _Q1 ni the middle of the string. So in the robots.txt can I use 2 wildcards in the string to take out all of the URLs with that in it? So something like /_Q1. Will that pickup and block every URL with those characters in the string? Also, this is not directly of the root, but in a secondary directory, so .com/.../_Q1. So do I have to format the robots.txt as //_Q1* as it will be in the second folder or just using /_Q1 will pickup everything no matter what folder it is on? Thanks.
Intermediate & Advanced SEO | | seo1234560