Robots.txt issue for international websites
-
In Google.co.uk, our US based (abcd.com) is showing:
A description for this result is not available because of this site's robots.txt – learn more
But UK website (uk.abcd.com) is working properly. We would like to disappear .com result totally, if possible. How to fix it?
Thanks in advance.
-
Can you share any information about your robots.txt?
-
My main problem is in the homepage. Both host similar type of products and brands.
You may check the screenshot. Sorry, I had to blanked out the text.
Thanks in advance.
-
Is it showing that for every page, or only some pages? If so, which types of pages? What's the contents of your robots.txt file for the US site?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 Question - issue
A while back we had a 'bleed' on one of our sites, which basically meant one of our sites started to leak across pages to another and that site started to rank for the same pages and now we have hundreds of pages ranking for urls that do not exists. It's hard to explain, bare with me. If you were to click on the cached view in Google for the ranked page it would show you the main site, but if you were to click it as usual, then you would be taken to the site but a 404 would show as the intended page was not for that site. We believe we fixed the 'bleed' and have setup 301s for all the affected pages to go to the home page for the site it affected. But these pages have not been removed from Google, which we thought a 301 would do. So we still have hundreds of pages being ranked but are redirected to the home page. Why hasn't these pages been removed?
Intermediate & Advanced SEO | | JH_OffLimits0 -
International SEO Options?
Hi we currently have a site which is a example.com domain in the Australian market (we have geo-targeted to Australia within search console). We are looking to expand to United States. I have added the potential options down below, just wondering which one you guys think would be best from a SEO and practical standpoint? Or if there are other options i should consider? Option 1 The Australian domain is strong so this option takes this into consideration. Keep example.com (Australian) Add on: Sub-Directory for US Which would be: example.com/us/ In Search Console set the sub-folder to target US and also setup hreflang tags. Setup the US site on the sub-directory. Option 2 Add sub-folders for both Aus and US example/au/ (Australian)
Intermediate & Advanced SEO | | jaynamarino
example/us/ (United States) Setup hreflang targeting. Cons
Need to set up redirects for the current site to new location which is .com/au/ might also see drop in performance due to redirects. Cheers.0 -
Need help for new website!
I want to a make new website. Can you please advise me what all things are involved which I should keep in mind before and during the website preparation. Like how to make pages, what to include in website, best way to create pages etc. Please provide me the link where I can study all the above information. I am planning to create global printing website.
Intermediate & Advanced SEO | | AlexanderWhite0 -
New Website. Changing TLD or not?
Hi, At my company we are making a new website because the days of the old one are numbered. We already decided that the folder structure will be changed so we have more "clean" url's. Now we also would like to change from .net/nl to .nl . Since we already are redirecting all url's (>10.000), we think this is the moment to switch the TLD. What do you guys think? Is their anyone who has some kind of experience/tip they would like to share?
Intermediate & Advanced SEO | | SEO_ACSI0 -
Ajax website and SEO
Hi all, A client of mine has a website similar to Pintrest. All in Ajax/. So imagine an ajax-grid based animal lover site called domain.com. The domain has three different Categories Cats, Dogs, Mice. When you click on a category, the site doesn't handle the URL and doesn't change the domain So instead of the domain going from domain.com to domain.com/cats, it uses the Ajax script and just shows all the cat pins. and when you click on each pin/post it opens a page such as domain.com/Pin/123/PostTitle It doesn't reference the category. However a page domain.com/cats does exist and you can go there directly. Is this an SEO issue for not grouping all pins under a category? How does Google handle Ajax these days, it use to be real bad but if Pintrest is going so well i'm assuming times have changed? Any other things to be wary of for a grid based/ajax site? I am happy to pay for an hour or two for a more in depth audit/tips if you can feed back on the above. Fairly urgent. Thanks
Intermediate & Advanced SEO | | Profero1 -
How canonical url harm our website???
Even though my website has no similar/copied content, i used rel=canonical for all my website pages. Is Google or yahoo make any harm to my SERP's?? EX: http://www.seomoz.org is my site, in that i used canonical as rel="<a class="attribute-value">canonical</a>" href="http://www.seomoz.org" to my home page like that similar to all pages, i created rel=canonical. Is search engine harm my website???
Intermediate & Advanced SEO | | MadhukarSV0 -
Indexing issue?
Hey guys when I do a search of site:thetechblock.com query in Google I don't seem to see any recent posts (nothing for August). In Google webmaster I see that the site is being crawled (I think), but I'm not sure. I also see the the sitemaps are being indexed but again it just seems really odd that I'm not seeing these in Google results. SEO seems all good too with SEO Moz. Is there something I'm not getting?
Intermediate & Advanced SEO | | ttb0 -
Internal competition
If we have two different sub domain pages that talk about the same service but with different content, how will Google react while ranking? Example : xxxx.abc.com/company/solutions/service1 yyyy.abc.com/service1 Suppose if www.abc.com has good authority, which URL will be more benefited?
Intermediate & Advanced SEO | | gmk15670