Robots.txt issue for international websites
-
In Google.co.uk, our US based (abcd.com) is showing:
A description for this result is not available because of this site's robots.txt – learn more
But UK website (uk.abcd.com) is working properly. We would like to disappear .com result totally, if possible. How to fix it?
Thanks in advance.
-
Can you share any information about your robots.txt?
-
My main problem is in the homepage. Both host similar type of products and brands.
You may check the screenshot. Sorry, I had to blanked out the text.
Thanks in advance.
-
Is it showing that for every page, or only some pages? If so, which types of pages? What's the contents of your robots.txt file for the US site?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I have multiple websites for my different brands or one main website with different tabs/areas?
My client creates apps. As well as the apps they create themselves, they have made some of their own that cover various different topics. Currently they have individual websites for each of these apps, and a website for their app making business. They are asking whether they should just have one website - their app building site, which also includes information about the two apps they've built themselves. My feeling is it's better to keep them separate. The app building site is trying to appeal to a B2B audience and gain business to build new apps. AppA is trying to help carehomes and carers to streamline their business, and AppB is trying to help workplace and employee welfare. Combining them all will mean lots of mixed messaging/keywords even if we have dedicated areas on the site. I also think it will limit how much content we can create on each without being completely overwhelming for the user. If we keep them all separate then we can have a very clear user journey. I would of course recommend having blog posts or some sort of landing page to link to AppA and AppB's websites. Thoughts? Thank you!
Intermediate & Advanced SEO | | WhitewallGlasgow0 -
Robots.txt wildcards - the devs had a disagreement - which is correct?
Hi – the lead website developer was assuming that this wildcard: Disallow: /shirts/?* would block URLs including a ? within this directory, and all the subdirectories of this directory that included a “?” The second developer suggested that this wildcard would only block URLs featuring a ? that come immediately after /shirts/ - for example: /shirts?minprice=10&maxprice=20 BUT argued that this robots.txt directive would not block URLS featuring a ? in sub directories - e.g. /shirts/blue?mprice=100&maxp=20 So which of the developers is correct? Beyond that, I assumed that the ? should feature a * on each side of it – for example - /? - to work as intended above? Am I correct in assuming that?
Intermediate & Advanced SEO | | McTaggart0 -
If Robots.txt have blocked an Image (Image URL) but the other page which can be indexed has this image, how is the image treated?
Hi MOZers, This probably is a dumb question but I have a case where the robots.tags has an image url blocked but this image is used on a page (lets call it Page A) which can be indexed. If the image on Page A has an Alt tags, then how is this information digested by crawlers? A) would Google totally ignore the image and the ALT tags information? OR B) Google would consider the ALT tags information? I am asking this because all the images on the website are blocked by robots.txt at the moment but I would really like website crawlers to crawl the alt tags information. Chances are that I will ask the webmaster to allow indexing of images too but I would like to understand what's happening currently. Looking forward to all your responses 🙂 Malika
Intermediate & Advanced SEO | | Malika11 -
If other websites implement our RSS feed sidewide on there website, can that hurt our own website?
Think about the switching anchors from the backlinks and the 100s of sidewide inlinks... I gues Google will understand that it's just a RSS feed right?
Intermediate & Advanced SEO | | Zanox0 -
Blogs and E-Commerce websites
I have recently launched an e-commerce website which has a whopping domain authority of 1! I was thinking about adding a blog to it (it's in open cart), but that would mean creating it in a wordpress but using the same domain name. Would this be beneficial from an SEO stand point (i.e sending traffic to w blog that isn't actually on the e-commerce website itself) , or am I better off creating content as blogs/articles on other people sites?
Intermediate & Advanced SEO | | lindsayjhopkins0 -
Ecommerce website consolidation
I have a large ecommerce site and several smaller nitche ecommerce sites. All have the same products, but the smaller sites are loosing traffic. I want to combine all the sites to the larger site so it will be easier to manage, but I don't want to loose any rank on the smaller sites. Example: www.yourpromopeople.com - This is the large site I want to use. www.logocoolies.com www.fourcolormagnets.com - These are a couple of the smaller sites I want to combine with the larger one. Questions: What are the pros and cons in doing this? What would be the best way to do this? Would redirecting the URL's to the larger site's product pages do the trick or is there a better option? Thanks for the help.
Intermediate & Advanced SEO | | JHSpecialty0 -
Google indexing issue?
Hey Guys, After a lot of hard work, we finally fixed the problem on our site that didn't seem to show Meta Descriptions in Google, as well as "noindex, follow" on tags. Here's my question: In our source code, I am seeing both Meta descriptions on pages, and posts, as well as noindex, follow on tag pages, however, they are still showing the old results and tags are also still showing in Google search after about 36 hours. Is it just a matter of time now or is something else wrong?
Intermediate & Advanced SEO | | ttb0 -
Robots.txt is blocking Wordpress Pages from Googlebot?
I have a robots.txt file on my server, which I did not develop, it was done by the web designer at the company before me. Then there is a word press plugin that generates a robots.txt file. How Do I unblock all the wordpress pages from googlebot?
Intermediate & Advanced SEO | | ENSO0