What is the appropriate Robot.txt to unblock if Google cannot get all the resources from my homepage?
-
Hello everyone. I did some research as to why my website has decreased in the Google search rankings recently. After reading this Yoast article I believe it's because the robot.txt files I have set up on my wordpress website. The following is a screen shot of the results of a fetch & render query of my webpage.Googlebot couldn't get all resources for this page. Here's a list:
My current robot.txt settings are as follows;
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: */xmlrpc.php Disallow: */wp-*.php Disallow: */trackback/ Disallow: *?wptheme= Disallow: *?comments= Disallow: *?replytocom Disallow: */comment-page- Disallow: *?s= Disallow: */wp-content/ Allow: */wp-content/uploads/ ```What to I need to allow/disallow to allow Google spiders to properly read my website?
-
Remove the following line:
Disallow: */wp-content/ And add: Allow: /wp-includes/js/
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Googles Search Intent – Plural & Singular KW’s
This is more of a ‘gripe’ than a question, but I would love to hear people’s views. Typically, when you search for a product using the singular and plural versions of the keyword Google delivers different SERPs. As an example, ‘leather handbag’ and ‘leather handbags’ return different results, but surely the search intent is exactly the same? You’d have thought Google was now clever enough to work this out. We tend to optimise our webpages for both the plural and singular variations of the KW’s, but see a mixed bag of results when analysing rankings. Is Google trying to force us to create a unique webpage for the singular version, and another unique webpage for the plural version? This would confuse the visitor, and make no sense.. the search intent is the same! How do you combat this problem? Many thanks in advance. Lee.
Algorithm Updates | | Webpresence0 -
Fred Google Update & Ecommerce Sites
Hi I've seen a couple areas of our site drop in average rankings for some areas since the 'Fred' update. We don't have ads on our site, but I'm wondering if it's 'thin' content - http://www.key.co.uk/en/key/ We are an ecommerce site and we have some content on our category pages - which is a bit more generic about the section/products within that section - but how can it not be if it's a category page with products on? I am working on adding topic based content/user guides etc to be more helpful for customers, but I'd love some advice on generating traffic to category pages. Is it better to rank these other topic/user guide pages instead of the category page & then hope the customer clicks through to products? Advice welcome 🙂
Algorithm Updates | | BeckyKey0 -
How long for google to de-index old pages on my site?
I launched my redesigned website 4 days ago. I submitted a new site map, as well as submitted it to index in search console (google webmasters). I see that when I google my site, My new open graph settings are coming up correct. Still, a lot of my old site pages are definitely still indexed within google. How long will it take for google to drop off or "de-index" my old pages? Due to the way I restructured my website, a lot of the items are no longer available on my site. This is on purpose. I'm a graphic designer, and with the new change, I removed many old portfolio items, as well as any references to web design since I will no longer offering that service. My site is the following:
Algorithm Updates | | rubennunez
http://studio35design.com0 -
Google is simplifying my meta titles. What does it mean?
Hi Guys. I've noticed that google seems to be simplifying my meta titles in their SERP for some keyword searchs. Should I be worried about this? Perhaps a sign of being a little 'spammy' in my meta titles? Or is Google just aiming to bring up the most relevant looking result? Based on that specific search? Isaac.
Algorithm Updates | | isaac6630 -
Next Google PR update
When is next google Pagerank update is expected to arrive.
Algorithm Updates | | csfarnsworth
I know it takes one month to one year for Google to update it but I know many people sitting here at Moz know some secrets for sure.0 -
How can I tell Google two sites are non-competing?
We have two sites, both English language. One is a .ca and the other is a .com, I am worried that they are hurting one another in the search results. I'd like to obviously direct google.ca towards the .ca domain and .com towards the .com domain and let Google know they are connected sites, non-competing.
Algorithm Updates | | absoauto0 -
How could Google define "low quality experience merchants"?
Matt Cutts mentioned at SXSW that Google wants to take into consideration the quality of the experience ecommerce merchants provide and work this into how they rank in SERPs. Here's what he said if you missed it: "We have a potential launch later this year, maybe a little bit sooner, looking at the quality of merchants and whether we can do a better job on that, because we don’t want low quality experience merchants to be ranking in the search results.” My question; how exactly could Google decide if a merchant provides a low and high quality experience? I would image it would be very easy for Google to decide this with merchants in their Trusted Store program. I wonder what other data sets Google could realistically rely upon to make such a judgment. Any ideas or thoughts are appreciated.
Algorithm Updates | | BrianSaxon0 -
Do we have a timeline of google, bing updates
I thought it would be handy if we had a timeline with dates of any updates to the algo's.
Algorithm Updates | | AlanMosley
Does one exists here at SEOMoz or elsewhere.
Thanks3