Default Robots.txt in WordPress - Should i change it??
-
I have a WordPress site as using theme Genesis i am using default robots.txt. that has a line Allow: /wp-admin/admin-ajax.php, is it okay or any problem. Should i change it?
-
Yes, we're a news site as well and in our case we want to make sure the low quality pages on TNW aren't indexed.
-
Thank you both for your response.
@Martijn your robots.txt is really a nice example but for my new site is it good practice to block this areas??
@Peter To be a safe side I was using the same robots.txt...
-
In addition of Martijn here is mine robots.txt:
User-agent: *
Disallow:Sitemap: http://peter.nikolow.me/sitemap_index.xml
But using Yoast - categories, tags, most of archives and other generated pages are disabled for indexing.
-
Hi Peter,
Usually I would say it's not enough as the robots.txt is forgeting about excluding the search pages and in most cases you want to make sure the WP core files are not included + tag pages. Take a look at our robots.txt to see what we've included there: http://thenextweb.com/robots.txt then you'll notice we include for example these:
User-agent: *
Disallow: ?p=
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /wp-admin/*
Disallow: /wp-register.php
Disallow: /wp-content/themes/icetea/includes/*
Disallow: /tag/
Disallow: ?s=
Disallow: /search/*Other cases in our robots.txt are very specifically in there because of our site and may not apply to others.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Changing Canonical Tags on Indexed Pages that are Ranking Well
Hi Guys, I recently rolled out a domain wide canonical tag change. Previously the website had canonical tags without the www, however the website was setup to redirect to www on page load. I noticed that the site competitors were all using www and as far as I understand www versus non www, it's based on preference. In order to keep things consistent, I changed the canonical tag to include the www. Will the site drop in rankings? Especially if the pages are starting to rank quite well. Any feedback is appreciated. Thanks!
Intermediate & Advanced SEO | | QuickToImpress0 -
Block subdomain directory in robots.txt
Instead of block an entire sub-domain (fr.sitegeek.com) with robots.txt, we like to block one directory (fr.sitegeek.com/blog).
Intermediate & Advanced SEO | | gamesecure
'fr.sitegeek.com/blog' and 'wwww.sitegeek.com/blog' contain the same articles in one language only labels are changed for 'fr' version and we suppose that duplicate content cause problem for SEO. We would like to crawl and index 'www.sitegee.com/blog' articles not 'fr.sitegeek.com/blog'. so, suggest us how to block single sub-domain directory (fr.sitegeek.com/blog) with robot.txt? This is only for blog directory of 'fr' version even all other directories or pages would be crawled and indexed for 'fr' version. Thanks,
Rajiv0 -
Best way to structure urls wordpress and Yoast?
I am using Wordpress and Yoast. I have Parent pages and child pages. Yoast recommends you have the keyword in the url. For the parent page I have the city name in the url. Question is, should the child pages also have the city name in the url or would that be considered keyword stuffing? Here is the current structure. http://forestparkdental.info/st-louis-dental-services/restorative-dentistry/inlays-and-onlays So didn't know if should have the end of that url as /restorative-dentistry-st-louis /inlays-and-onlays-st louis since those are separate pages and Yoast and Moz plugin doesn't give you the Green light in in all areas unless you do it like this? Thanks Scott
Intermediate & Advanced SEO | | scott3150 -
Yoast Wordpress SEO premium
As anyone installed Yoast Wordpress SEO premium. Love to ear some feedbacks. I have some problems on installations that i am trying to solve as i am not an expert on SEO.
Intermediate & Advanced SEO | | maestrosonrisas0 -
Multilingual blog in wordpress needs URL suggestion
I am working on a multilingual blog build in WordPress.From the first day I see the URL structure getting abrupt when I add an article in other language.
Intermediate & Advanced SEO | | csfarnsworth
The following is an example of abrupt URL.
http://muslim-academy.com/%D9%81%D8%B6%D9%84-%D9%82%D8%B1%D8%A7%D8%A1%D8%A9-%D8%A7%D9%84%D9%82%D8%B1%D8%A2%D9%86-3/ is their some plugin to fix it or some manual change?0 -
Changing the G+ Author
Hello, We have a website we are linking using schema / google places to a G+ profile. however, this is a person in the company and may change in the future. Is there any issue (e.g. TOS issue) with Google to have one author of a website and then change to another (e.g. change from author being one G+ account ot another with a new image/profile/person)? Thanks!
Intermediate & Advanced SEO | | onlinemktg10 -
Wordpress: Too Many Links + Trackback 302
Hey, I see that all the blogposts that we have done to date (6) are being showing as having too many on page links in the seomoz crawl but I am quite confused about this as I cannot count more than 30 (including side bar, footer and header) per post. Can anyone shed any light on why this may be occuring and/or how I can check which links are being picked up? Secondly I have a number of temporary redirect warnings all related to the blog. The 'trackback' URL of each post to date has a 302 direct to it's respective blog post. What is the best solution here? Change to a 301 possibly? Any help will be greatly appreciated. Thanks in advance.
Intermediate & Advanced SEO | | jannkuzel0 -
How do I go about changing a 302 redirect to a 301.
Hello Friends! Thanks for viewing my question. Ok,My question today is How do I go about redirecting a 302 link to a 301 link. I understand the benefits of doing this as far as link juice and how the Search Engines views the two Re-Directs. I am wanting to know where I would start to do this. Thank you in advance for any help or suggestions!
Intermediate & Advanced SEO | | FrontlineMobility0