Wp-login.php
-
To block or not to block with robots.txt?
-
Michael, the best thing to do for you would be to check Google Webmasters Tools and see what Google sees there as duplicate content.
This will help you decide whether you want or not to block /tag/.
-
I get that. Just not sure about blocking them. A lot of sites don't block them and yoast.com is one of them. So, if big seo guys aren't blocking tag then are we 100% sure that Google sees "/tag/" as duplicate content?
-
If you're happy with your results, why block anything in Robots? The short term answer is tags are working for you. The long term answer is you're creating duplicate content. So it just depends - are you optimizing for today and tomorrow's sales or for long term success. There is no "right" answer to that but that's the question.
-
With all the talk here about not indexing /tag/ why do you not include it in your robots?
-
I understand that but what's happening is that i have tag results in the top 10 and not the post result. The post result is indexed but not anywhere close to the tag. If i block tags, in my head, I think i run the risk of losing traffic. I have to be 100% certain of this for I'm getting and been getting a lot of traffic from those "tag" results.
-
You don't want to index tags in Wordpress. You'll have duplicate content on any post that has more than one tag on it (which is almost all of them.) You want to get the canonical permalinks indexed and that's it. Search for a post you have indexed under a tag and then do a keyword search to see if it appears a few times (archives, multiple tag pages, etc.) That's bad - you want to fix that.
-
I would start with a robots.txt like the one below and add in anything else you don't want to be crawled/indexed:
Sitemap: http://www.example.com/sitemap.xml # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: Mediapartners-Google* Disallow: # digg mirror User-agent: duggmirror Disallow: / # global User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/plugins/ Disallow: /wp-content/cache/ Disallow: /wp-content/themes/ Disallow: /trackback/ Disallow: /feed/ Disallow: /comments/ Disallow: /category/*/* Disallow: */trackback/ Disallow: */feed/ Disallow: */comments/ Disallow: /*? Allow: /wp-content/uploads/
-
Not sure I follow. I have indexed tag results on Google that have the /tag/ in the url. If i noindex them, won't i lose those urls that get indexed?
-
I block wp-admin and wp-includes just because they don't need to be indexed.
As far as tags, you can easily no-follow all of those with any half decent Wordpress SEO plugin like Yoast or All in One SEO. I would definitely noindex them.
-
You may want to add a short description and custom titles for those tags. There are some ways to do that in WordPress.
This way your content will have something unique and will not be 100% duplicate.
-
Thanks for the reply. Makes sense. What about tags? Should they be blocked in robots.txt too? Reason I ask is that i have indexed tag content in Google and if I blog /tags/ from being indexed then I'm worried that I'll lose that indexed content. The reason for the blocking of /tags/ is because seomoz is finding duplicate content from /tags/ and I don't want that to hurt me.
-
Do you have any links from your main pages pointing to wp-login.php?
If not, the search engines will not know about that URL, so you can choose to not block it.
But if you have links to wp-login.php and don't want the page indexed, then you should block it.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Removing non www and index.php
Hi, I'm green when it comes to altering the htaccess file to remove non www and index.php. I think I've managed to redirect the urls to www however not sure if I've managed to remove the index.php. I'm pasting the contents of the htaccess file here maybe someone can identify if I have unwanted lines of code and if it is up to standard (there are a lot of comments in #) not sure if needed but I've left them as I don't want to screw up anything. Thanks 🙂 @package Joomla @copyright Copyright (C) 2005 - 2016 Open Source Matters. All rights reserved. @license GNU General Public License version 2 or later; see LICENSE.txt READ THIS COMPLETELY IF YOU CHOOSE TO USE THIS FILE! The line 'Options +FollowSymLinks' may cause problems with some server configurations. It is required for the use of mod_rewrite, but it may have already been set by your server administrator in a way that disallows changing it in this .htaccess file. If using it causes your site to produce an error, comment it out (add # to the beginning of the line), reload your site in your browser and test your sef urls. If they work, then it has been set by your server administrator and you do not need to set it here. No directory listings IndexIgnore * Can be commented out if causes errors, see notes above. Options +FollowSymlinks
On-Page Optimization | | KeithBugeja
Options -Indexes Mod_rewrite in use. RewriteEngine On
RewriteCond %{REQUEST_URI} ^/index.php/
RewriteRule ^index.php/(.*) /$1 [R,L] Begin - Rewrite rules to block out some common exploits. If you experience problems on your site then comment out the operations listed below by adding a # to the beginning of the line. This attempts to block the most common type of exploit attempts on Joomla! Block any script trying to base64_encode data within the URL. RewriteCond %{QUERY_STRING} base64_encode[^(]([^)]) [OR] Block any script that includes a0 -
How to transfer old WP blog to new URL
I have a 9 year old WP website with a WP blog which is still getting 300+ new visitors a day even though I have not written a blog for 5 years and have not updated content. Some posts have over 25,000 links. However the Moz analytics is fraught with significant errors-404 redirects, page not found, dup content, no metatags, title too long etc. I was totally inexperienced 5 years ago and made many errors. However the basic content was sound and still is producing new visitors. I am starting a new ecommerce website using the same name but the URL and server will be different. I want to transfer my WP blog to the new site. I am concerned however that bringing the posts over can create the same errors on the new site. If I update all of the blogs on the old site using Yoast before transferring the blog to the new site will that help. I suppose I could check those flagged dup content and only transfer one of that category?
On-Page Optimization | | wianno1680 -
Index.php getting Duplicate page content.
I am quite new to SEO and have now got my first results. I am getting all my index.php pages returned as Duplicate page content. ie: blue-widgets/index.php
On-Page Optimization | | ivoryred
blue-widgets/ green-widgets/large/index.php
green-widgets/large/ How do solve this issue?0 -
Indexed iframed content behind login
Hi, I have a question regarding iframed content. I would like to get my non cms content which is served via an iframe solution (from the same domain) behind a anonymous or personal login indexed by search engines. How can we make this work? I've looked at the following solutions: http://googlewebmastercentral.blogspot.nl/2008/10/first-click-free-for-web-search.htmlhttp://productforums.google.com/forum/#!topic/webmasters/l9n8oGLQRkUBut I would like the content to be crawlable deeper than the just one page (if this is possible using the iframe solution).We could also setup different new pages in our CMS with the same content...Any suggestions?Thanks!Arnout
On-Page Optimization | | hellemans0 -
Website redesign: site going from .php to .html
A site I'm working on is being redesigned because the current platform does not allow for content to be changed easily. In the process, they are going from .php to .html. I am concerned about their losing link juice. Can a site work with the old content remaining .php and the new content being .html or should all pages stay .php?
On-Page Optimization | | cakelady0 -
Duplicate page content & title for www.mydomain.com and www.mydomain.com/index.php?
Hi, First post so please be gentle! My Crawl Diagnostics Summary is showing an error relating to duplicate page content and duplicate page title for www.mydomain.com and www.mydomain.com/index.php which are, in my view, the same thing/page? Could anyone shed any light please? Thanks Carl
On-Page Optimization | | Carl2870 -
Rename index.php or keyword in URL?
It is important for me to get good search results for keyword + city name For instance: tulips amsterdam What would be better: renaming index.php or adding the cityname to the URL? www.example.com/amsterdam/tulips OR www.example.com/pages/tulips-amsterdam
On-Page Optimization | | svdg0 -
How To Prevent Crawling Shopping Carts, Wishlists, Login Pages
What's the best way to prevent engines from crawling your websites shopping cart, wishlist, log in pags, ect... Obviously have it in robots.txt but is their any other form of action that should be done?
On-Page Optimization | | Romancing0