Will a Robots.txt 'disallow' of a directory, keep Google from seeing 301 redirects for pages/files within the directory?
-
Hi- I have a client that had thousands of dynamic php pages indexed by Google that shouldn't have been. He has since blocked these php pages via robots.txt disallow. Unfortunately, many of those php pages were linked to by high quality sites mulitiple times (instead of the static urls) before he put up the php 'disallow'.
If we create 301 redirects for some of these php URLs that area still showing high value backlinks and send them to the correct static URLs, will Google even see these 301 redirects and pass link value to the proper static URLs? Or will the robots.txt keep Google away and we lose all these high quality backlinks? I guess the same question applies if we use the canonical tag instead of the 301. Will the robots.txt keep Google from seeing the canonical tags on the php pages?
Thanks very much,
V
-
No problem
-
Hello Dmitrii,
Yes, that clarifies things perfectly. Thanks very much for your explanation. And I missed this particular WBF, so I will give it a close look as well.
Thanks again for your quick help.
-
Hello, my friend.
You should realize how exactly htaccess' 301 redirects work. They are server side commands/operations. So, when bots request a page, they wait until server response. In case of 301s - they get response "Don't go here, go there". Now, they also may get response from robots.txt saying "you're not allowed to look at the contents of this file/directory", however this will not prevent the server response. That's why sometimes you can see indexed pages, which are saying "blocked by robots". They are indexed though.
Now, in case of canonical links you are correct, since canonical is IN the content of the page, then robots won't be able to read it, therefore won't be able to be told that there is a canonical page.
There is a recent WBF on this subject - https://mza.seotoolninja.com/blog/controlling-search-engine-crawlers-for-better-indexation-and-rankings-whiteboard-friday
Hope this clarifies some things.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 Domain Redirect And Old Domain to a New one including pages
Hi, I need to 301 an old domain to a new one (new website) I need to 301 the domain to a new page not the new domain direct for example www.olddomain.co.uk to www.newdomain.co.uk/pagenew Then I need to also 301 all the other pages on the old domain to the new one for example... www.oldmain.co.uk/oldpage to www.newdomain.co.uk/newpage Issue is I can do one or the other not both, I can get the other pages to redirect but then the main domain wont redirect to the correct new page. Or I can get the old domain to redirect but not the internal pages. Thanks
Technical SEO | | David-Sharpe0 -
Selective 301 redirections of pages within folders
Redirection Puzzle - it's got me puzzled anyhow! The finished website has just been converted from an old aspx affair to a wordpress site. Some directory structures have changed significantly; there appears to be a load of older medical articles that have not been added back in and it sounds unlikely that they will be. Therefore unmatched old news articles need to be pointed to the top news page to keep hold of any link value they may have accrued. The htaccess file starts with ithemes security's code, Followed by the main wordpress block and I have added the user redirects to the final section of the htaccess file . I have been through the redirects and rewrites line by line to verify them and the following sections are giving me problems. This is probably just my aging brain failing to grasp basic logic. If I can tap into anybody's wisdom for a bit of help I would appreciate it. My eyes and brain are gone to jelly. I have used htaccesscheck.com to check out the underlying syntax and ironed out the basic errors that I had previously missed. The bulk of the redirects are working correctly. #Here there are some very long media URLs which are absent on the new site and I am simply redirecting visiting spiders to the page that will hold media in future. Media items refuse to redirect
Technical SEO | | TomVolpe
Line 408 redirect 301 /Professionals/Biomedicalforum/Recordedfora/Rich%20Media%20http:/kplayer.kcl.ac.uk/ess/echo/presentation/15885525-ff02-4ab2-b0b9-9ba9d97ca266 http://www.SITENAME.ac.uk/biomedical-forum/recorded-fora/ Line 409 redirect 301 /Professionals/Biomedicalforum/Recordedfora/Quicktime%20http:/kplayer.kcl.ac.uk/ess/echo/presentation/15885525-ff02-4ab2-b0b9-9ba9d97ca266/media.m4v http://www.SITENAME.ac.uk/biomedical-forum/recorded-fora/ Line 410 redirect 301 /Professionals/Biomedicalforum/Recordedfora/Mp3%20http:/kplayer.kcl.ac.uk/ess/echo/presentation/15885525-ff02-4ab2-b0b9-9ba9d97ca266/media.mp3 http://www.SITENAME.ac.uk/biomedical-forum/recorded-fora/ #Old site pagination URLs redirected to new "news" top level page - Here I am simply pointing all the pagination URLs for the news section, that were indexed, to the main news page. These work but append the pagination code on to the new visible URL. Have I got the syntax correct in this version of the lines to suppress the appended garbage? RewriteRule ^/LatestNews.aspx(?:.*) http://www.SITENAME.ac.uk/news-events/latest-news/? [R=301,L] #On the old site many news directories (blog effectively) contained articles that are unmatched on the new site, have been redirected to new top level news (blog) page: In this section I became confused about whether to use Redirect Match or RewriteRule to point the articles in each year directory back to the top level news page. When I have added a redirectmatch command - it has been disabling the whole site! Despite my syntax check telling me it is syntactically correct. Currently I'm getting a 404 for any of the old URLs in these year by year directories, instead of a successful redirect. I suspect Regex lingo is not clicking for me 😉 My logic here was rewrite any aspx file in the directory to the latest news page at the top. This is my latest attempt to rectify the fault. Am I nearer with my syntax or my logic? The actual URLs and paths have been substituted, but the structure is the same). So what I believe I have set up is: in an earlier section; News posts that have been recreated in the new site are redirected 1 - 1 and they are working successfully. If a matching URL is not found, when the parsing of the file reaches the line for the 1934 directory it should read any remaining .aspx URL request and rewrite it to the latest news page as a 301 and stop processing this block of commands. The subsequent commands in this block repeat the process for the other year groups of posts. Clearly I am failing to comprehend something and illumination would be gratefully received. RewriteRule ^/Blab/Blabbitall/1934/(.*).aspx http://www.SITENAME.ac.uk/news-events/latest-news/ [R=301,L] #------Old site 1933 unmatched articles redirected to new news top level page RewriteRule ^/Blab/Blabbitall/1933/(.*).aspx http://www.SITENAME.ac.uk/news-events/latest-news/ [R=301,L] #------Old site 1932 unmatched articles redirected to new news top level page RewriteRule ^/Blab/Blabbitall/1932/(.*)/.aspx http://www.SITENAME.ac.uk/news-events/latest-news/ [R=301,L] #------Old site 1931 unmatched articles redirected to new news top level page RewriteRule ^/Blab/Blabbitall/1931/(.*)/.aspx http://www.SITENAME.ac.uk/news-events/latest-news/ [R=301,L] #------Old site 1930 unmatched articles redirected to new news top level page RewriteRule ^/Blab/Blabbitall/1930/(.*)/.aspx http://www.SITENAME.ac.uk/news-events/latest-news/ [R=301,L] Many thanks if anyone can help me understand the logic at work here.0 -
New Page Showing Up On My Reports w/o Page Title, Words, etc - However, I didn't create it
I have a WordPress site and I was doing a crawl for errors and it is now showing up as of today that this page : https://thinkbiglearnsmart.com/event-registration/?event_id=551&name_of_event=HTML5 CSS3 is new and has no page title, words, etc. I am not even sure where this page or URL came from. I was messing with the robots.txt file to allow some /category/ posts that were being hidden, but I didn't re-allow anything with the above appendages. I just want to make sure that I didn't screw something up that is now going to impact my rankings - this was just a really odd message to come up as I didn't create this page recently - and that shouldnt even be a page accessible to the public. When I edit the page - it is using an Event Espresso (WordPress plugin) shortcode - and I don't want to noindex this page as it is all of my events. Sorry this post is confusing, any help or insight would be appreciated! I am also interested in hiring someone for some hourly consulting work on SEO type issues if anyone has any references. Thank you!
Technical SEO | | webbmason0 -
301 redirects reverting to 302 redirects
We recently built a new website with a new site structure. To prevent there being a load of 404's I redirected the old pages to the new relevant pages with 301 redirects. A few days later the SEOmoz crawl report alerted me to a load of 302 redirects. When I looked into this for some reason all of the 301 redirects I set up are reverting to 302 redirects. I did a test by 301 redirecting a made up URL to an existing page and the same thing happens - it 302 redirects. I can't find any settings in WordPress to possibly explain why this is happening. Has anyone got any ideas why this could be?
Technical SEO | | Tone_Agency0 -
Block or remove pages using a robots.txt
I want to use robots.txt to prevent googlebot access the specific folder on the server, Please tell me if the syntax below is correct User-Agent: Googlebot Disallow: /folder/ I want to use robots.txt to prevent google image index the images of my website , Please tell me if the syntax below is correct User-agent: Googlebot-Image Disallow: /
Technical SEO | | semer0 -
Does Bing ignore robots txt files?
Bonjour from "Its a miracle is not raining" Wetherby Uk 🙂 Ok here goes... Why despite a robots text file excluding indexing to site http://lewispr.netconstruct-preview.co.uk/ is the site url being indexed in Bing bit not Google? Does bing ignore robots text files or is there something missing from http://lewispr.netconstruct-preview.co.uk/robots.txt I need to add to stop bing indexing a preview site as illustrated below. http://i216.photobucket.com/albums/cc53/zymurgy_bucket/preview-bing-indexed.jpg Any insights welcome 🙂
Technical SEO | | Nightwing0 -
301 Redirect
The SEOmoz crawl campaign found some 404 errors in my Joomla site poker-brands.ca. So, I figured I would set up 301 redirects in my hosting account to make sure bots don't read that there is a page missing. For example: This link gave a 404 error in the crawl: http://www.poker-brands.ca/download-pokertracker-software/holdemmanager I redirected it to: http://www.poker-brands.ca/download-pt3-pokertracker-software/holdemmanager-hem-hm2 However, when I visit the first link it doesn't send me to the second link. Am I supposed to get forwarded to the second link now?
Technical SEO | | Uramark0 -
Site 'filtered' by Google in early July.... and still filtered!
Hi, Our site got demoted by Google all of a sudden back in early July. You can view the site here: http://alturl.com/4pfrj and you may read the discussions I posted in Google's forums here: http://www.google.com/support/forum/p/Webmasters/thread?tid=6e8f9aab7e384d88&hl=en http://www.google.com/support/forum/p/Webmasters/thread?tid=276dc6687317641b&hl=en Those discussions chronicle what happened, and what we've done since. I don't want to make this a long post by retyping it all here, hence the links. However, we've made various changes (as detailed), such as getting rid of duplicate content (use of noindex on various pages etc), and ensuring there is no hidden text (we made an unintentional blunder there through use of a 3rd party control which used CSS hidden text to store certain data). We have also filed reconsideration requests with Google and been told that no manual penalty has been applied. So the problem is down to algorithmic filters which are being applied. So... my reason for posting here is simply to see if anyone here can help us discover if there is anything we have missed? I'd hope that we've addressed the main issues and that eventually our Google ranking will recover (ie. filter removed.... it isn't that we 'rank' poorly, but that a filter is bumping us down, to, for example, page 50).... but after three months it sure is taking a while! It appears that a 30 day penalty was originally applied, as our ranking recovered in early August. But a few days later it dived down again (so presumably Google analysed the site again, found a problem and applied another penalty/filter). I'd hope that might have been 30 or 60 days, but 60 days have now passed.... so perhaps we have a 90 day penalty now. OR.... perhaps there is no time frame this time, simply the need to 'fix' whatever is constantly triggering the filter (that said, I 'feel' like a time frame is there, especially given what happened after 30 days). Of course the other aspect that can always be worked on (and oft-mentioned) is the need for more and more original content. However, we've done a lot to increase this and think our Guide pages are pretty useful now. I've looked at many competitive sites which list in Google and they really don't offer anything more than we do..... so if that is the issue it sure is puzzling if we're filtered and they aren't. Anyway, I'm getting wordy now, so I'll pause. I'm just asking if anyone would like to have a quick look at the site and see what they can deduce? We have of course run it through SEOMoz's tools and made use of the suggestions. Our target pages generally rate as an A for SEO in the reports. Thanks!
Technical SEO | | Go2Holidays0