Access denied in google webmaster tools
-
Hi I have just checked on my google webmaster tools and it is showing i 11 urls that are coming back as access denied. Now the urls are working, and they have been redirected using 301 redirect, so i have done everything right but for some reason google is not able to crawl them.
Does anyone know what i have done wrong for it to come back as access denied and how i can solve this problem. the site is www.in2town.co.uk many thanks
| | | |
| | 2 | Gardening/Gardening-Advice-What-is-Hydroponic-Gardening/menu-id-4991 | 403 | 4/11/13 |
| | 3 | Top-Showbiz-News/Super-Injunctions-Are-Right-Says-Hugh-Grant | 403 | 4/29/13 |
| | 4 | Entertainment-Tonight/Cheryl-Cole-wants-to-spice-up-The-X-Factor | 403 | 4/11/13 |
| | 5 | Tiger-Woods-paid-10000-a-time-for-sex | 403 | 4/20/13 |
| | 6 | Thousands-of-children-hurt-trying-to-stop-arguments-between-adults/Thousands-of-children-hurt-trying-to-stop-arguments-between-adults/menu-id-4448 | 403 | 4/24/13 |
| | 7 | News-Showbiz/Doctor-Who-changed-my-life-says-Matt-Smith | 403 | 4/29/13 |
| | 8 | The-Latest-Health-News/Hypnosis-Hypnotherapy-for-Relationships/menu-id-4744 | 403 | 4/11/13 |
| | 9 | Soap-Gossip-Latest-News/Emmerdale-Marks-bit-on-the-side-comes-to-home-farm/menu-id-4615 | 403 | 4/11/13 |
| | 10 | news/eastenders/ | 403 | 4/11/13 |
| | 11 | entertainment-news/Prince-William-Stag-Do-To-Be-Held-in-Cape-Town | 403 | 3/24/13 || | |
-
cheers for this, i have contacted the company to see what they say about this issue and hopefully it will be resolved.
-
As I said the clue is likely in the message you get from the front end "Forbidden Access (flooding)"
If you search for this, the results all seem to mention joomla and that module. If you look through those results there are some mentions of the security features of this SEF module and how to turn them on/off. It is impossible to say if this is 100% the cause of your issue, but if your hosting company say everything is fine, and the message shown is specific to this joomla module, then it is a likely candidate. All things being equal, try turning off this security feature and see if the access denied errors in GWT go away.
-
just going into my webmaster tools and it says the following
| | Response Code | Detected |
| --- | --- | --- |<colgroup><col style="width: 45px;"><col style="width: 80px;"><col><col style="width: 120px;"><col style="width: 90px;"></colgroup>
| | 1 | Headlines-Celebrity-/Simon-Cowell-The-Wedding-Is-Back-On | 403 | 4/20/13 |
| | 2 | Gardening/Gardening-Advice-What-is-Hydroponic-Gardening/menu-id-4991 | 403 | 4/11/13 |
| | 3 | Top-Showbiz-News/Super-Injunctions-Are-Right-Says-Hugh-Grant | 403 | 4/29/13 |
| | 4 | Entertainment-Tonight/Cheryl-Cole-wants-to-spice-up-The-X-Factor | 403 | 4/11/13 |
| | 5 | Tiger-Woods-paid-10000-a-time-for-sex | 403 | 4/20/13 |
| | 6 | Thousands-of-children-hurt-trying-to-stop-arguments-between-adults/Thousands-of-children-hurt-trying-to-stop-arguments-between-adults/menu-id-4448 | 403 | 4/24/13 |
| | 7 | News-Showbiz/Doctor-Who-changed-my-life-says-Matt-Smith | 403 | 4/29/13 |
| | 8 | The-Latest-Health-News/Hypnosis-Hypnotherapy-for-Relationships/menu-id-4744 | 403 | 4/11/13 |
| | 9 | news/eastenders/ | 403 | 4/11/13 |
| | 10 | entertainment-news/Prince-William-Stag-Do-To-Be-Held-in-Cape-Town | 403 | 3/24/13 ||
when i have looked at more info on this it says the following
Access denied errors
In general, Google discovers content by following links from one page to another. To crawl a page, Googlebot must be able to access it. If you’re seeing unexpected Access Denied errors, it may be for the following reasons:
- Googlebot couldn’t access a URL on your site because your site requires users to log in to view all or some of your content. (Tip: You can get around this by removing this requirement for user-agent Googlebot.)
- Your robots.txt file is blocking Google from accessing your whole site or individual URLs or directories.
- Test that your robots.txt is working as expected. The Test robots.txt tool lets you see exactly how Googlebot will interpret the contents of your robots.txt file. The Google user-agent is Googlebot. (How to verify that a user-agent really is Googlebot.)
- The Fetch as Google tool helps you understand exactly how your site appears to Googlebot. This can be very useful when troubleshooting problems with your site's content or discoverability in search results.
- Your server requires users to authenticate using a proxy, or your hosting provider may be blocking Google from accessing your site.
i have asked my hosting company about this and they say everything is fine.
any help would be great to solve this
| |
-
going to go through these today as they may have changed since the update. so you feel the sh404sef could be causing the blocking problems, i will contact them.
-
Hi Tim,
Well both those urls you give for the 301 are returning a 404, but I don't think they are the cause of your original problem which is the access denied issue. For that I am pretty sure you need to be looking at that joomla SH404SEF module.
-
hi both sides are there, for example from above,
Redirect 301 /Lingerie-Brands-expert-says-Lingerie-improves-your-sex-life /lingerie-helps-improve-your-sex-life
so i have the original page and then pointing to the destination page
i just do not understand after all the checks i have done why the error is happening
-
Hi Tim,
Your robots.txt looks ok from what I can tell. The 301s dont look odd (although what you have there is only one side of them right? I don't see the final page).
I think the clue is the message you get on the front end "Forbidden Access (flooding)". If you search for this phrase you start seeing references to the joomla SH404SEF module. See here for example: http://forum.joomla.org/viewtopic.php?p=1368937
I am not a joomla expert, but maybe it is a joomla issue instead of a server one, worth looking into.
-
they have also said my 301 redirects are causing the problems but i thought i had done this correctly
here are some of my redirects
The lines that are causing the issue are:
Redirect 301 /Jennifer-Aniston-upset-over-Brad-Pitt-Marriage /news/have-your-say/jennifer-aniston-upset-over-brad-pitt-marriage
Redirect 301 /In2town-Gossip/Liz-Hurley-Wants-Her-Husband-Back /news/have-your-say/liz-hurley-wants-her-husband-back
Redirect 301 /News-Celebrity/Take-That-and-Robbie-Williams-do-it-again /news/have-your-say/take-that-and-robbie-williams-do-it-again
Redirect 301 /Kevin-McCloud-does-not-like-the-word-Poverty /news/have-your-say/kevin-mccloud-does-not-like-the-word-poverty
Redirect 301 /Latest-Travel-News/Singapore-Tourist-Information-Singapore-a-must-for-Holidays/menu-id-4592 /news/holidays/singapore-tourist-information-singapore-a-must-for-holidays
Redirect 301 /Travel-Articles/Holiday-makers-are-rushing-to-buy-cheap-flights-to-Benidorm/menu-id-4998 /news/flight-news/holiday-makers-are-rushing-to-buy-cheap-flights-to-benidorm
Redirect 301 /The-Latest-Health-News/Stop-Biting-Your-Nails/menu-id-4744 /news/healthy-living/stop-biting-your-nails-with-hypnotherapy
Redirect 301 /Woman-celebrates-after-losing-weight-with-Weight-Loss-Hypnosis /news/gastric-band-hypnotherapy/woman-celebrates-after-losing-weight-with-gastric-band-hypnosis
Redirect 301 /Health-News-/-Stop-Smoking-Hypnosis-really-works-says-expert /news/health/stop-smoking-hypnosis-really-works-says-stop-smoking-expert
Redirect 301 /Animal-Health-News/Pet-Advice-Kennel-Cough-Advice-for-your-Pets/menu-id-4954 /news/dog-care/dog-kennel-cough
Redirect 301 /Travel-News/Flying-to-Australia-consumers-are-turning-their-back-on-Travel-Agents-over-cheap-flights/menu-id-4592 /news/flight-news/flying-to-australia-consumers-turn-their-backs-on-travel-agents-for-cheap-flights-to-australia
Redirect 301 /The-Latest-Health-News/Childbirth-Hypnotherapy/menu-id-4744 /news/health/childbirth-hypnotherapy
Redirect 301 /Travel-News/Brazil-Holidays-is-becoming-a-huge-hit-with-British-Tourist/menu-id-4592 /news/holidays/brazil-holidays-is-becoming-a-huge-hit-with-british-tourist
Redirect 301 /Celebrity-Gossip-Celebrity-News-and-latest-celebrity-gossip /Showbiz-Gossip
Redirect 301 /Latest-Travel-News/Travel-Magazine-reveals-secrets-to-finding-Cheap-Flights /news/flight-news/saving-money-on-flights
Redirect 301 /Lingerie-Brands-expert-says-Lingerie-improves-your-sex-life /lingerie-helps-improve-your-sex-life -
contacted my hosting company and they said the following robots file could be blocking google and causing the 403 errors, i thought it looked standard to me, can anyone please have a look and let me know if my robots file is causing the problem
many thanks
If the Joomla site is installed within a folder such as at
e.g. www.example.com/joomla/ the robots.txt file MUST be
moved to the site root at e.g. www.example.com/robots.txt
AND the joomla folder name MUST be prefixed to the disallowed
path, e.g. the Disallow rule for the /administrator/ folder
MUST be changed to read Disallow: /joomla/administrator/
For more information about the robots.txt standard, see:
http://www.robotstxt.org/orig.html
For syntax checking, see:
http://www.sxw.org.uk/computing/robots/check.html
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/ -
thank you for this, what would be the best way to solve this, as this must be affecting my rankings
-
Hi Tim,
It looks like you have some sort of server setup that is trying to block dos attacks or similar. If you put your site into screaming frog after the first few pages it starts returning 403 errors (access denied). If you then look at a page in a browser your see the message: Forbidden Access (flooding).
Is it likely that this is happening when google is trying to spider the site also?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why can't google mobile friendly test access my website?
getting the following error when trying to use google mobile friendly tool: "page cannot be reached. This could be because the page is unavailable or blocked by robots.txt" I don't have anything blocked by robots.txt or robots tag. i also manage to render my pages on google search console's fetch and render....so what can be the reason that the tool can't access my website? Also...the mobile usability report on the search console works but reports very little, and the google speed test also doesnt work... Any ideas to what is the reason and how to fix this? LEARN MOREDetailsUser agentGooglebot smartphone
Technical SEO | | Nadav_W0 -
Deindexed homepage by Google
I just noticed that my homepage was de-indexed by google. Any thoughts would be appreciated.
Technical SEO | | Jenny_H0 -
Google Webmaster Tools is saying "Sitemap contains urls which are blocked by robots.txt" after Https move...
Hi Everyone, I really don't see anything wrong with our robots.txt file after our https move that just happened, but Google says all URLs are blocked. The only change I know we need to make is changing the sitemap url to https. Anything you all see wrong with this robots.txt file? robots.txt This file is to prevent the crawling and indexing of certain parts of your site by web crawlers and spiders run by sites like Yahoo! and Google. By telling these "robots" where not to go on your site, you save bandwidth and server resources. This file will be ignored unless it is at the root of your host: Used: http://example.com/robots.txt Ignored: http://example.com/site/robots.txt For more information about the robots.txt standard, see: http://www.robotstxt.org/wc/robots.html For syntax checking, see: http://www.sxw.org.uk/computing/robots/check.html Website Sitemap Sitemap: http://www.bestpricenutrition.com/sitemap.xml Crawlers Setup User-agent: * Allowable Index Allow: /*?p=
Technical SEO | | vetofunk
Allow: /index.php/blog/
Allow: /catalog/seo_sitemap/category/ Directories Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /includes/
Disallow: /lib/
Disallow: /magento/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /stats/
Disallow: /var/ Paths (clean URLs) Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /aitmanufacturers/index/view/
Disallow: /blog/tag/
Disallow: /advancedreviews/abuse/reportajax/
Disallow: /advancedreviews/ajaxproduct/
Disallow: /advancedreviews/proscons/checkbyproscons/
Disallow: /catalog/product/gallery/
Disallow: /productquestions/index/ajaxform/ Files Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt Paths (no clean URLs) Disallow: /.php$
Disallow: /?SID=
disallow: /?cat=
disallow: /?price=
disallow: /?flavor=
disallow: /?dir=
disallow: /?mode=
disallow: /?list=
disallow: /?limit=5
disallow: /?limit=10
disallow: /?limit=15
disallow: /?limit=20
disallow: /*?limit=250 -
Webmaster tools
Hello, My sites are showing odd "links to your site" data in WMT. Its not showing any links to the homepages and reduced links for other pages. Anyone else seeing this? Penguin refresh maybe?
Technical SEO | | jwdl0 -
Google Sitelinks
Is there anyway to control the sitelinks under a listing in Google? I have a group of lawyers where 1 of the them is showing up in the sitelinks. They want all of the lawyers to show up. Right now it is showing 1 lawyer, about page, contact us page, etc. Thanks!!!!
Technical SEO | | SixTwoInteractive0 -
Recent Webmaster Tools Glitch Impacting Site Quality?
The ramifications of this would not be specific to myself but to anyone with this type of content on their pages... Maybe someone can chime in here, but I'm not sure how much if at all site errors (for example 404 errors) as reported by Google Webmaster Tools are seen as a factor in site quality, which would impact SEO rankings. Any insight on that alone would be appreciated. I've noticed some fairly new weird stuff going on in the WMT 404 error reports. It seems as though their engine is finding objects within the source code of the page that are NOT links but look a URL, then trying to crawl them and reporting them as broken. I've seen a couple different of cases in my environment that seem to trigger this issue. The easiest one to explain are Google Analytic virtual pageview Javascript calls where for example you might send a virtual pageview back to GA for clicks on outbound links. So in the source code of your page you would have something like: onclick="<a class="attribute-value">_gaq.push(['_trackPageview', '/outboundclick/www.othersite.com']);</a> Although this is obviously not a crawl-able link, sure enough Webmaster Tools now would be reporting the following broken page with a 404: www.mysite.com/outboundclick/www.otherwite.com I've seen other such cases of thing that look like URLs but not actual links being pulled out of the page source and reported as broken links. Has anyone else noticed this? Do 404 instances (in this case false ones) reported by Webmaster Tools impact site quality rankings and SEO? Interesting issue here, I'm looking forward to hear some people's thoughts on this. Chris
Technical SEO | | cbubinas0 -
Will google let me do this
Hi i am working on my site at the moment www.in2town.co.uk and i am adding new sections and was thinking about buying domain names that best describe that section and which people would remember. so for example i am looking at adding a tenerife magazine to my site and would like to know if it would be wise to buy a domain name for example tenerife magazine and then have it directed to the section of my site. would this benefit my site in any way and would google allow this. instead of having in2town.co.uk and then tenerife magazine after it, sorry cannot find the slash as i am on a spanish keyborad at the moment, i would like to have something like tenerifemagazine.co..uk etc If anyone can give me advice on this then that would be great. also can anyone let me know if this is a wise idea or not, to have sub domain names on my main site. i would like to know if i had tenerifemagazine under the in2town domain name would it slow the site down or should i consider building a brand new site just for that and then making people aware that it comes under the in2town umbrella many thanks
Technical SEO | | ClaireH-1848861