Summarize your question.Sitemap blocking or not blocking that is the question?
-
Hi from wet & overcast wetherby UK
Ones question is this...
" Is the sitemap plus boxes blocking bots ie they cant pass on this page http://www.langleys.com/Site-Map.aspx "
Its just the + boxes that concern me, i remeber reading somewherte javascript nav can be toxic.
Is there a way to test javascript nav set ups and see if they block bots or not?
Thanks in advance
-
I use Screaming Frog SEO Spider (free version) to check the internal link structure of a website. If a page is blocking ALL spiders it will pick it up.
Another thing I would say would be to check in Google Webmaster Tools to see if there are any crawl errors.
And the last thing I would add is to make sure that you have a non-JavaScript way to find all the pages on your website - through strong internal linking or a manual sitemap page that isn't generated through JS.
Hope this helps
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blocking Google from telemetry requests
At Magnet.me we track the items people are viewing in order to optimize our recommendations. As such we fire POST requests back to our backends every few seconds when enough user initiated actions have happened (think about scrolling for example). In order to eliminate bots from distorting statistics we ignore their values serverside. Based on some internal logging, we see that Googlebot is also performing these POST requests in its javascript crawling. In a 7 day period, that amounts to around 800k POST requests. As we are ignoring that data anyhow, and it is quite a number, we considered reducing this for bots. Though, we had several questions about this:
Technical SEO | | rogier_slag
1. Do these requests count towards crawl budgets?
2. If they do, and we'd want to prevent this from happening: what would be the preferred option? Either preventing the request in the frontend code, or blocking the request using a robots.txt line? The latter question is given by the fact that a in-app block for the request could lead to different behaviour for users and bots, and may be Google could penalize that as cloaking. The latter is slightly less convenient from a development perspective, as all logic is spread throughout the application. I'm aware one should not cloak, or makes pages appear differently to search engine crawlers. However these requests do not change anything in the pages behaviour, and purely send some anonymous data so we can improve future recommendations.0 -
Sitemaps:
Hello, doing an audit found in our sitemaps the tag which at the time was to say that the url was mobile. In our case the URL is the same for desktop and mobile.
Technical SEO | | romaro
Do you recommend leaving or removing it?
Thank you!0 -
Generation 'child' sitemaps?
First off, am I correct in thinking that a 'child' sitemap is a sitemap of a subfolder and everything that sits under it, i.e. www.example.com/example If so, can someone give me a good recommendation for generation a free child sitemap please? Many thanks, Rhys
Technical SEO | | SwanseaMedicine0 -
Header Tag Question
While reviewing code on a site, I found the following: <h1 class="<a class="attribute-value">logo</a>"> <a id="<a class="attribute-value">logo</a>" href="[http://siteexampleh1.com](view-source:http://dmbinc.com/)"><span>Example of most important content on this page- Companyspan>a> h1> Is this the correct way to place code for an h1 tag? The content is cached within the page and is hidden to the viewer. The content that is assigned as the h1, is a logo. Majority of code I have been reviewing does not use this setup. The code would instead read as ( This is heading 1 ). Can anyone provide insights on this? Thanks!
Technical SEO | | jfeitlinger0 -
Blogger Blog URL Structure Questions
I'm starting to use my blog more and wanted to ask about an issue I've read about on SEOmoz in the past. I use blogger instead of wordpress. It's quick and simple - I have no interest in switching to wordpress for this particular blog. My blog is currently setup as blog.site.com. Is it still important (for seo reasons) to switch from blog.site.com to site.com/blog? If so, is there a way to do this in blogger? And if I do this, will my past posts lose their authority if their redirected to the new url structure? Rand mentions in this article: http://www.seomoz.org/blog/11-best-practices-for-urls "never use multiple subdomains" - This is an old article, but I've seen this mentioned several times. Does this still hold true? Am I losing out on links to my blog? Thanks in advance.
Technical SEO | | ChaseH0 -
Robots.txt versus sitemap
Hi everyone, Lets say we have a robots.txt that disallows specific folders on our website, but a sitemap submitted in Google Webmaster Tools that lists content in those folders. Who wins? Will the sitemap content get indexed even if it's blocked by robots.txt? I know content that is blocked by robot.txt can still get indexed and display a URL if Google discovers it via a link so I'm wondering if that would happen in this scenario too. Thanks!
Technical SEO | | anthematic0 -
Best XML Sitemap generator
Do you guys have any suggestions on a good XML Sitemaps generator? hopefully free, but if it's good i'd consider paying I am using a MAC so would prefer a online or mac version
Technical SEO | | kevin48030 -
Ror.xml vs sitemap.xml
Hey Mozzers, So I've been reading somethings lately and some are saying that the top search engines do not use ror.xml sitemap but focus just on the sitemap.xml. Is that true? Do you use ror? if so, for what purpose, products, "special articles", other uses? Can sitemap be sufficient for all of those? Thank you, Vadim
Technical SEO | | vijayvasu0