Angular.js + Crawlers
-
I am working with a site that recently deployed Angular.js on the site. From an SEO standpoint its a little more tricky than we thought. We have deployed a couple updates to render pages for the bots but we not seeing changes in Moz weekly reports.
When it comes to Angular.js, will the Moz bots read/access the site the same as the other major engines? I'm trying to figure out if our deployments are working or if there's something off in the Moz reports.
Thanks.
-
I am using prerender to cache/render static pages to crawl agents but MOZ is not able to crawl through my website (http://www.exambazaar.com/). Hence it has a domain authority of 1/100. I have been in touch with Prerender support to find a fix for the same and have also added dotbot to the list of crawler agents in addition to Prerender default list which includes rogerbot. Do you have any suggestions to fix this?
List: https://github.com/prerender/prerender-node/commit/5e9044e3f5c7a3bad536d86d26666c0d868bdfff
Adding dotbot:
prerender.crawlerUserAgents.push('dotbot'); -
Within prerender you are able to determine which user agents will receive the HTML snapshot. It is here that you can add rogerbot. This is allowing Moz to crawl the site as if they were Google and receive the HTML snapshot version.
Additionally, you can always use the fetch as bot function within Webmaster Tools, to see exactly what is being presented/indexed.
-
With the current direction of web development this is something that needs to be addressed. Google has already confirmed that they are in fact crawling Javascript based sites.
Reference:
http://ng-learn.org/2014/05/SEO-Google-crawl-JavaScript/
https://support.google.com/webmasters/answer/174992?hl=enThe solution in this case is an HTML snapshot which, you could roll your own, but there are services like https://prerender.io/ that can do it for you.
This doesn't quite help the case for Moz Bot, maybe the HTML snapshots do work here - I haven't tested it yet. Either way, Javascript is becoming more and more a dominant language to code up websites. I hope Moz recognizes this because this toolset is awesome and I'd love to continue using it.
-
Is there still no update to this by MOZ?
A number of sites I work on are using Angularjs pushstate. Is there a way to point moz bot to the escaped fragment static pages?
-
Static rendering is not cloaking. It's a very common practice that Google actually recommends. The issue with angular js is that everything is code based. If you were to look at the code all the pages would look the same. In fact, MozBot sees this as every page is duplicate content.
https://developers.google.com/webmasters/ajax-crawling/docs/html-snapshot
It would be nice to see the MozBot act more like Google-bot.
-
What do you mean by "We have deployed a couple updates to render pages for the bots" that sounds like clocking?
-
Hello, Josh
Currently our crawlers do not process any kind of javascript found on pages (including pages created with angular.js.) I don't if the major search engines have this restriction or not.
For moz's crawlers, this means that links created through AJAX or other javascript will not be picked up. Links appearing in static content, including those within
<noscript>tags, should be noticed and indexed. Be aware that even if you've already made changes exposing links in the page's static content, it can take up to a week for the campaign crawl to catch up.</p> <p>Hopefully that answered your questions! Let us know if you have any more.</p></noscript>
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Site Crawler not working but on-demand crawler working
Hi, In Moz pro, when using Site crawler (or recrawl), we are seeing message site is banned. But when using on-demand crawler, it could generate report successfully. I just like to know if in both these cases, it is roberbot that is used! And kindly note, site crawler was perfectly working before. So the required setup is already in place from long time. Site crawler ban issue started appearing from nov/dec 2023. . Could you please us understand how could we possibly make site-crawler work?
Moz Pro | | gilesd
I am happy to provide more details if you need any. Thanks0 -
Unsolved Ooops. Our crawlers are unable to access that URL
hello
Moz Pro | | ssblawton2533
i have enter my site faroush.com but i got an error
Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct
what is problem ?0 -
How do i get the crawler going again?
The initial crawl only hit one page. Set up another campaign for another site and it crawled 260 pages. How can I get the crawler started up again or do I really have to wait a week ?
Moz Pro | | martJ0 -
How to set the crawler or reports to ignore
I have a mobile version of a site with a URL string that disables the mobile view on smartphones (view full site), string is like this example.com/page-name.html?mobile=off I need the seomoz pro reports or crawler to ignore it because the crawler visits both versions of the site then reports them as duplicate content. Is there a setting page I haven't visited yet that will set this?
Moz Pro | | Str82u0 -
Reset Crawler
Hello, Does anyone know how to reset the crawler? We recently uploaded our new website and deleted the current campaign but it seems the crawler is caching our old websites data and not the new so every time we try to create a new campaign with the same details, it's just pulling everything from cache it seems. Thanks
Moz Pro | | ForzaHost0 -
SEOMOZ Crawler unicode bug
for the last couple of weeks the SEOMOZ crawls my homepage only and gets 4xx error for most of the URL's. the crawler have no issues with English url's only with the unicode(Hebrew) ones. this is what is see in the csv export for the crawl (one sample) : http://www.funstuff.co.il/׳ž׳¡׳™׳‘׳×-׳¨׳•׳•׳§׳•׳× 404 text/html; charset=utf-8 you can see that the URL is Gibberish please help.
Moz Pro | | AsafY0 -
Why is the crawler following form action links?
I have an issue with one of my sites where the SEOMoz crawler is following some form action links. It is my understanding that the crawler will ignore these links. Why would it not be ignoring them in certain cases. If you need more detail, please ask. Thanks.
Moz Pro | | AmberHanson0 -
SEOMoz site crawlers created an issue for our servers
I have set up a number of campaigns with your pro tool. Unfortunately we have 7 sites on our server and our IT dept have said that we had an issue when your site crawlers visited for several sites at the same time - is there any way that I can retain the campaigns but have the sites crawled on request rather than automatically?
Moz Pro | | StephenALee0