Massive increase in Moz crawl.
-
I have a subdomain which has just started to be crawled by Moz, Previously this wasn't the case.
The sub-domain had 16,000+ issues. Why has Moz started to count sub-domains as part of the main domain, has Google started to do this aswell?
-
Hi Dave!
Thanks for reaching out! My name is Anne, and I helping the Mozzer Alliance today! There could be a few different reasons we've just started crawling this subdomain:
1. Did you recently link this subdomain to your site differently? Our crawler starts with your home page and uses recursive crawling to find pages on your site through the links we find. This could explain why we just started crawling the subdomain.
2. Did you change your robots.txt for this subdomain? Our crawler, Rogerbot, follows robots.txt. If you edited this site recently, this could also explain why we just started crawling your subdomain.
I hope this helps! If you have more questions, please reach out to the help team at https://mza.seotoolninja.com/help/contact. From there we can help you troubleshoot what's going on, but keep your campaign details off the interwebz.
Have a great weekend!
Anne
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
The links pointed to a multilanguage site, should increase the DA? (Wordpress question inside)
We are planning to make our site available to several language, using the plugin WPML in Wordpress. The site should look with /es/, /fr, etc. If someone point to an URL in the spanish version, the english version get any benefit from it? (better search ranking or something like that). Some side question: WPML works fine with SEO and Moz?
On-Page Optimization | | carlostinca0 -
Big problem with my new crawl report
I am owner of small opencart online store. I installed http://www.opencart.com/index.php?route=extension/extension/info&extension_id=6182&filter_search=seo. Today my new crawl report is awful. The number of errors is up by 520 (30 before), up with 1000 (120 before), notices up with 8000 (1000 before). I noticed that the problem is with search. There is a lot duplicate content in search only. What to do ?
On-Page Optimization | | ankali0 -
Why so many crawl errors?
Our site is showing it has a ton of crawl errors in the back end, mostly concerning duplicate content within our blog. The content is unique however. We know this for certain because it's done in house or put together by some of the freelance writers we work with. The site is for an RV dealership and we're using a template-based system from a well known company. Any ideas on what may be causing this?
On-Page Optimization | | BlakeArbogast0 -
Crawl Diagnostics - Duplicates and canonical problem
SEOmoz crowl diagnostic reports duplicates (title, content) issue on this addres: http://www.meblobranie.pl/biurowe/fotele-biurowe/promocje page already has canonical tag - is this a bug of crowler, or smth wrong on page?
On-Page Optimization | | SITS0 -
Hi Moz peeps, google said that we should have two site maps...
Hi Moz peeps, google said that we should have two site maps... one for google and one for people. Now i know having a site map to submit to google for the first time is important for SEO, but is having a site map for people visitng the site help at all in terms of google's bots crawling your site? I know it might actually help human people navigate through your site, I just want to know if by not having it or having it affects on page SEO at all, thanks guys
On-Page Optimization | | david3051 -
How long after a URL starts showing a 404 does Google stop crawling?
Before hiring me to do SEO, a client re-launched their site and did not 301 the old URLs to the new. Only the home page URL stayed the same. For a month after the re-launch, the old URLs returned a 404. For the next month, all 404 pages (basically any non-existent URL) were 301'd to the home page. Finally, 2 months after launching, they properly 301'd the old URLs to the new. Now, the new URLs are not ranking well. I assume it's too late to realize any benefit from the 301's, just checking to see if anybody has any insight into how long Google keeps trying to crawl old/404/improperly 301'd URLs. Thanks!
On-Page Optimization | | AndrewMiller0 -
How To Prevent Crawling Shopping Carts, Wishlists, Login Pages
What's the best way to prevent engines from crawling your websites shopping cart, wishlist, log in pags, ect... Obviously have it in robots.txt but is their any other form of action that should be done?
On-Page Optimization | | Romancing0 -
Pages crawled
I noticed there is a limited in the number of pages crawled on galena.org? Will this number increase over time?
On-Page Optimization | | nskislak240