Moz not crawling opencart product pages
-
hi,
i have waited for over 2 weeks now and the crawler only got 8 pages, and is not getting all the open cart pages and products.
any idea of what can be wrong?
im using joomla 2.5.11 and mijoshop 2.0.5 (which uses opencart 1.5.5.1).
thanks
-
Hi Marlon,
There are a lot of different issues that may cause Rogerbot to be unable to crawl your site. For troubleshooting tips, I suggest reading http://moz.com/help/pro/why-can-t-rogerbot-crawl-my-site. Hope that helps!
-
ok thanks
-
Screaming Frog has an option in the menu -
Configuration > Spider > "Ignore Robots.txt"
Tick this box and re-scan and see if you get different results.
And also, may be worth checking this out:
http://moz.com/learn/seo/robotstxtGood luck!
-
hi
thanks for the answer, tried screaming frog now as well and seems it has the same problem,
is it possible to allow only certain spiders/robots to crawl all files, with the robots.txt?
thanks
-
It's hard to say without knowing the domain name -
Have you tried Screaming Frog software to see if it also has issues crawling?
Double check your robots.txt to see if some directories and pages are blocked.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Crawl 1-page 301 status error but httpstatus.io says its 403
I am trying to run a site crawl for my website and MOZ is only resulting in 1 page crawled with the home page URL Status Code of 301. However when I run it in httpstatus.io it is giving me a 403 status error. Im curious as to why MOZ is saying its a 301 and httpstatus.io is saying 403. Is there anything I can do in MOZ first to get the site crawled before asking my developers to look into the 403 error?
Moz Bar | | JohnConover0 -
Strange insight "tracked keywords" from Moz
Hi there, I was wondering if somebody is seeying the same problem. Since a couple of days i got a new keyword insights in my mail and i've seen some strange things for two accounts. The branded keywords are dropped lower then position 51. The thing is that i got an insight, i checked it in my private mode and the branded keywords are still there. Other keywords are still ranking, and there is no effect on my traffic. There is no connection between the sites. One has a couple of links and the other a lot. One has a very good page with nice optimalisation and the other has no text. So it couldn't be over-optimalisation and incoming links...
Moz Bar | | NielsVos
Does anybody have the same problem? Greetings,
Niels0 -
On-Page Grader Url is inaccessible
Hi everybody. I'm trying to use on -page grader for https://www.upscaledinnerclub.com and get "Sorry, but that URL is inaccessible." Robots.txt are empty, another thread on MOZ was talking about DNS check - it's all good. So, I can't figure out why this is happening. Also I am trying the same for another website https://www.regexseo.com - the same story. Common thing is that they both are on Google App Engine. And at first i thought that was the problem. Bu then i checked this one : https://www.logitinc.com/ and it's working, even though this website is on GAE as well. None of these website have robots.txt or any differences in setup or settings. Any thoughts?
Moz Bar | | DmitriiK0 -
Crawl Test Takes Long Time
Hi Moz, I have submitted our website for a crawl test. Usually it would only take a few hours to do the crawl. However this time, it takes quite long time and the result still shows in progress 😞 This is a small website which only contains less than 10 pages. Just wondering if this is our website setting issue or it is a technical issue at your end? Many thanks in advance. sFjAERG.png
Moz Bar | | russellbrown0 -
Where to find one off crawl report
Hello, I don't know if I am being a bit daft but I don't seem to be able to find the area where I can request a one off crawl report anymore (rather than setting up a campaign). Can someone let me know where this is now? Thanks!
Moz Bar | | RikkiD220 -
Probs with campaign overview page ??
Hi As of today my overview pages are devoid of formatting and just text and basic display, anyone else getting this ? Cheers Dan
Moz Bar | | Dan-Lawrence1 -
On page grades - quick question
My hardwood flooring client just added carpet, luxury vinyl, ceramic tile, area rugs, design, and ceramic tile to his lineup - joy, joy. He really thought the months and months of working together and creating a bunch of solid, original content and promoting it and building links and seeing great results for Hardwood Flooring means he would instantly rank well for ALL flooring. I digress, sorry. I am trying to understand the on page reports a little better. When I grade the new carpet page on his site for the keyword 'carpet' it comes back with an A - when I grade it for 'Carpet, city name' it comes back with an F. Which should I be targeting for best results? 95% of my clients are concerned about local results and not getting business other than within 20-30 miles of their office. Any guidance in understanding on how BEST to utilize the on page reporting would be greatly appreciated! (I did watch the video and check out the guide) Thank you in advance for your time! Matthew
Moz Bar | | Mrupp440 -
Where is the Crawl Tool we had before Research?
Hello, I can't seem to find the crawl tool for other domains that aren't in our campaigns. We've lost it 😞 Thanks, Romeo.
Moz Bar | | RomeoMadrid1