Crawler doesn't discover the links in the main nav
-
Hi Moz Community,
We have a headless ecom (Magento) client that I'm trying to crawl the site. During the crawl, the tool (Screaming Frog) cannot discover the sub-category URLs in the main navigation when I start crawling via homepage.
Similarly, when I start crawling with one of the sub-category page, it doesn't crawl any of the product URLs on the sub-category page itself.
When I inspect product and sub-cat URLs through Search Console, they seem as indexed and if I view how Googlebot rendered the sub-category page, I can see the product URLs on the sub-cat page too.
If you have any idea what's the issue with Screaming Frog and would like to help me out, I'd be so grateful!
Thanks in advance
-
Hi Kate,
Thank you! I followed you on Twitter, my user name is @curetuvana
-
Find me on twitter @katemorris and follow me. Tell me your name and I'll follow you and we can DM.
-
Hi Kate,
Thank you for your time to respond! Is there any way that I can contact you directly?
By the way, I've tried crawling after changing configurations as 'rendering Javascript' however, it still didn't discover product URLs.
Thank you!
-
Ah, I might know your problem. What is your site? We had this issue at my last company, had to do with crawling using JS. If you will send me the site, I can take a look.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google doesn't show proper meta for my subpage, how to fix it?
We have a subdomain blog.companyname.com. I am working on its English version blog.companyname.com/en but for some reason Google shows meta description from blog.companyname.com in search results which is not in Englsih language. How do I force google to show blog.companyname.com/en 's own meta?
Intermediate & Advanced SEO | | SofyaFr0 -
Link building
ok mozers i have a few questions. I am starting a new seo campaign and i want to target traffic for "how to make money on autopilot" Question 1. when it comes to link building i have seen some articles saying that i should not send all of my links to my landing page at once but to send links to my backlinks then index then using tiered link building. Is this a must or not? will i get penalized if i build 20 targeted links to my landing page in 1 day, lets say 20, pr7-9 domains? or should i tier it out and link maybe 5 pr9 domains to my landing page, then link 10 pr5 domains to each of those 5 pr9 domains and maybe link 20-pr1 domains to each of those tiered 2 pr5 domains? eq: Tier 1 = 5 PR-9 Tier 2 = 50 PR-5 Tier 3 = 1,000 PR-1 Question 2. Is their a certain amount of backlinks i need to use in order to out do my competitor? or does it just matter on the metrics of my backlinks? and when it comes to indexing these links do i need to index just the 5 pr9 links? or do i need to index all of them? or should i just index the landing page through google webmasters tools and hope it indexes all connecting pages? will doing any of these get my landing page indexed faster in order to rank faster? Question 3. Types of link building. Ok i am targeting guest blogs, wordpress sites, etc to put a link on. Should i focus on smm 'social media marketing' as well? or can i just focus on the traditional seo tactics first? Question 4. Keyword research. ok so my blog post is 'how to make money on autpilot' and from my keyword suggestion tools it picked up a list of keywords suggestions to target. Competition ranges from low to high, search volume ranges from 10 to 1900 visitors per month, after organizing the most relevant keywords to add to my campaign should i target each of the these keywords by creating a link building campaign for each one and target it to my landing page or use it as my 2nd or 3rd tier? those are the questions i really have for now. Here is my blog post http://www.vemomedia.com/how-to-make-money-on-autopilot/ Please feel me in on what i am needing to do in order to get some ranking and on how to run a link building campaign the correct way. Thanx in advance!
Intermediate & Advanced SEO | | djgbshows1 -
'?q=:new&sort=new' URL parameters help...
Hey guys, I have these types of URLs being crawled and picked up on by MOZ but they are not visible to my users. The URLs are all 'hidden' from users as they are basically category pages that have no stock, however MOZ is crawling them and I dont understand how they are getting picked up as 'duplicate content'. Anyone have any info on this? http://www.example.ch/de/example/marken/brand/make-up/c/Cat_Perso_Brand_3?q=:new&sort=new Even if I understood the technicality behind it then I could try and fix it if need be. Thanks Guys Kay
Intermediate & Advanced SEO | | eLab_London0 -
Establishing if links are 'nofollow'
Wonder if any of you guys can tell me if there is any other way to tell google links are nofollow other than in the html (ie can you tell google to nofollow every link in a subdomain or something). I'm trying to establish if a couple of links on a very high ranking site are passing me pagerank or not without asking them directly and looking silly! Within the source code for the page they are NOT tagged as nofollow at present. Hope that all makes sense 😉
Intermediate & Advanced SEO | | mat20150 -
Penguin 3.0 - Very minor drops across the board. Don't think its a penalty, any ideas?
Hey All, I just can't figure this out. My site has been ranking well for years, i've never done anything suspicious with it and since the penguin update, my rankings have dropped across the board but only by about 4 - 8 places each, some terms have went up from nowhere to page 8 etc. I don't think i've been hit with a penalty, so I don't know what the problem is or how to recover from it. Does anybody have any ideas on what could be wrong? Update: Perhaps some sites that were linking to mine have been hit with a penalty? Update 2: I just found myself somehow in some spammy link network for 600 sites that looked identical, I don't know how or why my website is in this! I have disavowed all of these links 5 days ago, no change to rankings. pY80Dzi
Intermediate & Advanced SEO | | Paul_Tovey0 -
Why isn't my uneven link flow among index pages causing uneven search traffic?
I'm working with a site that has millions of pages. The link flow through index pages is atrocious, such that for the letter A (for example) the index page A/1.html has a page authority of 25 and the next pages drop until A/70.html (the last index page listing pages that start with A) has a page authority of just 1. However, the pages linked to from the low page authority index pages (that is, the pages whose second letter is at the end of the alphabet) get just as much traffic as the pages linked to from A/1.html (the pages whose second letter is A or B). The site gets a lot of traffic and has a lot of pages, so this is not just a statistical biip. The evidence is overwhelming that the pages from the low authority index pages are getting just as much traffic as those getting traffic from the high authority index pages. Why is this? Should I "fix" the bad link flow problem if traffic patterns indicate there's no problem? Is this hurting me in some other way? Thanks
Intermediate & Advanced SEO | | GilReich0 -
Any idea why I can't add a Panoramio image link to my Google Places page?
Hey guys & gals! Last week, I watched one of the Pro Webinars on here related to Google Places. Since then, I have begun to help one of my friends with his GP page to get my feet wet. One of the tips from the webinar was to geotag images in Panoramio to use for your images on the Places page. However, when I try to do this, I just get an error that says they can't upload it at this time. I tried searching online for answers, but the G support pages that I have found where someone asks the same question, there is no resolution. Can anyone help? PS - I would prefer not to post publicly the business name, URL, etc. So, if that info is needed, I can PM. Thanks a lot!
Intermediate & Advanced SEO | | strong11 -
Aside from creative link bait, what's a solid link building strategy involve?
All things considered, directories, blogs, articles, press releases, forums, social profiles, student discount pages, etc, what do you consider to be a strong, phased, link building strategy? I'm talking beyond natural/organic link bait, since many larger accounts will not allow you to add content to their website or take 6 months to approve a content strategy. I've got my own list, but would love to hear what the community considers to be a strong, structured, timeline-based strategy for link building.
Intermediate & Advanced SEO | | stevewiideman1