Website Indexing Issues - Search Bots will only crawl Homepage of Website, Help!
-
Hello Moz World,
I am stuck on a problem, and wanted to get some insight. When I attempt to use Screaming Spider or SEO Powersuite, the software is only crawling the homepage of my website. I have 17 pages associated with the main domain i.e. example.com/home, example.com/sevices, etc. I've done a bit of investigating, and I have found that my client's website does not have Robot.txt file or a site map. However, under Google Search Console, all of my client's website pages have been indexed.
My questions, Why is my software not crawling all of the pages associated with the website? If I integrate a Robot.txt file & sitemap will that resolve the issue?
Thanks ahead of time for all of the great responses.
B/R
Will H.
-
Hi Will,
It'd be impossible to find a solution to the problem without having the domain. If you want to message me the URL, I can take a quick look for ya.
A lack of either of those files shouldn't create any crawl issues. Setting up those files won't fix your problem, but you should add them both anyway.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Complicated Title Tag Issues. Experts, Please Help!
Hey there Moz community! This is the first time I ask a question here so please forgive me if I miss any forum etiquette. I am managing SEO for an educational site which is built in React Js, and so far much of the job has been keyword research and site optimization. The site still has slow PageSpeed though. The Issues - 4 weeks ago we published 20 or so content pieces, I had pre-prepared title tags and meta descriptions. But when we released the content there was a programming error that made all of the pages show another title tag for all 20 pages instead of the pre-prepared individual title tags. I noticed this after 3 days and the issue was fixed within 6 days, but by then Google had crawled and indexed the pages. And now I can't get Google to change to the pre-prepared tags no matter what I do! I've tried changing the content, changing the URL of one of the pages, and I've sent Google spiders to re-crawl the pages multiple times. The super weird thing is that the correct title tag shows in the 'navigation bar/tabs bar' on google chrome: But NOT when I view the source code for the page: Yesterday I was taking a walk in the park and I just couldn't stop thinking about it (it is really starting to get to me by now since nothing works), so I ran back home and looked closely at one of these pages in the Google search console. And I noticed something I hadn't seen before… BOTH of the title tags can be found in the HTML: Pre-prepared title tag: <title></strong>UK Seat Belt & Car Seat Laws: The Definitive Guide<strong></title> The other title tag (in src section): title=Ace%20The%20DMV%20Permit%20Test%20%26%20Get%20Your%20License Could this be the problem or what do you think? I've understood that Google has automated title tags and that they can choose their own if they think it fits better, but the title tags aren't even close to describing the topic as it is now so it doesn’t make any sense. All answers are greatly appreciated! Your advice is life-saving for a learner like me. P.S. I love SEO but it can be very frustrating sometimes! Thank you very much, Leo
Intermediate & Advanced SEO | | Leowa0 -
Removing indexed internal search pages from Google when it's driving lots of traffic?
Hi I'm working on an E-Commerce site and the internal Search results page is our 3rd most popular landing page. I've also seen Google has often used this page as a "Google-selected canonical" on Search Console on a few pages, and it has thousands of these Search pages indexed. Hoping you can help with the below: To remove these results, is it as simple as adding "noindex/follow" to Search pages? Should I do it incrementally? There are parameters (brand, colour, size, etc.) in the indexed results and maybe I should block each one of them over time. Will there be an initial negative impact on results I should warn others about? Thanks!
Intermediate & Advanced SEO | | Frankie-BTDublin0 -
How long after https migration that google shows in search console new sitemap being indexed?
We migrated 4 days ago to https and followed best practices..
Intermediate & Advanced SEO | | lcourse
In search console now still 80% of our sitemaps appear as "pending" and among those sitemaps that were processed only less than 1% of submitted pages appear as indexed? Is this normal ?
How long does it take for google to index pages from sitemap?
Before https migration nearly all our pages were indexed and I see in the crawler stats that google has crawled a number of pages each day after migration that corresponds to number of submitted pages in sitemap. Sitemap and crawler stats show no errors.0 -
Looking for help with my website
Hi does any one know of a good seo company that will get results, i.e., fix site issues and get the site improving in the serps.
Intermediate & Advanced SEO | | Taiger0 -
Help my site it's not being indexed
Hello... We have a client, that had arround 17K visits a month... Last september he hired a company to do a redesign of his website....They needed to create a copy of the site on a different subdomain on another root domain... so I told them to block that content in order to not affect my production site, cause it was going to be an exact replica of the content but different design.... The developmet team did it wrong and blocked the production site (using robots.txt), so my site lost all it's organica traffic, which was 85-90% of the total traffic and now only get a couple of hundreds visits a month... First I thought we had been somehow penalized, however when I the other site recieving new traffic and being indexed i realized so I switched the robots.txt and created 301 redirect from the subdomain to the production site. After resending sitemaps, links to google+ and many things I can't get google to reindex my site.... when i do a site:domain.com search in google I only get 3 results. Its been now almost 2 month and honestly dont know what to do.... Any help would be greatly appreciated Thanks Dan
Intermediate & Advanced SEO | | daniel.alvarez0 -
Domain Issue
Starting a new local travel guide site. Would like to buy a domain and have found one with decent Domain Authority and Trust, but they want $2500 for the domain which I feel is a bit steep since I will be not using any of the content and it is generating hardly any revenue now. . I would rather not start from scratch with no links and no trust. I have a few questions.... -Any suggestions on sites to look for domains or strategy for finding and offering to buy? Any guidelines on how to value domains? If I but it and change registration do I risk losing all the value? Cold I just change technical contact info? Any other suggestions are welcome. Thanks.
Intermediate & Advanced SEO | | Reportcard0 -
Will links to a subdomain help it rank?
I have an affiliate subdomain on a larger company's domain. (For example I have: www.victor.company.com on www.company.com). Would working to attain backlinks to the subdomain help it rank or will I just be putting forth my effort and helping the domain rank?
Intermediate & Advanced SEO | | VictorVC0 -
How to enable crawling for dynamic generated search result pages?
I want to enable crawling facility for dynamic generated search result pages which are generating by Magento Solr search. You can view more about it by following URLs. http://code.google.com/p/magento-solr/ http://www.vistastores.com/catalogsearch/result/?q=bamboo+table+lamp
Intermediate & Advanced SEO | | CommercePundit
http://www.vistastores.com/catalogsearch/result/?q=ceramic+table+lamp
http://www.vistastores.com/catalogsearch/result/?q=green+patio+umbrella Right now, Google is not crawling search result page because, I have added following syntax to Robots.txt file. Disallow: /*?q= So, How do I enable crawling of search result pages with best SEO practice? If any other inputs in same direction so, it will help me more to get it done.0