Implications of Disallowing A LOT of Pages
-
Hey everyone,
I just started working on a website and there are A LOT of pages that should not be crawled - probably in the thousands. Are there any SEO risks of disallowing them all at once, or should I go through systematically and take a few dozen down at a time?
-
Perfect, that's my intent. Thanks so much for your help!! I really appreciate it.
-
Correct.
Also, (i´ve should have said this earlier) there are no effective way to avoid robots crawling the web. The thing here (and what im answering) is to not show (and/or remove if are already in the index) certain pages in google search result.
-
That would happen if I used noindex, correct?
-
Exactly.
Also, remember that disalowing by robots.txt will not remove them from Google´s index.
-
LOL now I'm worried about what you mean by "correctly." Do you mean just making sure that I'm not disallowing valuable pages that should be crawled?
-
Hello Rachel,
If you do it correctly, this is not disallowing useful pages, there is no risk. Go ahead.
Best luck.
GR.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Roger returned single page
A search query in Google shows 156 indexed results. Yet Roger has completed a crawl and returned a single page. The website is table based, but there appear to be no redirects or Javascript blocking bots so I'm unsure why Roger has under delivered. The problem is not unique to Roger. I ran the site through Screaming Frog's desktop crawler and it also returned a single page. I'm wondering if there is something in the site code I don't know to look for that is stopping Roger & Screaming Frog from crawling the site. Appreciate any insights you can offer. PS. I've read https://seomoz.zendesk.com/entries/409821-Why-Isn-t-My-Site-Being-Crawled-You-re-Not-Crawling-All-My-Pages- and don't think these suggested causes apply.
Technical SEO | | NicDale0 -
Page Content
Our site is a home to home moving listing portal. Consumers who wants to move his home fills a form so that moving companies can cote prices. We were generating listing page URL’s by using the title submitted by customer. Unfortunately we have understood by now that many customers have entered exactly same title for their listings which has caused us having hundreds of similar page title. We have corrected all the pages which had similar meta tag and duplicate page title tags. We have also inserted controls to our software to prevent generating duplicate page title tags or meta tags. But also the page content quality not very good because page content added by customer.(example: http://www.enakliyat.com.tr/detaylar/evden-eve--6001) What should I do. Please help me.
Technical SEO | | iskq0 -
Container Page/Content Page Duplicate Content
My client has a container page on their website, they are using SiteFinity, so it is called a "group page", in which individual pages appear and can be scrolled through. When link are followed, they first lead to the group page URL, in which the first content page is shown. However, when navigating through the content pages, the URL changes. When navigating BACK to the first content page, the URL is that for the content page, but it appears to indexers as a duplicate of the group page, that is, the URL that appeared when first linking to the group page. The client updates this on the regular, so I need to find a solution that will allow them to add more pages, the new one always becoming the top page, without requiring extra coding. For instance, I had considered integrating REL=NEXT and REL=PREV, but they aren't going to keep that up to date.
Technical SEO | | SpokeHQ1 -
Missing page titles
Does anyone know why my SeoMoz crawl reads my page titles differen't to what they truly are on my active site? I changed my pages titles and optimised them several months ago. Is my old page titles still been crawled rather than the new ones how do i fix this is?
Technical SEO | | gimes0 -
What to do when you want the category page and landing page to be the same thing?
I'm working on structuring some of my content better and I have a dilemma. I'm using wordpress and I have a main category called "Therapy." Under therapy I want to have a few sub categories such as "physical therapy" "speech therapy" "occupational therapy" to separate the content. The url would end up being mysite/speech-therapy. However, those are also phrases I want to create a landing page for. So I'd like to have a page like mysite.com/speech-therapy that I could optimize and help people looking for those terms find some of the most helpful content on our site for those certain words. I know I can't have 2 urls that are the same, but I'm hoping someone can give me some feedback on the best way to about this. Thanks.
Technical SEO | | NoahsDad0 -
Getting More Pages Indexed
We have a large E-commerce site (magento based) and have submitted sitemap files for several million pages within Webmaster tools. The number of indexed pages seems to fluctuate, but currently there is less than 300,000 pages indexed out of 4 million submitted. How can we get the number of indexed pages to be higher? Changing the settings on the crawl rate and resubmitting site maps doesn't seem to have an effect on the number of pages indexed. Am I correct in assuming that most individual product pages just don't carry enough link juice to be considered important enough yet by Google to be indexed? Let me know if there are any suggestions or tips for getting more pages indexed. syGtx.png
Technical SEO | | Mattchstick0 -
Home page penalty?
What does it mean when your home page has a penalty? I have a site that has good rankings for many pages, but my home page seems to be penalized by Google. I tried searching for my home page URL in Google, www.xxxxxx.com and my page doesn't show up, but sub pages do show up? What would cause this penalty and how do you correct this issue.
Technical SEO | | tadden0 -
No. of links on a page
Is it true that If there is a huge number of links from the source page then each link will provide very little value in terms of passing link juice ?
Technical SEO | | seoug_20050