Screaming frog Advice
-
Hi
I am trying to crawl my site and it keeps crashing.
My sys admins keeps upgrading the virtual box it sits on and it now currently has 8GB of memory, but still crashes.
It gets to around 200k pages crawl and dies.
Any tips on how I can crawl my whole site, can u use screaming frog to crawl part of a site.
Thanks in advance for any tips.
Andy
-
Thanks, I tried all the tips on the screaming frog site, but I have just tried to 2 pages a second and lets hope that work.
-
Hi Andy. There are quite a few settings you can adjust to make the server load less while the crawl is running. These can be found with descriptions here: http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
For example, by not checking Images, CSS, SWF, and Javascript you'll be able to lessen load substantially, or if you'd like to crawl just a portion of the site you can set it to not check links outside of the start folder.
To have even more control over the crawl, you can use regular expressions to exclude certain pages, or sections that match a given pattern. The page above is fairly robust, so it should help you dial back the crawler to be friendlier to your server. Cheers!
-
Hey there mate,
Sorry to hear that you are having issues. You can actually ask Screaming Frog to use more RAM. If you haven't done that yet please give it a go.
You can find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
If you want to crawl part of your site it can surely do that. You can exclude pages or whole sections.
Find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
Hope this helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Keyword rank drop, any advice?
Screen Shot 2021-08-26 at 19.02.18.png My search visibility dropped from around 13% a few weeks ago to 8.29%. I know that Google launched a bunch of updates in this past few weeks to ignore spam links, and I'm pretty sure that was the reason for the drop - some of the links to my site date back over 10 years and those links were garbage. Confusingly, at the same time, my Domain Authority went up by 1 to 32, then back down a week later. How can I restore my previous rank in the short term? We're designing a new site at the moment with vastly improved page speed, but I'm not sure what effect that will have yet (thespacecollective.com).
Intermediate & Advanced SEO | | moon-boots0 -
How do I not mess up a national seo strategy? All advice appreciated in advance!
Hello Moz Friends So I have always worked on local seo, websites, and digital brand management. But one of my clients wants to go national. They sell building materials and the competition is pretty fierce. So here is my question and I thank you in advance for any advice: How do I not mess up a national seo strategy, based on local seo efforts? So for an example, say I put the NAP in the footer, but my client sells to 49 states, will this hurt a national campaign? And on the opposite, would getting local links from say construction websites in California, if my NAP is from Florida, hurt or help? I understand that most of this is seo 101, but I am so used to local seo, that I don't want to cause national issues by doing something local. Thank you again Moz Friends "Have a Positive Day" Chris
Intermediate & Advanced SEO | | asbchris1 -
Wordpresss Bug? Duplicate pages yet again, Adviced Needed
I have work up this morning with a large number of missing meta description, when I looked at th pages (knowing I have to missing metas) I have duplicates of my pages that look like this, These pages normally look like www.finalduties.co.uk/ NAME OF POST now i have urls with numbers in, seems as though all my blog posts have duplicated.. Now could this be down to a wordpress bug? I am so fed up, I had all my errors all cleared up only to wake and have pages like this, which is going to cause me a problem because my pages are now duplicating.. please help, advice needed from someone that knows wordpress? | Father and son battle over 13th century inheritance http://www.finalduties.co.uk/?p=1006 1 0 Court of Appeal rules against RSCPA http://www.finalduties.co.uk/?p=1007 1 0 Life insurance policy payout can skip probate process http://www.finalduties.co.uk/?p=1008 1 0 Government clamping down on inheritance tax avoidance http://www.finalduties.co.uk/?p=1010 |
Intermediate & Advanced SEO | | Chris__Chris0 -
Advice needed on how to handle alleged duplicate content and titles
Hi I wonder if anyone can advise on something that's got me scratching my head. The following are examples of urls which are deemed to have duplicate content and title tags. This causes around 8000 errors, which (for the most part) are valid urls because they provide different views on market data. e.g. #1 is the summary, while #2 is 'Holdings and Sector weightings'. #3 is odd because it's crawling the anchored link. I didn't think hashes were crawled? I'd like some advice on how best to handle these, because, really they're just queries against a master url and I'd like to remove the noise around duplicate errors so that I can focus on some other true duplicate url issues we have. Here's some example urls on the same page which are deemed as duplicates. 1) http://markets.ft.com/Research/Markets/Tearsheets/Summary?s=IVPM:LSE http://markets.ft.com/Research/Markets/Tearsheets/Holdings-and-sectors-weighting?s=IVPM:LSE http://markets.ft.com/Research/Markets/Tearsheets/Summary?s=IVPM:LSE&widgets=1 What's the best way to handle this?
Intermediate & Advanced SEO | | SearchPM0 -
Domain advice needed, please
Could i get a little domain advice please. Launching a new website project and want to put it on a domain we already own (both domains are in the same niche as the larger project). The new project will be aimed at the UK market. The choice is: .co.uk we own with a good name, however it's Domain Authority rank is 7 and it's only about 8 months old. .com domain which is 6 years old, has a Domain Authority rank of 33 but is not as good a domain name. The Competitive Link Analysis tells me that the rivals for the keywords we would be targeting are between 24 and 42. Which domain would people go with? All things equal it would be a fair guess that the older, higher Domain Authority ranked .com will require less work to rank in the engines, however it's not as good brand wise. Thanks Carl
Intermediate & Advanced SEO | | Grumpy_Carl0 -
Conversion Rate Optimisation - advice before seeking out a specialist
Hello! I have a site I've been working on for a lovely client, a small business start up since Christmas. The site has a very simple layout, is ranking well and maintaining its positions, has solid social media, is receiving enough traffic and ranking for a number of terms. The problem is - conversions! The site just isn't converting. I have spoken with a few peers who have said advanced CRO will be too much for me to learn in terms of Psychology of Buying, learning about colors, fonts etc. I understand meta descriptions for example are something that I can do, I was wondering if anyone could give me advice on any other basic CRO techniques I could apply to the site before going to a specialist. Any advice would be MUCH appreciated - the moz community is always so helpful! Charlotte 🙂
Intermediate & Advanced SEO | | CharlotteWaller1 -
An Infrastructure Change for a Large eCommerce Site - Any advice?
Hello Mozers, We're currently under going quite a large infrastructure change to our website and I wouldn't to hear your thoughts on the type of things we should be careful of. We currently have close to 4,000 individual products each with their own page. The seo work is then driven behind certain pages which house a catalog display of groups of products. The groups are done by style. An example is we have a page called "Style A" which displays 8 different colours of style A. We then seo the style A page and the individual items received minimal seo work. The change would involve having one individual product page for each style but on that page the user would have the ability to purchase the different colours/variations via menus. This will result in approximately a %70 reduction in the size of our site (as several products will no longer be published) The things we are currently concerned with are: 1. The lose of equity to those unwanted 'style A' pages - I think a series of careful planned 301s will be the solution. 2. Possible loss of long tail traffic to the individual products which might not be caught by one individual page per style. 3. Internal link structure will need to be monitored to make sure that we're still highlight the most important pages as well, important. Sorry for the long post, it's a difficult change to explain without revealing the clients name - any other things we should be thinking about would be greatly appreciated! Thanks Nigel
Intermediate & Advanced SEO | | NigelJ0 -
Need advice on local search optimization
HI all, I've found myself in a puzzling position and not quite sure which direction to push my current SEO project so if anyone who's done this particular type of SEO can offer some suggestions I'd be eternaly grateful. I am currently working on a project for a Law Firm based in New Jersey. Lets say the town they are in is Garfield. What I really want to try and achieve is see them appearing in the number one spot whenever anyone within Garfield or the immediate area searches for a lawyer relating to the individuals need. E..g searches like "personal injury lawyers", "real estate lawyer". The problem is I can see how I can easily make it to the number one position if people are specific and enter garfield in the search term but in reality they wouldn't be doing that. An additional problem is that peoples ISP's in garfield aren't located in Garfield, in some cases they're as far away as Newark so when they're doing a search for 'real estate lawyer' google is bringing up results for the Newark based firms. It seems using tools like market samurai to look at the traffic and competition is proving useless as searches like the ones I'm doing for local business are so closely tied to the ISP location I don't really know whether to target broad range searches like "Real Estate Lawyer", or to be really specific and include the town name in my page titles, H1 tags etc... I hope I put across my dilemma and someone can help me chose which direction to go in.. Thanks
Intermediate & Advanced SEO | | davebrown19750