Screaming frog Advice
-
Hi
I am trying to crawl my site and it keeps crashing.
My sys admins keeps upgrading the virtual box it sits on and it now currently has 8GB of memory, but still crashes.
It gets to around 200k pages crawl and dies.
Any tips on how I can crawl my whole site, can u use screaming frog to crawl part of a site.
Thanks in advance for any tips.
Andy
-
Thanks, I tried all the tips on the screaming frog site, but I have just tried to 2 pages a second and lets hope that work.
-
Hi Andy. There are quite a few settings you can adjust to make the server load less while the crawl is running. These can be found with descriptions here: http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
For example, by not checking Images, CSS, SWF, and Javascript you'll be able to lessen load substantially, or if you'd like to crawl just a portion of the site you can set it to not check links outside of the start folder.
To have even more control over the crawl, you can use regular expressions to exclude certain pages, or sections that match a given pattern. The page above is fairly robust, so it should help you dial back the crawler to be friendlier to your server. Cheers!
-
Hey there mate,
Sorry to hear that you are having issues. You can actually ask Screaming Frog to use more RAM. If you haven't done that yet please give it a go.
You can find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
If you want to crawl part of your site it can surely do that. You can exclude pages or whole sections.
Find more here http://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/
Hope this helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Keyword rank drop, any advice?
Screen Shot 2021-08-26 at 19.02.18.png My search visibility dropped from around 13% a few weeks ago to 8.29%. I know that Google launched a bunch of updates in this past few weeks to ignore spam links, and I'm pretty sure that was the reason for the drop - some of the links to my site date back over 10 years and those links were garbage. Confusingly, at the same time, my Domain Authority went up by 1 to 32, then back down a week later. How can I restore my previous rank in the short term? We're designing a new site at the moment with vastly improved page speed, but I'm not sure what effect that will have yet (thespacecollective.com).
Intermediate & Advanced SEO | | moon-boots0 -
Probably basic, but how to use image Title and Alt Text - and confusing advice from Moz!
I've been doing SEO on my business's site for years and have got good results. I've always used image Titles and Alt Text text. Our blog posts are image-intensive, often with 100-200 pictures (not surprising since we're photographers). For any given blog post, I've tended to have a uniform image Title for each image and then a more specialised Alt Text tag giving a description. A typical image on one of our blog posts would be like this: Image filename: wedding-photography-at-so-and-so-venue-001.jpg .... 002, 003 etc Image Title Attribute: Wedding Photography at So-And-So-Venue by Our-Company-Name - this would be the same for every image in the blog post. Alternative Text: Bride and groom exchanging vows during wedding ceremony at so-and-so-venue - this would be tailed for each image. So my question is - is this right? The Moz help page for image SEO is actually incorrect in one aspect: https://mza.seotoolninja.com/ugc/10-tips-for-optimizing-your-images-for-search "Alt text (short for “alternative text”) is used to highlight the identity of an image when you hover over it with your mouse cursor. It also shows as text to all users when there are problems rendering the image." This is not the case. Hovering over the image in Firefox, Chrome, Edge and Opera ALL display the Image Title, NOT Alt Text. Thoughts?
Intermediate & Advanced SEO | | robandsarahgillespie0 -
Need advice and smart solution for H1
Hi! I've been quite a long time trying to find a solution to the html structure of a webpage that I'am auditing right now. But I need your valuable help! The problem is the following: Actually, they have different sections inside. The structure is something like this Masters - Degrees - Programs - About us - News And if you go to the Masters Section you will find something like this Masters Master number 1 in Tourism (brand.com/master/master-number-1-tourism) Presentation
Intermediate & Advanced SEO | | teconsite
(brand.com/master/master-number-1-tourism) Objectives
(brand.com/master/master-number-1-tourism/objectives) Professional opportunites
(brand.com/master/master-number-1-tourism/professional-opportunities) Faculty
(brand.com/master/master-number-1-tourism/faculty) Qualification
(brand.com/master/master-number-1-tourism/qualification) Financial
(brand.com/master/master-number-1-tourism/financial) Master number 2 with a long name Presentation Objetives Profesisional opportunities Faculty ... Master number 3 in Sports and so on The Degrees section, has inside exactly the same structure with the same names. My doubt is related with the use of h1 tag What would be the best h1 strategy for each content page? Each master has 6 pages (presentation, objectives, faculty,...) For page Objetives,
brand.com/master/master-number-1-tourism/objectives If I choose to use as H1 just the word Objetives, what will happen is that I will have a lot of pages (one per master, degree or program), with the same H1, because each master will have its own page Objectives. If we have 10 masters + 10 degrees + 2 programs y will have 22 pages with the same H1 Objecives If I choose to use as H1 the following: "Objectives of the Master Number 1 in Tourisim and so on with long name" it will be difficult for the users to visually see the difference between the different pages. for instance Objectives of the Master Number 1 in Tourisim and so on with long name and Faculty of the Master Number 1 in Tourisim and so on with long name because they only differ in one word. What do you think of this solution? Objectives Master Number 1 in Tourisim and so on with long name is it correct to do this inside the h1? or would you use combinations of h1 and h2 like these
h1: Objectives
h2: Master Number 1 in Tourisim and so on with long name would be this appropiate?0 -
Strange 404s in Screaming Frog
I just ran a website (Drupal) through screaming frog and the only 404s I found related to web pages which were the same as URLs already used on the website plus the company phone number so... www.company.com/[their phone number] - www.company.com/services[their phone number] - any ideas what might be causing this problem?
Intermediate & Advanced SEO | | McTaggart0 -
[Need advice!] A particular question about a subdomain to subfolder switch
Hello Moz Community! I really was hoping to get your help on a issue that is bothering me for a while now. I know there is a lot of about this topic but I couldn’t find a good answer for my particular question. We are running several web applications that are similar but are also different from each other. Right now, each one has its own subdomain (which was mainly due to technical reasons). Like this: webapp1.rootdomain.com, webapp2.rootdomain.com etc. Our root domain currently points with 301 to webapp1.rootdomain.com. Now, we are thinking about making two changes: changing to a subfolder level like this: rootdomain.com/webapp1 , rootdomain.com/webapp2 etc. Changing our rootdomain to a landing page (lisitng all the apps) and take out the 301 to webapp1 We want to do these changes mainly for SEO reasons. I know that the advantages are not so clear between subdomain/subfolder but we think it could be the right way to go to push the root domain and profit more from juice passing to the different apps. The problem is that we had a bad experience when we first switched from our first wep app (rootdomain.com) to an subdomain (webapp1.rootdomain.com) to set them equal with the other apps. Our traffic dropped a lot and it took us 6 weeks to get back on the same level as before. Maybe it was the 301 not passing all juice or maybe it was the switch to the subdomain. We are not sure. So, I guess my question is do you think it is the right thing to do for web apps to go with subfolders to pass more juice from root to subfolders? Will it bring again huge drops in traffic once we make that change? Is it worth taking that risk or initial drop because it will pay off in the future? Thanks a lot in advance! Your answers would help me a lot.
Intermediate & Advanced SEO | | ummaterial0 -
Need advice on 301 domain redirection
Hello friends, We have two sites namely spiderman-example.com & avengers-example.com which sells the same product listed out under similar categories, since we are about to stop or put down the site “avengers-example.com” because we just want to concentrate in bringing up a single brand called spiderman-example.com. “Spiderman-example” has comparatively more visitors and conversion rates than ''avengers-example'' ie. 90 % more traffic and conversion. Avengers-example has a small fraction of loyal customers who still search for the brand-name & there are a hand-full of potential keywords those ranking on its own. So is it advisable to redirect Avengers-example to spiderman-example using 301-redirect? Will this help to gain any link-juice from Avengers-example? If so how can we effectively redirect between two domain’s with minimal loss in page authority & linkjuice to enhance ''spiderman-example''? Off beat:These names "Avengers" and "Spiderman" were just used as an example but the actual site names has no relation to the ones mentioned above.
Intermediate & Advanced SEO | | semvibe0 -
We've just bought a new domain - need advice on the exact procedure to follow...
Hi guys, We've just bought a 3 letter .co.uk domain to replace our current 20 character old domain. Our existing domain is PR5 with quite a few links (that we can modify no problem) We're currently .301 redirecting the new domain to the old domain. I was looking at the procedure in one of the guides but as it's slightly different - is this the correct procedure? 1. prep the duplicate site on new domain and prep the individual htaccess .301 redirects 2. Add new the domain to google webmaster tools bing Webmaster centre 2. On the switchover date - modify all possible incoming links from external sites 3. On the switchover date - apply the .301 redirects and make the new site live 4. On the switchover date - apply the new sitemaps to google & bing 5. on the switchover date - fill out the change of address form in webmaster tools 6. Do the happy dance? many thanks in advance, Tony.
Intermediate & Advanced SEO | | posh_tiger0 -
1st Campaign - Advice please.
We have just run our first campaign for our site and have found over 5,335 errors! It would appear that the majority of which are where the crawl has duplicated the product page with the "write a review - Tell a friend page"...hence a large number of errors. In addition we also have over 5,000 302 warnings for the following URL: URL: http://www.collarandcuff.co.uk/index.php?_a=login&redir=/index.php?_a=viewCat&catId=105 Please bear in mind we are fairly new to this type of data....so go easy on us. In short, will these errors have a significant bearing on our rankings etc and if so how do we rectify? Many thanks. Tony
Intermediate & Advanced SEO | | collar640