Thinking aloud - what if WE could run rogerbot from our desktops?
-
Total, total noob question, I know - but is rogerbot performance bound because of bandwidth and processing capacity? I understand if it is, but I am wondering for those of us with very large sites if we would be able to offload the burden on SEOmoz resources by running our own local licensed version of rogerbot, crawl the sites we want and the upload the data to SEOmoz for analysis.
If this was possible would we be getting more immediate results?
-
On the topic of a private crawl (or distributed crawl), these are cool ideas, but not something we currently have in our plans. Having the crawl centralized allows us to store historic data and ensure polite crawling. This may take a little extra time (we are indeed doing a lot of crawls, as well as processing them and retrieving link data for each of them), but we are actively working on on our infrastructure to reduce our crawling and processing time.
While the first crawl does take a number of days, subsequent crawls are started on the same day each week, and should take roughly the same amount of time to complete, controlling for external factors. So in general you should have fresh crawl data right around weekly, give or take a day or two.
As for your specific crawls, I'd be happy to look into them for you. I'll send you a separate email to discuss.
-
Still waiting for my custom crawl launched on April 28th (it's May 2nd) to complete... seriously? If SEOmoz is overwhelmed, first of all, congratualations on being so popular, but I am not getting any timely data at all.
-
Or, instead of running Rogerbot locally, we could run some distributed processing (ie. BOINC) to help offload some of the pressure from your cloud?
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do you think have to re-submit my site to search engines after I made improvements?
Some time ago I started to do SEO for a one-page website and didn't get any positive result: no traffic, no filled in online booking form (yet another, multiple page website offering the same service yielded in multiple filled-in "schedule an appointment" forms). I found out my one-page website was considered to be "keyword-spamming" and converted it to a multiple page one. Its domain authority went up, but it doesn't still bring any traffic. I am thinking maybe I have to let the search engines know that it has been updated so they stop penalizing it? Do you think it might help and if yes, what exactly I should do? Will be thankful very much for any suggestion!
Technical SEO | | kirupa0 -
Do mobile and desktop sites that pull content from the same source count as duplicate content?
We are about to launch a mobile site that pulls content from the same CMS, including metadata. They both have different top-level domains, however (www.abcd.com and www.m.abcd.com). How will this affect us in terms of search engine ranking?
Technical SEO | | ovenbird0 -
I'm thinking I might need to canonicalize back to the home site and combine some content, what do you think?
I have a site that is mostly just podcasts with transcripts, and it has both audio and video versions of the podcasts. I also have a blog that I contribute to that links back to the video/transcript page of these podcasts. So this blog I contribute to has the exact same content (the podcast; both audio and video but no transcript) and then an audio and video version of this podcast. Each post of the podcast has different content on it that is technically unique but I'm not sure it's unique enough. So my question is, should I canonicalize the posts on this blog back to the original video/transcript page of the podcast and then combine the video with the audio posts. Thanks!
Technical SEO | | ThridHour0 -
Does Google think My main keyword is my Company name due to over use in anchor text?
OK, i have recently noticed google is adding a keyword to my titles... I have attached an image which shows the results from a google site:mywebsite search... 8 out of the 10 results on the screen shot clearly have the keyword ( - CCTV Cameras) added to the end of the title. Now a lot of those pages are support pages i.e "Help & Support", "returns" etc etc.. nothing really to do with cctv cameras. The only thing i can think of, is the most common anchor text used on backlinks to my website (according to siteexplorer) is the keyword cctv cameras... (i am currently working on removing a load of the backlinks to my site due to what i think is some kind of google penalty as the anchor text is all keywords etc...) But it seems strange if google has started to discount these backlinks, that it then uses that keyword to add to my titles - unless of course, google thinks that it is my brand name... Any body have any thoughts on this? FqwooQY.gif
Technical SEO | | isntworkdull1 -
I think I have a penalty on my domain...
my domain is www.brighttights.com it is an affiliate marketing website in the niche of tights and lingerie. A few months back my traffic was pretty good, doing about 500 hits a day from product search terms only. After the panda updates I blocked all the product pages from google as they were duplicate content and I am now working on a program of seing for the category and homepages instead. I am using much more generic, and high volume, keywords for these. Several months later I seem to not only be down to 7 people a day on my website but i'm not even ranking for terms such as "bright tights". I used to be no1 for this. I have domain authority of 27 so it's not terrible, competitors on the first page range from 45 to 9. This lack of ranking for the sites name/domain name term is leading me to wonder if I have a penalty on the site. Any feedback would be gratefully received.
Technical SEO | | Grumpy_Carl0 -
Prevent mobile site from appearing in the sitelinks of desktop search
Hi, IWe have this mobile page that keeps on appearing in the google search. I even try to put it in the robots.txt to disallow the crawler but still it keeps on popping on the search results. How can I prevent it from displaying?
Technical SEO | | shebinhassan0 -
Does running 2 domains from the same website harm my seo efforts?
Hi I run a website which has 2 domains running from it. One domain has a .co.uk extension and shows products aimed at the UK market. The other domain is a .com which shows products aimed at the US market. As both domains run from the same website, the content is mostly identical APART from the product listings as they change depending upon whether you're in the UK or US. My reason for doing is this mainly for paid search purposes as I can have a more accurate domain depending upon which country I target. Both sites are set to encourage the bots to index it, and in google WMT each domain is targeted to its specific country. So, my questions are, Is this set-up possibly damaging my SEO efforts for either or both websites? If so, would setting one domain to no-index, improve SEO efforts for the other? Thanks in advance for any replies! Cheers Joe
Technical SEO | | joeprice0 -
.us domains vs .com - What does Google Think?
Suppose I had 2 domains, carloans.us & carloans.com with exactly the same links profiles, and content (not duplicate but you know what I mean). Would Google favour the .com domain? In my experience, yes. But I might be wrong?
Technical SEO | | Tom-R
Same with other not so standard domains like .biz etc. Am I right to believe that Google can prefer the more common domain extensions?0