Should /dev folder be blocked?
-
I have been experiencing a ranking drop every two months, so I came upon a new theory this morning...
Does Google do a deep crawl of your site say every 60-90 days and would they penalize a site if they crawled into your /dev area which would contain pretty the exact same urls and content as your production environment and therefore penalize you for duplicate content?
The only issue I see with this theory is that I have been penalized only for specific keywords on specific pages, not necessarily across the board.
Thoughts?
What would be the best way to block out your /dev area?
-
Hey Ryan,
It isn't really a competition thing because I bounce back to the same or better sport after about 30 days, so it seems very algorithmic. I just haven't been able to figure what/how I am getting penalized. However, this morning while looking at some rankings I noticed a dev page was indexed and in Google's rankings so I said "Ah oh...." and went to see my developer hadn't blocked out our development directory... which means if Google deep indexes a site every so often they would probably crawl it and find an exact copy of the live site...
The 60-90 deep indexing would also make sense as the ranking drop has always occurred around the 27th of the month.
Once I block out /dev what do you think would be the best way to get Google to re-crawl the site and perhaps remove any duplicate issues - delete out my robots.txt and sitemaps from Google and re-submit them?
-
It is a great idea to block the development portion of your site. You can do that with your robots.txt file:
User-Agent: *
Disallow: /dev
But you're right, it doesn't sound like the culprit in your case. Its more likely that your competition is slowly gaining the edge on you. Make sure your on-page SEO is optimized and then try to get some more inbound links to your pages.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a way to host my website.com/BLOG URL PATH from a different host than my main website.com host?
Is there a way to host my website.com/BLOG URL PATH from a different host than my main website.com host? Is it accomplish-able with DNS settings or are there other considerations that might lead to complications doing this? Specifically, we are investigating install WordPress on a dedicated host, JUST to power the blog for our main website, but our main website is on an internal proprietary hosting and CMS. So basically we're trying to host: website.com --> OFF OF CURRENT INTERNAL HOSTING website.com/blog/ --> OFF OF THIRD PARTY HOSTING (USING WORDPRESS) I know this is a technical question beyond the scope of SEO, but I'm figuring there are members of the community that may have tried this already so I'm floating it here. Many thanks! Cheers.
Web Design | | AlexVelazquez0 -
Pageless/Single Page Design and Migration Questions
Hello, We are starting a content audit and migration to a new CMS. We would like to take content and present more on a pageless/Single Page type design instead of having visitors drill down so many levels to find the content. What should we be aware of from an SEO perspective. Here is Example of current pages and structure: http://www.saintpetershcs.com/GraduateMedicalEducation/PediatricResidency/
Web Design | | sphcs
Subpages include: Overview, Curriculum, Faculty, Residents, Benefits, How to APply Here is example of what we would like to do:
http://themeforest.net/item/medicalpress-health-and-medical-wordpress-theme/full_screen_preview/7789703 As you scroll information is populated: Duke Medicine also has something similar. https://www.dukemedicine.org/treatments/cancer What are your thoughts?0 -
Google tag manager on blocked beta site - will it phone home to Google and cause site to get indexed?
We want to develop a beta site, in a directory with the robots.txt blocking bots. We want to include the Google Tag Manager tags and event layer tracking code on this beta site. My question is that by including the Google Tag Manager code, that phones home to Google, will it cause Google to index this beta site when we don't want it indexed?
Web Design | | CFSSEO0 -
Google pagespeed / lazy image load
Hi, we are using the apache module of google pagespeed. It works really great, helps a lot. But today I've asked me one question: Does the "lazy load" feature for images harm the ranking? The module reworks the page to load the images only if the are visible at the screen. Is this behavior also triggered by the google bot? Or are the images invisible for google? Any expirience about that? Best wishes, Georg.
Web Design | | GeorgFranz0 -
How to rewrite/redirect a folder name with .htaccess
I have a folder in my site that I want to rename. I don’t want to just rewrite the URL and keep my old folder name, I want to change the folder name and then do whatever is necessary with .hataccess to not lose search engine rankings. The folder name I want to change has a space in it and also is misspelled (whoops x2) Example. Mysite.com/old foldr/page.html Mysite.com/newfolder/page.html How would I go about doing this with .htaccess? Do I just switch the folder name on my server and then set up a redirect, or do I do a rewrite? Sorry now familiar with the terms or .htacces. Thanks all
Web Design | | SheffieldMarketing0 -
Duplicate home page /index.asp /index.php etc
We recently moved www.devoted2vntage.co.uk to shopify but seem to have multiple home page variants still in google index. I am concerned that these will be causing duplicate content. I have redirected the offending URLs below to www.devoted2vintage.co.uk/ and have set up a canonical URL but need an expect to tell me if I have taken the current steps and if not, exactly what I need to do. www.devoted2vintage.co.uk/index.php www.devoted2vintage.co.uk/index.htm www.devoted2vintage.co.uk/index.html www.devoted2vintage.co.uk/index.shtml www.devoted2vintage.co.uk/index.aspx www.devoted2vintage.co.uk/index.cfm www.devoted2vintage.co.uk/index.pl www.devoted2vintage.co.uk/index.asp
Web Design | | devoted2vintage0 -
RSS Submissions Positive/Negative/Neutral Impact...
We are looking to push our site content and blog content out to the masses... There are several sites and services that accept RSS feeds or submit RSS feeds to 50+ RSS sites. Have you seen an positive or negative impact of submitting feeds to these RSS directories? I am primarily looking at this for getting or content out and builing inbound links... Any thoughts or feedback would be appreciated... C
Web Design | | hireawizseo0 -
Need help setting up google analytics goals / tracking
I don't use Google analytics to see much more than how many visits I'm getting and what sort of keywords people are using to find our site. I'd like to step up my GA skills a bit. I'm wondring if you guys could give me some advice. I've never really set up any GA goals, or used it to track specific things, but I'd like to. Here are a few things off the top of my head that I would like to track. I'm wondring if these are posable in GA, and if someone could give me some feedback on how to track it / set up goals. Thanks.1 1. I'd like to know how many people click play on a video when they are on a page that has a video. 2. I'd like to know how many people are clicking "like" " google plus, etc.) 3. I'd like to know the path people are taking on our site. For instance, if they click a link from Facebook, and go to a landing page, what page are they visiting next..... 4. How long people are staying on the page I would really like to break this down further by people that visit a link I posted on Facebook, or twitter, or from the link on my twitter profile page, etc... Also if there are any other valuable goals / reports that would be useful for a blogger to track I'd appreciate your feedback. Thanks.
Web Design | | NoahsDad0