Should /dev folder be blocked?
-
I have been experiencing a ranking drop every two months, so I came upon a new theory this morning...
Does Google do a deep crawl of your site say every 60-90 days and would they penalize a site if they crawled into your /dev area which would contain pretty the exact same urls and content as your production environment and therefore penalize you for duplicate content?
The only issue I see with this theory is that I have been penalized only for specific keywords on specific pages, not necessarily across the board.
Thoughts?
What would be the best way to block out your /dev area?
-
Hey Ryan,
It isn't really a competition thing because I bounce back to the same or better sport after about 30 days, so it seems very algorithmic. I just haven't been able to figure what/how I am getting penalized. However, this morning while looking at some rankings I noticed a dev page was indexed and in Google's rankings so I said "Ah oh...." and went to see my developer hadn't blocked out our development directory... which means if Google deep indexes a site every so often they would probably crawl it and find an exact copy of the live site...
The 60-90 deep indexing would also make sense as the ranking drop has always occurred around the 27th of the month.
Once I block out /dev what do you think would be the best way to get Google to re-crawl the site and perhaps remove any duplicate issues - delete out my robots.txt and sitemaps from Google and re-submit them?
-
It is a great idea to block the development portion of your site. You can do that with your robots.txt file:
User-Agent: *
Disallow: /dev
But you're right, it doesn't sound like the culprit in your case. Its more likely that your competition is slowly gaining the edge on you. Make sure your on-page SEO is optimized and then try to get some more inbound links to your pages.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website Redesign & Ensuring Minimal Traffic/Rankings Lost
Hi there, We have undergone a website redesign (mycompany.com) and our site is ready to go live however the new website is built on a different platform so all of our blog pages will not be copied over - to avoid a large web developer expense. So our intention is to then leave all the blog pages as (on the old web design) but move it to the domain blog.mycompany.com with 301 redirects inserted on mycompany.com for each blog post pointing to the corresponding blog.mycompany.com. Is there anything else we should do to ensure minimal traffic/rankings are lost? Thank you so much for your help.
Web Design | | amitbroide0 -
Opencart vs. Wordpress/Woo
There are two issues facing me today. One is that my two e-commerce stores need updating after some 4 years, but I am seriously considering switching from Opencart to Wordpress/Woo. Opencart is a nightmare to work with at the best of times. Whenever I try to edit the footer of my current sites for instance nothing changes, the customisation of pages is sloppy and although the site works fine for perhaps the first 6 months, anytime after that it just slowly falls apart. Wordpress however features incredible customisation, is easy to edit the code but it lacks the backend functionality that Opencart is good at. Does anyone know the downsides of changing to Wordpress/Woo in respect to SEO?
Web Design | | moon-boots0 -
Is it still necessary to have a "home" page button/link in the top nav?
Or is it not necessary to have a "home" tab/link because everybody by this time knows you can get to the home page by clicking on the logo?
Web Design | | FindLaw0 -
How to split organic traffic for A/B testing
This might be a silly questions as I may be missing something completely obvious here, but we are completely new to A/B testing. Our site doesn't receive a phenomenal amount of traffic although we are looking to set up some A/B testing for our popular products. Is there a way to split organic traffic for a specific product page. I'm aware that we need to experiment which one performs better in Analytics but I'm unsure how to redirect 50% of the organic traffic.
Web Design | | Jseddon920 -
Fixing Render Blocking Javascript and CSS in the Above-the-fold content
We don't have a responsive design site yet, and our mobile site is built through Dudamobile. I know it's not the best, but I'm trying to do whatever we can until we get around to redesigning it. Is there anything I can do about the following Page Speed Insight errors or are they just a function of using Dudamobile? Eliminate render-blocking JavaScript and CSS in above-the-fold content Your page has 3 blocking script resources and 5 blocking CSS resources. This causes a delay in rendering your page.None of the above-the-fold content on your page could be rendered without waiting for the following resources to load. Try to defer or asynchronously load blocking resources, or inline the critical portions of those resources directly in the HTML.Remove render-blocking JavaScript: http://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js http://mobile.dudamobile.com/…ckage.min.js?version=2015-04-02T13:36:04 http://mobile.dudamobile.com/…pts/blogs.js?version=2015-04-02T13:36:04 Optimize CSS Delivery of the following: http://fonts.googleapis.com/…:400|Great+Vibes|Signika:400,300,600,700 http://mobile.dudamobile.com/…ont-pack.css?version=2015-04-02T13:36:04 http://mobile.dudamobile.com/…kage.min.css?version=2015-04-02T13:36:04 http://irp-cdn.multiscreensite.com/kempruge/files/kempruge_0.min.css?v=6 http://irp-cdn.multiscreensite.com/…mpruge/files/kempruge_home_0.min.css?v=6 Thanks for any tips, Ruben
Web Design | | KempRugeLawGroup0 -
SEO strategy for UK / US websites
Hi, We currently have a UK-focused site on www.palmatin.com ; We're now targeting the North American market as well, but the contents of the site need to be different from UK. One option was to create another domain for the NA market but I assume it would be easier to rank with palmatin.com though. What would you suggest to do, if a company is targeting two different countries in the same language? thanks, jaan
Web Design | | JaanMSonberg0 -
Privacy Policy: index it/? And where to place it?
Hi Everyone, Two questions, first: should you allow google to index your privacy policy? Second: for a service based site (not e-commerce, not selling anything) should you put the policy in the footer so it's site wide or just on the "contact us" form page? Best, Ruben
Web Design | | KempRugeLawGroup0 -
What to do with blog pagination / categories
On our new website we are going to have a blog that is custom built. However, you can browse the general blog with pagination. You can also nest into the categories or by month. Doing each will have a different link: Category: www.mywebsite.com/blog/web-guides/Month: www.mywebsite.com/blog/2014/january/ My question is what is the best thing to do with these pages. Rel Canonical them to the blog home? Or nofollow/noindex them? Will Google not see these as duplicate content pages?
Web Design | | vortexuk0