How did my dev site end up in the search results?
-
We use a subdomain for our dev site. I never thought anything of it because the only way you can reach the dev site is through a vpn. Google has somehow indexed it. Any ideas on how that happened? I am adding the noindex tag, should I used canonical? Or is there anything else you can think of?
-
Personally, I'd still recommend using robots.txt to disallow all crawlers, even if more steps are taken.
-
Don't use tool removal, it can go bad indeed. Now, are you sure that there are no external links coming from anywhere?
For now I'd recommend putting noindex, nofollow on that dev subdomain and do manual recrawl through GWT.
-
It just uses internal links. Do you think I should try the webmaster tools removal? That seems like it could go wrong.
-
I never used screaming frog, does it check both external and internal links?
-
I have ran screaming frog to see if there are any links to any pages and but couldn't see any. Even if Google did try to follow it the firewall would stop them. It is so strange.
-
Then my first assumption is that it's linked from somewhere - read my comment a little above.
-
Then there is a leak somewhere - Google bots can "see" your subdomain.
Or it's been simply linked from somewhere. Then Google will try to follow the link and that would make it indexed.
-
They are telling me that there are no holes, and I have tried getting to the pages but can not do it unless I am on my vpn.
-
We never updated the robots.txt because the site was behind a firewall. If you click on any of the results it will not load the page unless on my VPN.
-
Robots.txt won't help anyhow. Bots still can see that there is such directory, they just won't see what's inside of those directories/subdomains.
-
Hi there.
If what you say is true, then there are only two answers: you got a leak somewhere or your settings/configuration is messed up.I'd say go talk to your system admin and make sure that everything what's supposed to be closed is closed, IPs, which are supposed to be open for use are open and those IPs only.
-
Have you updated the dev sites robots.txt to disallow everything? It is up to the bot to listen, but that combined with removing all of the dev URLs from Google Webmaster tools should do the trick.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Suddenly keywords Disappeared from Google Search Results
Hello Guys Please help me, suddenly all of my site's keywords are disappeared from google search result, most of keywords are no.1 on google but today after 6pm i see the traffic decreasing and when i search my keywords there is no any keywords in search result. Only homepage keyword is showing. Please Help what is Happening with me.
Intermediate & Advanced SEO | | mianazeem4180 -
Google Indexed Site A's Content On Site B, Site C etc
Hi All, I have an issue where the content (pages and images) of Site A (www.ericreynolds.photography) are showing up in Google under different domains Site B (www.fastphonerepair.com), Site C (www.quarryhillvet.com), Site D (www.spacasey.com). I believe this happened because I installed an SSL cert on Site A but didn't have the default SSL domain set on the server. You were able to access Site B and any page from Site A and it would pull up properly. I have since fixed that SSL issue and am now doing a 301 redirect from Sites B, C and D to Site A for anything https since Sites B, C, D are not using an SSL cert. My question is, how can I trigger google to re-index all of the sites to remove the wrong listings in the index. I have a screen shot attached so you can see the issue clearer. I have resubmitted my site map but I'm not seeing much of a change in the index for my site. Any help on what I could do would be great. Thanks
Intermediate & Advanced SEO | | cwscontent
Eric TeVM49b.png qPtXvME.png1 -
Indexed Pages Different when I perform a "site:Google.com" site search - why?
My client has an ecommerce website with approx. 300,000 URLs (a lot of these are parameters blocked by the spiders thru meta robots tag). There are 9,000 "true" URLs being submitted to Google Search Console, Google says they are indexing 8,000 of them. Here's the weird part - When I do a "site:website" function search in Google, it says Google is indexing 2.2 million pages on the URL, but I am unable to view past page 14 of the SERPs. It just stops showing results and I don't even get a "the next results are duplicate results" message." What is happening? Why does Google say they are indexing 2.2 million URLs, but then won't show me more than 140 pages they are indexing? Thank you so much for your help, I tried looking for the answer and I know this is the best place to ask!
Intermediate & Advanced SEO | | accpar0 -
Merging Two Unrelated Sites into a Third Site
We have a new client interested in possibly merging 2 sites into one under the brand of a new parent company. Here's a breakdown of the scenario..... BrandA.com sells a variety of B2B widget-services via their online store. BrandB.com sells a variety of B2B thing-a-majig products and services (some of them large in size) not sold through an online store. These are sold more consultatively via a sales team. The new parent company, BrandA-B.com is considering combining the two sites under the new brand parent company domain. The Widget-services and Thing-A-Majigs have very little similarity or purchase crossover; so just because you're interested in one doesn't make you a good candidate for the other. We feel pretty confident that we can round-up all the necessary pages and inbound links to do proper transitioning to a new, separate third domain though we're not in agreement that this is the best course of action. Currently the individual brand sites are fairly well known in their industry and each ranks fairly well for a variety of important terms though there is room for improvement and each site has good links with the exception of the new site which has considerably fewer. BrandA.com DA = 73 - 19 years old
Intermediate & Advanced SEO | | OPM
BrandB.com DA = 55 - 18 years old
BrandA-B.com DA = 40 - 1 year old Our SEO team members have opinions on what the potential outcome(s) of this would be but are wondering what the community here thinks. Will the combining of the sites cause a dilution of the topics of the two sites and hurt rankings? Will the combining of the domain authority help one set part of the business but hurt the other? What do you think? What would you do?0 -
Is my site being penalized?
I launched http://rumma.ge in February of this year. Because I'm using a domain hack (the Georgian domain), I'd really like to rank for just the word "rummage". After launching, I was steady at around page 4/5 on searches for "rummage". However since then I've tumbled out of the first 100. In fact I can't even find the site in the first 20 pages on Google for that search. Even a search for my exact homepage title text doesn't bring up the site, despite the fact that the site is still in the index. I'm wondering if one of the following could be the root cause: We have a ccTLD (.ge)--not sure about the impacts of this, but seems like it might not be the root cause because we were ranking for "rummage" when we first launched. Tried running an Adwords campaign but the site was flagged as a "bridge page" (working on getting this addressed). I'm wondering if this could have carryover impacts into natural search rankings? We've tried doing some press and built up a decent number of backlinks over the past couple of months, many of which had "rummage" in the anchor text. This was all organic, but happened over the span of a month which may be too fast? Am I being penalized? Beyond checking indexing of the site, is there a way to tell if I've been flagged for some bad behavior? Any help or thoughts would be greatly appreciated. I'm really confused by this since I feel like I've been doing things right and my rankings have been travelling downward. Thanks!! Matt
Intermediate & Advanced SEO | | minouye0 -
Search result clicks for one of my posts down significantly
I started receiving a warning warning in google webmaster tools about 2 weeks ago that said "big traffic change for top url". On reading the message i saw "Search results clicks for http://goo.gl/EyhUJ have decreased significantly". When I search google using the keyword "sore breasts" for which that post used to rank at least number 2 on google, I dont see anything. The related post http://goo.gl/vP025 is still ranking well. Can anyone give me an idea of what might have happened? I am totally at sea. Thanks.
Intermediate & Advanced SEO | | adaeze0 -
Building a mobile site.
We are building a mobile site that will be launching in another month. I’m concerned that the mobile site will start catabolizing our traditional rankings. Is there a way to keep this from happening? Should we utilize the cross domain canonical tag and point back to the traditional site URLs?
Intermediate & Advanced SEO | | SEO-Team0