Robots.txt help
-
Hi Moz Community,
Google is indexing some developer pages from a previous website where I currently work:
ddcblog.dev.examplewebsite.com/categories/sub-categories
Was wondering how I include these in a robots.txt file so they no longer appear on Google. Can I do it under our homepage GWT account or do I have to have a separate account set up for these URL types?
As always, your expertise is greatly appreciated,
-Reed
-
The robots.txt would allow the OP to go back into GWT and request removal of the dev site from the index. Password protecting a dev site is usually a pretty good idea, too.
-
Can you not just add a htaccess password to the directory to keep the dev site up, but keep bots out?
-
You'll want a separate account for that subdomain, and also put the robots.txt excluding that subdomain in that subdomain itself.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt Help
I need help to create robots.txt file. Please let me know what to add in the file. any real example or working example.?
Intermediate & Advanced SEO | | Michael.Leonard0 -
Does having shorter URLs help with rankings?
Hello here.I own an e-commerce website (virtualsheetmusic.com), and some of our most important category pages have pretty long URLs. Here is an example: http://www.virtualsheetmusic.com/downloads/Indici/Violin.html I am evaluating the possibility to shorten URLs like the above to something like: http://www.virtualsheetmusic.com/violin/ But since it is going to pretty hard and time consuming (considering the custom system we have in place on that site), I am trying to find out if it really matters and worth doing it from a SEO stand point. I am aware that from a user prospective shorter URLs are preferable, and we plan to pursue a better URL architecture on our website in the near future just for that, but this question, at the moment, should be strictly related to SEO. Any thoughts on this topic are very welcome!
Intermediate & Advanced SEO | | fablau0 -
Redirect help
I work for a company that has a domain that is something like www.neat-stuff.com but most people just use www.neatstuff.com without the dash. The redirect for the homepage works fine. We recently launched a new site and if you use www.neat-stuff.com/category it redirects from the old site to the new site just fine. However if you use www.neatstuff.com/category it does not properly redirect to www.neat-stuff.com/category. How do I fix that?
Intermediate & Advanced SEO | | EcommerceSite0 -
DMOZ help
So yesterday I got a DMOZ editor account. I would like to know if Google indexes the editor profile pages on DMOZ: http://www.dmoz.org/public/profile?editor= here are some examples http://www.dmoz.org/public/profile?editor=thehelper http://www.dmoz.org/public/profile?editor=raph3988 http://www.dmoz.org/public/profile?editor=skasselea I would like to know if it is worth while to build up this page so it will pass link juice. And can anyone tell me how frequently Google crawls for new editors (if that's possible?)
Intermediate & Advanced SEO | | raph39880 -
Do I use a .org or .co.uk domain to help UK rankings?
Hi Guys, I own to good domains one with a .ORG and the other .CO.UK Can anyone advise which one is best to use to help UK rankings? Or does it not make much difference?? Thanks guys Gareth
Intermediate & Advanced SEO | | GAZ090 -
Why specify robots instead of googlebot for a Panda affected site?
Daniweb is the poster child for sites that have recovered from Panda. I know one strategy she mentioned was de-indexing all of her tagged content, fo rexample: http://www.daniweb.com/tags/database Why do you think more Panda affected sites specifying 'googlebot' rather than 'robots' to capture traffic from Bing & Yahoo?
Intermediate & Advanced SEO | | nicole.healthline0 -
Reciprocal Links and nofollow/noindex/robots.txt
Hypothetical Situations: You get a guest post on another blog and it offers a great link back to your website. You want to tell your readers about it, but linking the post will turn that link into a reciprocal link instead of a one way link, which presumably has more value. Should you nofollow your link to the guest post? My intuition here, and the answer that I expect, is that if it's good for users, the link belongs there, and as such there is no trouble with linking to the post. Is this the right way to think about it? Would grey hats agree? You're working for a small local business and you want to explore some reciprocal link opportunities with other companies in your niche using a "links" page you created on your domain. You decide to get sneaky and either noindex your links page, block the links page with robots.txt, or nofollow the links on the page. What is the best practice? My intuition here, and the answer that I expect, is that this would be a sneaky practice, and could lead to bad blood with the people you're exchanging links with. Would these tactics even be effective in turning a reciprocal link into a one-way link if you could overlook the potential immorality of the practice? Would grey hats agree?
Intermediate & Advanced SEO | | AnthonyMangia0 -
Old pages still crawled by SE returning 404s. Better to put 301 or block with robots.txt ?
Hello guys, A client of ours has thousand of pages returning 404 visibile on googl webmaster tools. These are all old pages which don't exist anymore but Google keeps on detecting them. These pages belong to sections of the site which don't exist anymore. They are not linked externally and didn't provide much value even when they existed What do u suggest us to do: (a) do nothing (b) redirect all these URL/folders to the homepage through a 301 (c) block these pages through the robots.txt. Are we inappropriately using part of the crawling budget set by Search Engines by not doing anything ? thx
Intermediate & Advanced SEO | | H-FARM0