Does RogerBot read URL wildcards in robots.txt
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to 301 trailing URLs to new domain home page - wildcard?
How would I add a redirect rule so all old domain URLs redirect to a new domain? All the old pages no longer exist on a new website. The domains have been through several CMS platforms, so it would be unnecessary to recreate them. Problem is, they're indexed in search engines from the past 10 years, so it's causing a lot of 404s. Example: search "NARI Tampa Bay" and you'll find 2 old domains: nari-tampabay.com & nari-tampabay.org. The new domain is naritb.org Those 2 old domains are now pointed to the same nameservers as the new and listed as parked domains. Here's the current rules in htaccess: <code>RewriteEngine On RewriteCond %{HTTP_HOST} ^nari-tampabay.org [NC,OR] RewriteCond %{HTTP_HOST} ^www.nari-tampabay.org [NC] RewriteRule ^(.*)$ https://www.naritb.org/$1 [L,R=301] RewriteEngine On RewriteCond %{HTTP_HOST} ^nari-tampabay.com [NC,OR] RewriteCond %{HTTP_HOST} ^www.nari-tampabay.com [NC] RewriteRule ^(.*)$ https://www.naritb.org/$1 [L,R=301]</code>
Technical SEO | | CartoMark0 -
URL Parameters to Ignore
Hi Mozers, **We have a glossary of terms made up of a main page that lists out ALL of the terms, and then individual pages per alphabet letter that limit the results to that specific alphabet letter. These pages look like this: ** https://www.XXXX.XXX/publications/dictionaries/XXX-terms?expand=A https://www.XXXX.XXX/publications/dictionaries/XXX-terms?expand=B https://www.XXXX.XXX/publications/dictionaries/XXX-terms?expand=C https://www.XXXX.XXX/publications/dictionaries/XXX-terms?expand=D etc. If I'd like Google to remove all of these "expand=" pages from the index, such that only the main page is indexed, what is the exact parameter that I should ask Google to ignore in Search Console? "expand=" ? Just want to make sure! Thanks for the help!!!
Technical SEO | | yaelslater1 -
Robots txt. in page with 301 redirect
We currently have a a series of help pages that we would like to disallow from our robots txt. The thing is that these help pages are located in our old website, which now has a 301 redirect to current site. Which is the proper way to go around? 1- Add the pages we want to disallow to the robots.txt of the new website? 2- Break the redirect momentarily and add the pages to the robots.txt of the old one? Thanks
Technical SEO | | Kilgray0 -
Which URL structure is better?
Quick question - Have a real estate site focused on "apartments", but apartments in not part of my company name. That being said, should which of the following URL structures should I use? http://website.com/city/neighborhood/property-name OR http://website.com/city-apartments/neighborhood/property-name
Technical SEO | | ChaseH0 -
Duplicate content error - same URL
Hi, One of my sites is reporting a duplicate content and page title error. But it is the same page? And the home page at that. The only difference in the error report is a trailing slash. www.{mysite}.co.uk www.{mysite}.co.uk/ Is this an easy htaccess fix? Many thanks TT
Technical SEO | | TheTub1 -
URL paths and keywords
I'm recommending some on-page optimization for a home builder building in several new home communities. The site has been through some changes in the past few months and we're almost starting over. The current URL structure is http://homebuilder.com/oakwood/features where homebuilder = builder name Oakwood Estates= name of community features = one of several sub-paths including site plan, elevations, floor plans, etc. The most attainable keyword phrases include the word 'home' and 'townname' I want to change the URL path to: http://homebuilder.com/oakwood-estates-townname-homes/features Is there any problem with doing this? It just seems to make a lot of sense. Any input would be appreciated.
Technical SEO | | mikescotty0 -
301 an old URL with a ? in the URL?
I am redoing a site and the URL's are changing structure. The client's site was in magento and in the store they would get two URLs, for example: /store/categoryname/productname and /store/categoryname/productname?SID=dslkajsfdoiu947598whouieht983hg98 Do I have to 301 redirect both of these URL's to their new counterpart? Both go to the same content but magento seemed to add these SIDs into the navigation and Google has both versions in the index.
Technical SEO | | DanDeceuster0 -
How do I use the Robots.txt "disallow" command properly for folders I don't want indexed?
Today's sitemap webinar made me think about the disallow feature, seems opposite of sitemaps, but it also seems both are kind of ignored in varying ways by the engines. I don't need help semantically, I got that part. I just can't seem to find a contemporary answer about what should be blocked using the robots.txt file. For example, I have folders containing site comps for clients that I really don't want showing up in the SERPS. Is it better to not have these folders on the domain at all? There are also security issues I've heard of that make sense, simply look at a site's robots file to see what they are hiding. It makes it easier to hunt for files when they know the directory the files are contained in. Do I concern myself with this? Another example is a folder I have for my xml sitemap generator. I imagine google isn't going to try to index this or count it as content, so do I need to add folders like this to the disallow list?
Technical SEO | | SpringMountain0