Why do old URL format are still being crawled by Rogerbot?
-
Hi,
In the early days of my blog, I used permalinks with the following format:
http://www.mysitesamp.com/2009/02/04/heidi-cortez-photo-shoot/
I then decided to change this format using .htaccess to this format:
http://www.mysitesamp.com//heidi-cortez-photo-shoot/
My question is, why do rogerbot still crawls my old URL format since these urls' no longer exists in my website or blog.
-
Thanks Alan,
That solved my problem...
-
-
Hi Alan,
After disallowing the directory in robots.txt, Rogerbot still includes the non-existing URLs. Here is a sample URL that is being reported by Rogerbot
www.lugaluda.com/2009/08/05/chase-online-banking-chase-checking-bonus/
-
If you give me the url, i can crawl it fior you if you like.
-
Thanks Alan, I really appreciate your help. Gave me an idea since all the old URLs are coming from a virtual 2009 directory, I tried to add a disallow statement for that directory in the robots.txt section. Hopefully this will help solve the problem.
I will let you know the results after rogerbot finishes recrawling my site...
Thanks Dude....
-
You need to search your site, but bots start on a page and follow the links, if the report them then they must of found them, bots like googlebot or bingbot can find them on other sites, but rogerbot is only crawling within your site.
-
How will I know if they still exists on my site? If I tried to access the specific URLs, they are no longer active.
-
The old format must still exist in your site somewhere, bots follow links from your home page though your site.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Long product urls ecommerce store
Hi we have a site in the mens fashion space who have long product urls which look like this: https://www.domain.com/catalog/product/view/id/13700/s/the-mate-tee-grey-marle-upm618g/category/120/ The site is on Magento. Are there any serious SEO negatives of having such a long product url and including irrelevant information in the url like product/view/id/13700/s/ & /category/120/ in the URL. Or are the benefits of changing them to more URL friendly product urls like: https://www.domain.com/the-mate-tee-grey-marle-upm/ Minimal? Cheers.
Intermediate & Advanced SEO | | wozniak650 -
Reuse an old juicy URL or create a new with the best practices?
I'm optimizing a site with all new URL`s, categories, titles, descriptions. All URL's will change but I've old URLs with a lot of backlinks, SEO juice. What is better for SEO with them: 1 - Change those URLs and 301 redirect traffic to the new page.
Intermediate & Advanced SEO | | Tiedemann_Anselm
2 - Keep the URL and work just on new title, description, etc. In option 1 I understand that I'll lose some SEO juice because of the redirect, but the new URL will be correct. In option 2 everything will be strong except from the URL that will make less sense than with option 1. It will not exactly match the product name, title. It`s a reuse of a strong URL.0 -
What is the fastest way disassociate an old URL with a new domain name?
We have a client with an old domain which was spammy (bad links). Until two months ago, it was forwarding to his current domain and (I believe) causing a penalty. Two months ago we transferred ownership of the spammy URL to a third party and setup an unrelated blog for Google to pick up on. Google did pick up on the URL. After two months Google Webmaster Tools is still showing 200 links from the old domain, to the new domain (from the spammy domain). Also, when you search the company name, the spammy domain still appears in the results (page two). Is there a faster way disassociate the old domain entirely from the business? I.e., just delete the domain, forward the domain to another website, etc.? If you have experience in this, I'd love to hear from you. Thanks!
Intermediate & Advanced SEO | | mgordon0 -
Backlinks from old domain
Hi, We have gone through a change of company brand name including a new domain name.
Intermediate & Advanced SEO | | Agguk
We followed google recommendations at: https://support.google.com/webmasters/answer/83106?hl=en and it seems to have worked really well, the new domain has replaced the old in the google search results. My question: Still most of our backlinks, both anchor text and links use the old brand name and domain and it´s a slow process trying to update all references. Although they get redirected fine to the new domain (also following google recommendations), I wonder if the current scenario is doing any harm, SEO wise (other than the missed visual exposure of the new brand name) ? ...since the old brand name is not present at the new site I´m thinking of including "New brand name - previously old brand name" somewhere just to provide some sort of connection to all old backlinks, would that be unnecessary? I should mention that the old brand name actually includes our most important keyword but the new brand name does not. Thanks!0 -
URL structure for SEO
Hi Mozzers, I have a site which is a combination of product pages, and news and advice pages that relate to the products. How would you approach the URL structure for this, following SEO best practice? Approach 1 Product pages:
Intermediate & Advanced SEO | | A_Q
www.website.com/product-category/product-page News and advice pages:
www.website.com/product-category/product-page/news-and-advice-story-1
www.website.com/product-category/product-page/news-and-advice-story-2
etc or Approach 2 Product pages:
www.website.com/product-category/product-page News and advice pages:
www.website.com/news/product-category/news-and advice-story-1 (with internal linking to relevant product page)
www.website.com/news/product-category/news-and advice-story-2 (with internal linking to relevant product page)
etc Or would a different approach be better?0 -
Strange URLs, how do I fix this?
I've just check Majestic and have seen around 50 links coming from one of my other sites. The links all look like this: http://www.dwww.mysite.com
Intermediate & Advanced SEO | | JohnPeters
http://www.eee.mysite.com
http://www.w.mysite.com The site these links are coming from is a html site. Any ideas whats going on or a way to get rid of these urls? When I visit the strange URLs such as http://www.dwww.mysite.com, it shows the home page of http://www.mysite.com. Is there a way to redirect anything like this back to the home page?0 -
Moving Code for Faster Crawl Through?
What are best practices for moving code into other folders to help speed up a crawling for bots? We once moved some javascript from an SEO's suggestion and the site suddenly looked like crap until we undid the changes. How do you figure our what code should be consolidated? What code do you use to indicate what has been moved and to where?
Intermediate & Advanced SEO | | siteoptimized0 -
URL or Domain length
Hi All, I am wondering if google still does give importance to the length of the domain or url. If yes then how much is the acceptable length of a domain and URL. Many Thanks!
Intermediate & Advanced SEO | | HiteshBharucha0