Disallowing https URLs
-
It there a problem disallowing all https URLs to be indexed in order to avoid duplication?
This is the article recommending this practice -
http://blog.leonardchallis.com/seo/serve-a-different-robots-txt-for-https/
Thanks!
-
no there is nothing wrong at all. Generaly you would either force pages to be either one or the other. There is not too many senarios where you would want to allow both.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Woocommerce URL Structure Issue
Hi everyone ! To put you in context, I am doing an audit on an E-Commerce site selling auto parts with WooCommerce. I have some concerns regarding the url structure and here's why: Product category page url: /auto/drivetrain/cv-axle-shaft-assembly/
Technical SEO | | alexrbrg
Product page url included in the product category page: /product/acura-integra-cv-axle-shaft-90-01-honda-civic/ The way I see my situation is that the product page is considered by Google as an intern link and not as a page included in in the subfolder of the category page. 1. Am I right?
2. If yes, is there a solution to fix the issue with woocommerce to improve the category page ranking ? Thanks y'all !0 -
Sitemaps, 404s and URL structure
Hi All! I recently acquired a client and noticed in Search Console over 1300 404s, all starting around late October this year. What's strange is that I can access the pages that are 404ing by cutting and pasting the URLs and via inbound links from other sites. I suspect the issue might have something to do with Sitemaps. The site has 5 Sitemaps, generated by the Yoast plugin. 2 Sitemaps seem to be working (pages being indexed), 3 Sitemaps seem to be not working (pages have warnings, errors and nothing shows up as indexed). The pages listed in the 3 broken sitemaps seem to be the same pages giving 404 errors. I'm wondering if auto URL structure might be the culprit here. For example, one sitemap that works is called newsletter-sitemap.xml, all the URLs listed follow the structure: http://example.com/newsletter/post-title Whereas, one sitemap that doesn't work is called culture-event-sitemap.xml. Here the URLs underneath follow the structure http://example.com/post-title. Could it be that these URLs are not being crawled / found because they don't follow the structure http://example.com/culture-event/post-title? If not, any other ideas? Thank you for reading this long post and helping out a relatively new SEO!
Technical SEO | | DanielFeldman0 -
Advice on strange URL problem
I'm considering doing some pro bono work for a local non-profit and upon initial review they have a number of serious issues but there is one in particular I'd like to check my thinking on. The developer who set up the site some years ago implemented a javascript redirect on their root domain so that it redirects to: http://domain.com/wordpress This is wrong for all kinds of reasons and I want to recommend eliminating this redirect and getting rid of the 'wordpress' part of the path altogether. However, the site is quite established with good PR and they would take a hit by changing the path. I'd do 301 redirects to the new URLs that would not have 'wordpress' in the path in addition to other remediation. My question - is my thinking here good? It's worth it, right? The other option is just get rid of the weird redirect and keep 'wordpress' in the path but this seems unacceptable to me. Any opinions?
Technical SEO | | friendlymachine0 -
Is there actual risk to having multiple URLs that frame in main url? Or is it just bad form and waste of money?
Client has many urls that just frame in the main site. It seems like a total waste of money, but if they are frames, is there an actual risk?
Technical SEO | | gravityseo0 -
Canonical URL
In our campaign, I see this notices Tag value
Technical SEO | | shebinhassan
florahospitality.com/ar/careers.aspx Description
Using rel=canonical suggests to search engines which URL should be seen as canonical. What does it mean? Because If I try to view the source code of our site, it clearly gives me the canonical url.0 -
How do you disallow HTTPS?
I currently have a site (startuploans.org) that runs everything as http, recently we decided to start an online application to process loan apps. Now, for one certain section we configured ssl to work (https://www.startuploans.org/secure/). If I go to the HTTPS url for any of my other pages they show up...I was going to just 301 everything from https but because it is in a subdirectiory I can't... Also, canonical URL's won't work either because it's a totally different system and the pages are generated in an odd manor. It's really just 1 page that needs to be disallowed.. Is there any way to disallow all HTTPS requests from robots.txt while keeping all the HTTP requests working as normal?
Technical SEO | | WebsiteConsultants0 -
URL's for news content
We have made modifications to the URL structure for a particular client who publishes news articles in various niche industries. In line with SEO best practice we removed the article ID from the URL - an example is below: http://www.website.com/news/123/news-article-title
Technical SEO | | mccormackmorrison
http://www.website.com/news/read/news-article-title Since this has been done we have noticed a decline in traffic volumes (we have not as yet assessed the impact on number of pages indexed). Google have suggested that we need to include unique numerical IDs in the URL somewhere to aid spidering. Firstly, is this policy for news submissions? Secondly (if the previous answer is yes), is this to overcome the obvious issue with the velocity and trend based nature of news submissions resulting in false duplicate URL/ title tag violations? Thirdly, do you have any advice on the way to go? Thanks P.S. One final one (you can count this as two question credits if required), is it possible to check the volume of pages indexed at various points in the past i.e. if you think that the number of pages being indexed may have declined, is there any way of confirming this after the event? Thanks again! Neil0