How to stop Search Bot from crawling through a submit button
-
On our website http://www.thefutureminders.com/, we have three form fields that have three pull downs for Month, Day, and year. This is creating duplicate pages while indexing. How do we tell the search Bot to index the page but not crawl through the submit button?
Thanks
Naren
-
Hi Dan
What is happening is this - since we have all the months [12], all the dates [31] and years[1921 through 2011] in the form fields, the robot seems to be taking these incrementally and then using the submit button. After the submit button, user is presented with a registration page. While we do want the search to index the rest of the page and the crawl through the rest of the page links we do not want it to crawl through that submit button. I hope I am making sense.
Naren
-
The advantage of blocking a page from being indexed via a meta tag is it is less likely to have unexpected consequences. I've often seen in the past cases where an incorrectly modified robots.txt file leads to a site being blocked by accident.
-
Hi
To my knowledge, you don't stop it from crawling through the button (like a nofollowed link), rather you block the robot at the page it ends up on after clicking submit.
Say the user hits submit and it takes them to mydomain.com/confirm.html On that page you'll want to add;
....if you want it to NOT index the page but follow the links on it.
or
...if you want it to NOT index and NOT follow the links on that page.
Its advised that its better to do this with the meta tag than in robots.txt.
Hopefully I've understood the question correctly!
-Dan
-
Block the pages/folders you do not wish to be indexed with robots.txt file:
User-agent: * Disallow: /folder1/ Disallow: /folder2/
OR you can add canonical tags to the other pages which are creating duplicate content.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What can i do to move my site up the search engines
Hi. my site www.in2town.co.uk is currently number five in google for the search word lifestyle magazine, sometimes it moves to four but for over a year it has not got past four. before we had to do the site from scratch due to a major problem upgrading, we were number one in the search engines and our traffic was around 30% higher than it is now. For the keyword lifestyle news, we are on the fifth page of google and would really like to improve this. I would like to know what i need to do on our home page to try and improve our rankings for these two words. the most important word for us is lifestyle news. any help in my goal to improve our rankings would be great. We have improved our design which we are still working on, and we have upgraded to a bigger dedicated server to improve the speed.
Technical SEO | | ClaireH-1848860 -
Site Recovered from hack, should I submit a reinclusion request?
Hello, The site i'm referring to is http://www.pokeronamac.com, it was hacked via something called the "WordPress Pharma Hack" http://theblawblog.wordpress.com/2012/06/21/restoring-a-pharma-hacked-wordpress-site-wp-3-4/ We restored it as far as I can tell, but if anyone can confirm this by doing a site search and not getting redirected it would be appreciated. You will see that some search results still show up as spam, but when I click on them, they 404. I want to know If I should submit a reinclusion request, I wasn't notified by WMT of malaware, so I want to know the SOP here. Thanks Zach
Technical SEO | | Zachary_Russell1 -
Need advice on search listings and link building
Search results on my keyword (engraved wedding glasses) produces several pages of linked domains. (My domain is giftthings.net) Some are good. And admittedly, some are not so good. My question then is simply, why does seomoz link analysis show such a small number of links? And the second part of my question is, "Is there some sort of "magic number", some sort of thresh hold that triggers Google's interest? With a link list that is small but growing, am I missing something in my concern that I'm not moving up in the search listings? I've written a few articles, continuing my work on link building but I remain buried in the search results.
Technical SEO | | AhmadS1 -
Site Crawl
I was wondering if there was a way to use SEOmoz's tool to quickly and easily find all the URLs on you site and not just the ones with errors. The site that I am working on does not have a site map. What I am trying to do is find all the URLs along with their titles and description tags. Thank you very much for your help
Technical SEO | | pakevin0 -
Pagination and SEO: How do I fix it during search parameters?
Today, I have watched very interesting video on YouTube about Pagination and SEO. I have implemented pagination with rel="next" and rel="prev" on my paginated page. You can get more idea by visit following pages. www.vistastores.com/patio-umbrellas www.vistastores.com/patio-umbrellas?p=2 www.vistastores.com/patio-umbrellas?p=3 I have added NOINDEX FOLLOW attribute to page 2, page 3 and so on. There is simple question from my side. Can I remove NOINDEX FOLLOW attribute from paginated page or not? I have big confusion & issues when paginated URLs contain search parameters. You can get more idea by visiting following URLs. http://www.vistastores.com/patio-umbrellas?dir=asc&order=name&p=2 http://www.vistastores.com/patio-umbrellas?dir=asc&order=name&p=3 What is best suggestion for this kind of pages?
Technical SEO | | CommercePundit0 -
Page crawling is only seeing a portion of the pages. Any Advice?
last couple of page crawls have returned 14 out of 35 pages. Is there any suggestions I can take.
Technical SEO | | cubetech0 -
How to do a no follow on site search
We have a site search that is causing a huge amount of errors as the SEOmoz crawler is showing these as duplicate content. Our first thought was to do a no-follow on the site-search directory, but we realized that the site search is /site-search.aspx and URl strings appear at the end for hundreds of pages. How dow we/how can we no-follow an undetermined amount of URL strings?
Technical SEO | | Apptixweb0 -
Page that has no link is being crawled
http://www.povada.com/category/filters/metal:Silver/nstart/1/start/1.htm I have no idea how the above page was even found by google but it seems that it is being crawled and Im not sure where its being found from. Can anyone offer a solution?
Technical SEO | | 13375auc30