Is there a way to crawl some ASP.NET pages that uses doPostBack as events calling?
Example:
Page1.aspx:
Contains 1 LinkButton that redirects to Page2.aspx
Code-behind for LinkButton Click event:
Response.Redirect("Page2.aspx")
In client side this code is generated on click event:
doPostBack(...
Is it possible crawl pages using only HttpWebRequest?
I know that use Response.Redirect is not a good idea in this case, but I don’t have choice.
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
Yes, it’s possible if the code follows a well predictable pattern. You would have to gather the form data from the page and simulate what the doPostBack function does (putting some values in some hidden fields), and send a POST request to the server. What you get back would be a redirection page, so you would have to parse that to get the url of the target page.
If you mean if search engines like Google will crawl the pages, then that is very unlikely. They might attempt to follow some common patterns of posting and script linking, but generally you need to use proper links between the pages to be sure that they are crawlable.
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0