screen-scraping

Parse a .Net Page with Postbacks

by

I need to read data from an online database that’s displayed using an aspx page from the UN. I’ve done HTML parsing before, but it was always by manipulating query-string values. In this case, the site uses asp.net postbacks. So, you click on a value in box one, then box two shows, click on a value in box 2 and click a button to get your results.

Export ASPX to HTML

by

We’re building a CMS. The site will be built and managed by the users in aspx pages, but we would like to create a static site of HTML’s.
The way we’re doing it now is with code I found here that overloads the Render method in the Aspx Page and writes the HTML string to a file. This works fine for a single page, but the thing with our CMS is that we want to automatically create a few HTML pages for a site right from the start, even before the creator has edited anything in the system.
Does anyone know of any way to do this?

scrape html generated by javascript with python

by

I need to scrape a site with python. I obtain the source html code with the urlib module, but I need to scrape also some html code that is generated by a javascript function (which is included in the html source). What this functions does “in” the site is that when you press a button it outputs some html code. How can I “press” this button with python code? Can scrapy help me? I captured the POST request with firebug but when I try to pass it on the url I get a 403 error. Any suggestions?

Headless Browser for Python (Javascript support REQUIRED!)

by

I need a headless browser which is fairly easy to use (I am still fairly new to Python and programming in general) which will allow me to navigate to a page, log into a form that requires Javascript, and then scrape the resulting web page by searching for results matching certain criteria, clicking check boxes, … Read more