I’ve already seen this question about scraping ajax, but python isn’t mentioned there. I considered using scrapy, i believe they have some docs on that subject, but as you can see the website is down. So i don’t know what to do. I want to do the following:
I am writing a scraper that downloads all the image files from a HTML page and saves them to a specific folder. all the images are the part of the HTML page.
I’d like to use Python to scrape the contents of the “Were you looking for these authors:” box on web pages like this one: http://academic.research.microsoft.com/Search?query=lander
if a page has <div class="class1"> and <p class="class1">, then soup.findAll(True, 'class1') will find them both.
I want to grab the value of a hidden input field in HTML.
I am trying to use selenium from python to scrape some dynamics pages with javascript. However, I cannot call firefox after I followed the instruction of selenium on the pypi page(http://pypi.python.org/pypi/selenium). I installed firefox on AWS ubuntu 12.04. The error message I got is:
I have written many scrapers but I am not really sure how to handle infinite scrollers. These days most website etc, Facebook, Pinterest has infinite scrollers.