Python urllib2 Progress Hook

I am trying to create a download progress bar in python using the urllib2 http client. I’ve looked through the API (and on google) and it seems that urllib2 does not allow you to register progress hooks. However the older deprecated urllib does have this functionality.

Does anyone know how to create a progress bar or reporting hook using urllib2? Or are there some other hacks to get similar functionality?

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

Here’s a fully working example that builds on Anurag’s approach of chunking in a response. My version allows you to set the the chunk size, and attach an arbitrary reporting function:

import urllib2, sys

def chunk_report(bytes_so_far, chunk_size, total_size):
   percent = float(bytes_so_far) / total_size
   percent = round(percent*100, 2)
   sys.stdout.write("Downloaded %d of %d bytes (%0.2f%%)r" % 
       (bytes_so_far, total_size, percent))

   if bytes_so_far >= total_size:
      sys.stdout.write('n')

def chunk_read(response, chunk_size=8192, report_hook=None):
   total_size = response.info().getheader('Content-Length').strip()
   total_size = int(total_size)
   bytes_so_far = 0

   while 1:
      chunk = response.read(chunk_size)
      bytes_so_far += len(chunk)

      if not chunk:
         break

      if report_hook:
         report_hook(bytes_so_far, chunk_size, total_size)

   return bytes_so_far

if __name__ == '__main__':
   response = urllib2.urlopen('http://www.ebay.com');
   chunk_read(response, report_hook=chunk_report)

Method 2

Why not just read data in chunks and do whatever you want to do in between, e.g. run in a thread, hook into a UI, etc etc

import urllib2

urlfile = urllib2.urlopen("http://www.google.com")

data_list = []
chunk = 4096
while 1:
    data = urlfile.read(chunk)
    if not data:
        print "done."
        break
    data_list.append(data)
    print "Read %s bytes"%len(data)

output:

Read 4096 bytes
Read 3113 bytes
done.

Method 3

urlgrabber has built-in support for progress notification.

Method 4

Simplified version:

temp_filename = "/tmp/" + file_url.split('/')[-1]
f = open(temp_filename, 'wb')
remote_file = urllib2.urlopen(file_url)

try:
    total_size = remote_file.info().getheader('Content-Length').strip()
    header = True
except AttributeError:
    header = False # a response doesn't always include the "Content-Length" header

if header:
    total_size = int(total_size)

bytes_so_far = 0

while True:
    buffer = remote_file.read(8192)
    if not buffer:
        sys.stdout.write('n')
        break

    bytes_so_far += len(buffer)
    f.write(buffer)
    if not header:
        total_size = bytes_so_far # unknown size

    percent = float(bytes_so_far) / total_size
    percent = round(percent*100, 2)
    sys.stdout.write("Downloaded %d of %d bytes (%0.2f%%)r" % (bytes_so_far, total_size, percent))

Method 5

Minor modification to Triptych’s response to allow for actually writing out the file (python3):

from urllib.request import urlopen

def chunk_report(bytes_so_far, chunk_size, total_size):
    percent = float(bytes_so_far) / total_size
    percent = round(percent*100, 2)
    sys.stdout.write("Downloaded %d of %d bytes (%0.2f%%)r" %
                     (bytes_so_far, total_size, percent))

    if bytes_so_far >= total_size:
        sys.stdout.write('n')


def chunk_read(response, chunk_size=8192, report_hook=None):
    total_size = response.info().get("Content-Length").strip()
    total_size = int(total_size)
    bytes_so_far = 0
    data = b""

    while 1:
        chunk = response.read(chunk_size)
        bytes_so_far += len(chunk)

        if not chunk:
            break

        if report_hook:
            report_hook(bytes_so_far, chunk_size, total_size)

        data += chunk

    return data

Usage:

with open(out_path, "wb") as f:
    response = urlopen(filepath)
    data_read = chunk_read(response, report_hook=chunk_report)

    f.write(data_read)

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0