How to specify an authenticated proxy for a python http connection?

What’s the best way to specify a proxy with username and password for an http connection in python?

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

This works for me:

import urllib2

proxy = urllib2.ProxyHandler({'http': 'http://
username:<a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="7f0f1e0c0c08100d1b3f0f0d1007060a0d13">[email protected]</a>:proxyport'})
auth = urllib2.HTTPBasicAuthHandler()
opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler)
urllib2.install_opener(opener)

conn = urllib2.urlopen('http://python.org')
return_str = conn.read()

Method 2

Use this:

import requests

proxies = {"http":"http://username:<a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="2555445656524a57416555574a5d5c">[email protected]</a>_ip:proxy_port"}

r = requests.get("http://www.example.com/", proxies=proxies)

print(r.content)

I think it’s much simpler than using urllib. I don’t understand why people love using urllib so much.

Method 3

Setting an environment var named http_proxy like this: http://username:[email protected]_url:port

Method 4

The best way of going through a proxy that requires authentication is using urllib2 to build a custom url opener, then using that to make all the requests you want to go through the proxy. Note in particular, you probably don’t want to embed the proxy password in the url or the python source code (unless it’s just a quick hack).

import urllib2

def get_proxy_opener(proxyurl, proxyuser, proxypass, proxyscheme="http"):
    password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
    password_mgr.add_password(None, proxyurl, proxyuser, proxypass)

    proxy_handler = urllib2.ProxyHandler({proxyscheme: proxyurl})
    proxy_auth_handler = urllib2.ProxyBasicAuthHandler(password_mgr)

    return urllib2.build_opener(proxy_handler, proxy_auth_handler)

if __name__ == "__main__":
    import sys
    if len(sys.argv) > 4:
        url_opener = get_proxy_opener(*sys.argv[1:4])
        for url in sys.argv[4:]:
            print url_opener.open(url).headers
    else:
        print "Usage:", sys.argv[0], "proxy user pass fetchurls..."

In a more complex program, you can seperate these components out as appropriate (for instance, only using one password manager for the lifetime of the application). The python documentation has more examples on how to do complex things with urllib2 that you might also find useful.

Method 5

Or if you want to install it, so that it is always used with urllib2.urlopen (so you don’t need to keep a reference to the opener around):

import urllib2
url = 'www.proxyurl.com'
username = 'user'
password = 'pass'
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
# None, with the "WithDefaultRealm" password manager means
# that the user/pass will be used for any realm (where
# there isn't a more specific match).
password_mgr.add_password(None, url, username, password)
auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
print urllib2.urlopen("http://www.example.com/folder/page.html").read()

Method 6

Here is the method use urllib

import urllib.request

# set up authentication info
authinfo = urllib.request.HTTPBasicAuthHandler()
proxy_support = urllib.request.ProxyHandler({"http" : "http://ahad-haam:3128"})

# build a new opener that adds authentication and caching FTP handlers
opener = urllib.request.build_opener(proxy_support, authinfo,
                                     urllib.request.CacheFTPHandler)

# install it
urllib.request.install_opener(opener)

f = urllib.request.urlopen('http://www.python.org/')
"""

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0