In today's digital landscape, the use of proxies has become an essential technique for various tasks, from web scraping and data collection to maintaining privacy and security. Python, with its extensive libraries and user-friendly syntax, offers a powerful toolkit for leveraging proxies effectively. In this guide, we'll walk you through the fundamentals of using proxies with Python, covering everything from understanding proxies to implementing them in your projects.

When you're working with Python web apps, you might run into situations where leveraging a proxy becomes a necessity for sending requests to external resources. Essentially, a proxy takes on the role of a middleman, inserting itself between your application and the vast online realm. In the following post, we're diving into the world of proxy utilization in Python, shedding light on the 'how' and delving into some use cases.

Here are quick links to the different methods of using Python with a proxy for you convenience.

  1. Using Requests
  2. Using Requests Session
  3. With environment variables

What is a Proxy?

As a quick reminder if you don't already know, a proxy is a server that sits between your application and the internet. When you make a request through a proxy, the request is sent to the proxy server first. The proxy server then forwards the request to the destination server on your behalf. The response from the destination server is sent back to the proxy server, which then forwards the response back to your application. Mobile proxies and DC proxies work in the same way, the difference is whether or not your proxy server has a mobile IP address or a DC IP address.

There are several reasons why you might want to use a proxy or a mobile proxy. For example, you may find yourself wanting to:

  • Access resources that are blocked by a firewall or network restrictions
  • Hide your IP address or location
  • Improve performance by caching requests
  • Looking to scrape the web
  • Looking to test websites as users in different countries

Proxy Details

Okay great, now that we know what a proxy is, let's get some proxy details of our own set up!

If you don't already have a proxy, signup for an account and purchase one of our mobile proxy offerings!

For the sake of this post, you will need to have a proxy hostname, port, username and password (unless you are authenticating via an IP whitelist

Using a Proxy in Python

Thankfully Python provides a library called requests for making HTTP requests. To use a proxy with requests, you need to pass the proxies argument to your request call.

Here's a really simple example of how you can use a proxy or mobile proxy with requests: 


import requests

proxy = {
    'http': 'http://your-proxy-address:port',
    'https': 'https://your-proxy-address:port'
}

response = requests.get('https://example.com', proxies=proxy)
print(response.text)

In the example above, we're calling requests.get and passing the proxies parameter. This parameter takes a dictionary of proxy definitions, where the key is the protocol and the value is the proxy address. If you need to authenticate your proxy you can use the URL sytnax: http://user:pass@host:port

If you instead want to use a requests session object to make requests, you can setup proxies like so: 


import requests

s = requests.Session()

s.proxies = {
    'http': 'http://your-proxy-address:port',
    'https': 'https://your-proxy-address:port'
}

response = s.get('https://example.com')

print(response.text)

This is very similar to the previous example, but instead we are assigning our proxies to the session object, not the request call itself. Again authentication will work in the same way here.

Using environment variables

Alternatively we can use a proxy in Python through environment variables, for example you can set the following variable: 


    export HTTP_PROXY='http://user:pass@proxy_host:proxy_port'
    export HTTPS_PROXY='http://user:pass@proxy_host:proxy_port'

If you set the above, and use requests, it will use the proxy by default for all requests that you make. This means you no longer need to pass any parameters to a request call or a request session! Super handy stuff.

Conclusion

Using a proxy in Python can be a powerful tool for accessing external resources and improving performance.

By understanding how to use the requests library and passing in the appropriate proxy server information, you can easily make requests through a proxy in your Python applications.

Sign up for an account and join our grid today to keep you ahead of your competitors.