Python Requests: How to Set Custom User-Agent Headers

When making HTTP requests with Python's Requests library, setting custom User-Agent headers is crucial for web scraping and API interactions. Let's explore how to effectively manage these headers.

Understanding User-Agent Headers

A User-Agent header identifies your client application to web servers. By default, Python requests uses a generic user agent, but customizing it can help avoid blocking and improve request success rates.

Default User-Agent in Requests

Let's first see the default User-Agent that Python Requests uses:

import requests

response = requests.get('https://httpbin.org/user-agent')
print(response.json())

{
  "user-agent": "python-requests/2.28.1"
}

Setting a Custom User-Agent

There are two main ways to set a custom User-Agent. You can either use the headers parameter or create a session object. Here's how to use the headers parameter:

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}
response = requests.get('https://httpbin.org/user-agent', headers=headers)
print(response.json())

Using Session Objects

For multiple requests, using a Session object is more efficient. This approach is particularly useful when working with session management:

session = requests.Session()
session.headers.update({
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
})

response = session.get('https://httpbin.org/user-agent')
print(response.json())

Best Practices

When setting User-Agent headers, consider these important practices:

• Use realistic User-Agent strings
• Rotate User-Agents for large-scale scraping
• Respect robots.txt and website terms of service

Error Handling

Always implement proper error handling when working with custom headers. You can learn more about this in our guide about HTTP error handling.

try:
    headers = {'User-Agent': 'Custom Bot 1.0'}
    response = requests.get('https://example.com', headers=headers)
    response.raise_for_status()
except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")

Using with JSON Requests

When working with APIs that require both custom User-Agent and JSON handling, combine headers appropriately. For more details, see our guide on handling JSON responses.

headers = {
    'User-Agent': 'Custom Bot 1.0',
    'Content-Type': 'application/json'
}
response = requests.post('https://api.example.com', headers=headers, json={'key': 'value'})

Conclusion

Setting custom User-Agent headers in Python Requests is essential for reliable web scraping and API interactions. Remember to use appropriate headers and follow best practices for optimal results.