scraping data from google search engine by using python requests BeautifulSoup

• Last modified: 22 January 2020
• Category: python tutorial

In this article, we gonna make a simple script that scraping data from google search engine by using requests and BeautifulSoup libraries. so in this example, we'll enter our search query and getting the title, URL, and description of the search result

Code:

 
import requests
from bs4 import BeautifulSoup



query = input('Enter Your Query: ') #search query
lang = input('Enter language ex:[en,fr,ar,jp,cn...]: ') #search language

# requests
url = 'https://www.google.com/search?hl={}&q={}&start=3i&num=10&ie=UTF-8'.format(lang, query) #url
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0'}#headers
source=requests.get(url, headers=headers).text # url source

# BeautifulSoup
soup = BeautifulSoup(source, 'lxml')
search_div = soup.find_all(class_='rc') # find all divs tha contains search result
for result in search_div: # loop result list
    print('Title: %s'%result.h3.string) #geting h3 
    print('\n')
    print('Url: %s'%result.a.get('href')) #geting a.href 
    print('\n')
    print('Description: %s'%result.find(class_='st').text) #description
    print('\n###############\n')

Output

Enter Your Query: facebook
Enter language ex:[en,fr,ar,jp,cn...]: en




Title: Facebook - Log In or Sign Up

Url: https://www.facebook.com/

Description: Create an account or log into Facebook. Connect with friends, family and other people you know. Share photos and videos, send messages and get updates.

###############

Title: About Facebook

Url: https://about.fb.com/

Description: The Facebook company builds technologies that give people the power to connect with friends and family, find communities and grow businesses.

###############

Title: Newsroom | About Facebook

Url: https://about.fb.com/news/

Description: More Than $50 Million Raised for Australia Wildfire Relief Efforts. ... Expanded Transparency and More Controls for Political Ads. ... Food for Thought: AI Researchers Develop New Way to Reverse Engineer Recipes From Photos.

###############

Title: Facebook - Apps on Google Play

Url: https://play.google.com/store/apps/details?id=com.facebook.katana&hl=en_US

Description: Keeping up with friends is faster and easier than ever. Share updates and photos, engage with friends and Pages, and stay connected to communities important ...

###############

Title: Facebook - Wikipedia

Url: https://en.wikipedia.org/wiki/Facebook

Description: Facebook (FB) is an American online social media and social networking service based in Menlo Park, California and a flagship service of the namesake ...

###############

Title: Facebook - Forbes

Url: https://www.forbes.com/companies/facebook/

Description: Facebook, Inc. is a social networking company, which allows people to communicate with their family, friends, and coworkers. Its services include timeline, news ...

###############

 

As you can see I have searched for facebook and I've got the result.