scraping data from google search engine by using python requests BeautifulSoup
- Last modified: 22 January 2020
- Category: python tutorial
In this article, we gonna make a simple script that scraping data from google search engine by using requests and BeautifulSoup libraries. so in this example, we'll enter our search query and getting the title, URL, and description of the search result
Code:
import requests
from bs4 import BeautifulSoup
query = input('Enter Your Query: ') #search query
lang = input('Enter language ex:[en,fr,ar,jp,cn...]: ') #search language
# requests
url = 'https://www.google.com/search?hl={}&q={}&start=3i&num=10&ie=UTF-8'.format(lang, query) #url
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0'}#headers
source=requests.get(url, headers=headers).text # url source
# BeautifulSoup
soup = BeautifulSoup(source, 'lxml')
search_div = soup.find_all(class_='rc') # find all divs tha contains search result
for result in search_div: # loop result list
print('Title: %s'%result.h3.string) #geting h3
print('\n')
print('Url: %s'%result.a.get('href')) #geting a.href
print('\n')
print('Description: %s'%result.find(class_='st').text) #description
print('\n###############\n')
Output
Enter Your Query: facebook Enter language ex:[en,fr,ar,jp,cn...]: en Title: Facebook - Log In or Sign Up Url: https://www.facebook.com/ Description: Create an account or log into Facebook. Connect with friends, family and other people you know. Share photos and videos, send messages and get updates. ############### Title: About Facebook Url: https://about.fb.com/ Description: The Facebook company builds technologies that give people the power to connect with friends and family, find communities and grow businesses. ############### Title: Newsroom | About Facebook Url: https://about.fb.com/news/ Description: More Than $50 Million Raised for Australia Wildfire Relief Efforts. ... Expanded Transparency and More Controls for Political Ads. ... Food for Thought: AI Researchers Develop New Way to Reverse Engineer Recipes From Photos. ############### Title: Facebook - Apps on Google Play Url: https://play.google.com/store/apps/details?id=com.facebook.katana&hl=en_US Description: Keeping up with friends is faster and easier than ever. Share updates and photos, engage with friends and Pages, and stay connected to communities important ... ############### Title: Facebook - Wikipedia Url: https://en.wikipedia.org/wiki/Facebook Description: Facebook (FB) is an American online social media and social networking service based in Menlo Park, California and a flagship service of the namesake ... ############### Title: Facebook - Forbes Url: https://www.forbes.com/companies/facebook/ Description: Facebook, Inc. is a social networking company, which allows people to communicate with their family, friends, and coworkers. Its services include timeline, news ... ###############
As you can see I have searched for facebook and I've got the result.
English today is not an art to be mastered it's just a tool to use to get a result