How to use Beautifulsoup with Django
In this tutorial, we'll use Django and Beautifulsoup to build a simple form that extracts the title tag content from any web page. To create the app, we need:
- Django
- Beautifulsoup
- Requests
If you are ready, let's get started.
Beautifulsoup with Django FBV
FBV or Function Based View is a view that is set by function. The following example will use Beautifulsoup in FBV to extract the title content from any website page.
Let's create the function in views.py:
from django.shortcuts import render
import requests # 👉️ Requests module
from bs4 import BeautifulSoup # 👉️ BeautifulSoup module
def get_title(request):
if request.method == "POST":
url = request.POST.get('url') # 👉️ Get URL from Input
req = requests.get(url) # 👉️ Make request
web_s = req.text # 👉️ Get The content of the request
soup = BeautifulSoup(web_s, "html.parser") # 👉️ Parse
title = soup.title.string # 👉️ Get Value of Title tag
return render(request, "fbv_get_title.html", {"title":title})
return render(request, "fbv_get_title.html")
As you can see in the code, we've used:
- requests.get() method to make a get request.
- .text property to get HTML page source.
- beautifulsoup() to parse the HTML
- .string to get the title tag
- .string to get the content of the title tag
Finally, We've returned data in fbv_get_title.html.
Now, we need to add the path in urls.py:
path('fbv-get-title/', get_title, name='fbv_get_title')In fbv_get_title.html, let's write our form:
<!DOCTYPE html>
<html>
<head>
<title>Django FBV With Beautifulsoup</title>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">
</head>
<style>
p {
font-size: 20px
}
h1 {
text-align: center;
}
</style>
<body>
<div class="container">
<h1 class="mt-5 mb-5">Get Title From Any Website</h1>
<!-- Form -->
<form method="POST" action="{% url 'fbv_get_title' %}"> {% csrf_token %} <div class="mt-4">
<div class="col">
<input type="url" name="url" class="form-control" placeholder="Enter URL" rows="5" required="required">
</textarea>
</div>
<button type="submit" class="btn btn-primary mt-2">Go</button>
</form>
<!-- Result -->
<div class="result mt-4">
<h2>{{title}}</h2>
</div>
</div>
</body>
</html>Now let's try our form and see the result.
Beautifulsoup with Django CBV
CBV or Class Based Views is a view set by a class object. You can also use Beautifulsoup with this type of view.
In the following example, we'll do the same thing we did with TemplateView.
view.py:
from django.shortcuts import render
from django.views.generic import TemplateView
import requests # 👉️ Requests module
from bs4 import BeautifulSoup # 👉️ BeautifulSoup module
class GetTitle(TemplateView):
template_name = "index.html" # 👉️ Template name
def post(self, request, *args, **kwargs):
if request.method == "POST":
url = request.POST.get('url')
req = requests.get(url)
web_s = req.text
soup = BeautifulSoup(web_s, "html.parser")
title = soup.title.string
return render(request, "cbv_get_title.html", {"title":title})
return render(request, "cbv_get_title.html")urls.py:
path('cbs-get-title/', GetTitle.as_view(), name='cbs_get_title')fbv_get_title.html:
<!DOCTYPE html>
<html>
<head>
<title>Django CBV With Beautifulsoup</title>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">
</head>
<style>
p {
font-size: 20px
}
h1 {
text-align: center;
}
</style>
<body>
<div class="container">
<h1 class="mt-5 mb-5">Get Title From Any Website</h1>
<!-- Form -->
<form method="POST" action="{% url 'cbs_get_title' %}"> {% csrf_token %} <div class="mt-4">
<div class="col">
<input type="url" name="url" class="form-control" placeholder="Enter URL" rows="5" required="required">
</textarea>
</div>
<button type="submit" class="btn btn-primary mt-2">Go</button>
</form>
<!-- Result -->
<div class="result mt-4">
<h2>{{title}}</h2>
</div>
</div>
</body>
</html>Conclusion
In this tutorial, we've learned how to use Django and Beautifulsoup by using two types of views Function-Based-Views And Function-Based-Views.
