Last modified: Jan 10, 2023 By Alexander Williams

How to use Beautifulsoup with Django

In this tutorial, we'll use Django and Beautifulsoup to build a simple form that extracts the title tag content from any web page. To create the app, we need:

  • Django
  • Beautifulsoup
  • Requests

If you are ready, let's get started.

Beautifulsoup with Django FBV

FBV or Function Based View is a view that is set by function. The following example will use Beautifulsoup in FBV to extract the title content from any website page.

Let's create the function in views.py:

from django.shortcuts import render
import requests # 👉️ Requests module
from bs4 import BeautifulSoup # 👉️ BeautifulSoup module

def get_title(request):
   
    if request.method == "POST":
        url = request.POST.get('url') # 👉️ Get URL from Input
        req = requests.get(url) # 👉️ Make request
        web_s = req.text # 👉️ Get The content of the request
        soup = BeautifulSoup(web_s, "html.parser") # 👉️ Parse 
        title = soup.title.string # 👉️ Get Value of Title tag

        return render(request, "fbv_get_title.html", {"title":title})

    return render(request, "fbv_get_title.html")

As you can see in the code, we've used:

  • requests.get() method to make a get request.
  • .text property to get HTML page source. 
  • beautifulsoup() to parse the HTML
  • .string to get the title tag
  • .string to get the content of the title tag

Finally, We've returned data in fbv_get_title.html.

Now, we need to add the path in urls.py:

path('fbv-get-title/', get_title, name='fbv_get_title')

In fbv_get_title.html, let's write our form:

<!DOCTYPE html>
<html>
  <head>
    <title>Django FBV With Beautifulsoup</title>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">
  </head>

  <style>
    p {
      font-size: 20px
    }

    h1 {
      text-align: center;
    }
  </style>

  <body>

    <div class="container">
      <h1 class="mt-5 mb-5">Get Title From Any Website</h1>
      
      <!-- Form -->
      <form method="POST" action="{% url 'fbv_get_title' %}"> {% csrf_token %} <div class="mt-4">
          
        <div class="col">
            <input type="url" name="url" class="form-control" placeholder="Enter URL" rows="5" required="required">
            </textarea>
        </div>

        <button type="submit" class="btn btn-primary mt-2">Go</button>

      </form>

      <!-- Result -->
      <div class="result mt-4">
        <h2>{{title}}</h2>
      </div>

    </div>
  </body>
</html>

Now let's try our form and see the result.

Beautifulsoup with Django CBV

CBV or Class Based Views is a view set by a class object. You can also use Beautifulsoup with this type of view.

In the following example, we'll do the same thing we did with TemplateView.

view.py:

from django.shortcuts import render
from django.views.generic import TemplateView
import requests # 👉️ Requests module
from bs4 import BeautifulSoup # 👉️ BeautifulSoup module

class GetTitle(TemplateView):

     template_name = "index.html" # 👉️ Template name

     def post(self, request, *args, **kwargs):

        if request.method == "POST":
            url = request.POST.get('url')
            req = requests.get(url)
            web_s = req.text
            soup = BeautifulSoup(web_s, "html.parser")
            title = soup.title.string

            return render(request, "cbv_get_title.html", {"title":title})

        return render(request, "cbv_get_title.html")

urls.py:

path('cbs-get-title/', GetTitle.as_view(), name='cbs_get_title')

fbv_get_title.html:

<!DOCTYPE html>
<html>
  <head>
    <title>Django CBV With Beautifulsoup</title>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">
  </head>

  <style>
    p {
      font-size: 20px
    }

    h1 {
      text-align: center;
    }
  </style>

  <body>

    <div class="container">
      <h1 class="mt-5 mb-5">Get Title From Any Website</h1>
      
      <!-- Form -->
      <form method="POST" action="{% url 'cbs_get_title' %}"> {% csrf_token %} <div class="mt-4">
          
        <div class="col">
            <input type="url" name="url" class="form-control" placeholder="Enter URL" rows="5" required="required">
            </textarea>
        </div>

        <button type="submit" class="btn btn-primary mt-2">Go</button>

      </form>

      <!-- Result -->
      <div class="result mt-4">
        <h2>{{title}}</h2>
      </div>

    </div>
  </body>
</html>

Conclusion

In this tutorial, we've learned how to use Django and Beautifulsoup by using two types of views Function-Based-Views And Function-Based-Views.

Related Tutorials:

Recent Tutorials: