Understand How to Use the attribute in Beautifulsoup Python
In this tutorial, we're going to cover how to use the attribute in Beautifulsoup.
Contents
1. Beautifulsoup: Find all by attribute
To find by attribute, you need to follow this syntax.
syntax:
soup.find_all(attrs={"attribute" : "value"})
let's code some examples.
example #1:
from bs4 import BeautifulSoup html_source = ''' <div class="rightSideBarParent"> <div class="leftSideBar"> <ul class="leftBarList"> <li><a id="link" href="setting-up-django-sitemaps">How to Create Django Sitemaps</a></li> </ul> </div> </div> <div class="d-flex justify-content-center mt-2" id="ads3"> <div id="ad"> <p>good</p> </div> </div> <form method="POST"> <input name="username"> </form> ''' soup = BeautifulSoup(html_source, 'html.parser') els = soup.find_all(attrs={"href" : "setting-up-django-sitemaps"}) print(els)
In the above example, we tried to find all elements that have "setting-up-django-sitemaps" in the href attribute.
output:
[<a href="setting-up-django-sitemaps">How to Create Django Sitemaps</a>]
Example #2:
In this example, we'll find all elements that have POST in the method attribute.
els = soup.find_all(attrs={"method" : "POST"}) print(els)
output:
[<form method="POST"> <input name="username"/> </form>]
2. Beautifulsoup: Get the attribute value of an element
syntax:
element['attribute name']
example #1:
html_source = ''' <div class="rightSideBarParent"> <div class="leftSideBar"> <ul class="leftBarList"> <li><a id="link" href="setting-up-django-sitemaps">How to Create Django Sitemaps</a></li> </ul> </div> </div> <div class="d-flex justify-content-center mt-2" id="ads3"> <div id="ad"> <p>good</p> </div> </div> <form method="POST"> <input name="username"> </form> ''' els = soup.find_all("ul") for el in els: print(el['class'])
Let me explain:
1. Find all by ul tag.
2. Iterate over the result.
3. Get the class value of each element.
output:
['leftBarList']
Example #2:
In the following example, well get the href attribute value.
els = soup.find_all("a") for el in els: print(el['href'])
output:
setting-up-django-sitemaps
3. Beautifulsoup: Find all by multiple attributes
syntax:
attrs={"attribute":"value", "attribute":"value",...}
example:
Let say we want to find all elements that have "setting-up-django-sitemaps" in the href attribute and "link" in the Id.
html_source = ''' <div class="rightSideBarParent"> <div class="leftSideBar"> <ul class="leftBarList"> <li><a id="link" href="setting-up-django-sitemaps">How to Create Django Sitemaps</a></li> </ul> </div> </div> <div class="d-flex justify-content-center mt-2" id="ads3"> <div id="ad"> <p>good</p> </div> </div> <form method="POST"> <input name="username"> </form> ''' soup = BeautifulSoup(html_source, 'html.parser') els = soup.find_all(attrs={"href":"setting-up-django-sitemaps", "id":"link"}) print(els)
output:
[<a href="setting-up-django-sitemaps" id="link">How to Create Django Sitemaps</a>]
4. Beautifulsoup: Check if an attribute exists
In this part of the tutorial, we'll learn how to check an element attribute is exists.
syntax:
has_attr('some_attribute')
In the following example, we'll check if <a> attribute href and name exist.
If so, it will return True, and if not, it will return False.
example:
html_source = ''' <div class="rightSideBarParent"> <div class="leftSideBar"> <ul class="leftBarList"> <li><a id="link" href="setting-up-django-sitemaps">How to Create Django Sitemaps</a></li> </ul> </div> </div> <div class="d-flex justify-content-center mt-2" id="ads3"> <div id="ad"> <p>good</p> </div> </div> <form method="POST"> <input name="username"> </form> ''' soup = BeautifulSoup(html_source, 'html.parser') els = soup.find_all("a") for el in els: #check href attribute print(el.has_attr('href')) #check name attribute print(el.has_attr('name'))
output:
True False
5. Beautifulsoup: Find attribute contains a number
in this last part of this tutorial, we'll find elements that contain a number in the id attribute value.
To do this, we need to use Regex with Beautifulsoup.
example:
html_source = ''' <div class="rightSideBarParent"> <div class="leftSideBar"> <ul class="leftBarList"> <li><a id="link" href="setting-up-django-sitemaps">How to Create Django Sitemaps</a></li> </ul> </div> </div> <div class="d-flex justify-content-center mt-2" id="ads3"> <div id="ad"> <p>good</p> </div> </div> <form method="POST"> <input name="username"> </form> ''' soup = BeautifulSoup(html_source, 'html.parser') els = soup.find_all(id=re.compile("\d")) print(els)
"\d": Matches any decimal digit. Equivalent to [0-9].
output:
<div class="d-flex justify-content-center mt-2" id="ads3"> <div id="ad"> <p>good</p> </div> </div>
