Last modified: Apr 22, 2026 By Alexander Williams
Master BeautifulSoup find_all() for Web Scraping
Web scraping is a vital skill. It lets you gather data from websites. Python's BeautifulSoup library makes this task easy. Its find_all() function is the core tool for extraction.
This guide will teach you how to use find_all() effectively. You will learn its syntax, arguments, and practical applications. We will use clear examples to demonstrate its power.
What is BeautifulSoup find_all()?
The find_all() method searches a parsed HTML document. It finds all tags that match your specified criteria. It returns a list of all matching elements, or an empty list if none are found.
This function is your primary tool for data collection. Whether you need all links, specific paragraphs, or items with a certain class, find_all() is the answer.
Basic Syntax of find_all()
The basic syntax is straightforward. You call it on a BeautifulSoup object.
# Basic syntax
soup.find_all(name, attrs, recursive, string, limit, **kwargs)
Parameters Explained:
- name: The name of the tag to find (e.g., 'div', 'a').
- attrs: A dictionary of attribute filters (e.g., {'class': 'header'}).
- recursive: If True (default), searches all descendants. If False, searches only direct children.
- string: Searches for tags containing specific text.
- limit: Stops searching after finding a specified number of results.
- **kwargs: Allows you to pass attributes as keyword arguments (e.g., class_='menu').
Practical Examples of find_all()
Let's work with a sample HTML string. This will make the examples clear.
from bs4 import BeautifulSoup
html_doc = """
<html>
<head><title>Test Page</title></head>
<body>
<h1 class="title main">Welcome</h1>
<p>First paragraph.</p>
<p class="highlight">Second <strong>important</strong> paragraph.</p>
<ul id="menu">
<li><a href="/home">Home</a></li>
<li><a href="/about" class="active">About</a></li>
<li><a href="/contact">Contact</a></li>
</ul>
<div role="navigation">Nav div</div>
</body>
</html>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
1. Find All Tags by Name
Find every paragraph (<p>) tag in the document.
# Find all paragraph tags
all_paragraphs = soup.find_all('p')
print(f"Found {len(all_paragraphs)} paragraph(s).")
for p in all_paragraphs:
print(p.text)
Found 2 paragraph(s).
First paragraph.
Second important paragraph.
2. Find Tags by Class Attribute
You can find elements with a specific CSS class. Use the class_ keyword argument. Note the underscore to avoid conflict with Python's class keyword.
# Find all elements with class="highlight"
highlighted = soup.find_all(class_="highlight")
for elem in highlighted:
print(elem)
<p class="highlight">Second <strong>important</strong> paragraph.</p>
For finding a div with multiple classes, you need a more specific approach. You can learn the details in our dedicated guide on BeautifulSoup Find Div With Multiple Classes.
3. Find Tags by ID and Other Attributes
Find the unordered list with id="menu".
# Find element by ID
menu = soup.find_all(id="menu")
print(menu)
[<ul id="menu">
<li><a href="/home">Home</a></li>
<li><a href="/about" class="active">About</a></li>
<li><a href="/contact">Contact</a></li>
</ul>]
You can also search by any attribute. For instance, to find elements by their role attribute, check out our article on BeautifulSoup Find by Role Attribute.
4. Find Tags by Text Content (string)
Search for tags that contain specific text.
# Find tags containing the word "paragraph"
tags_with_text = soup.find_all(string="paragraph")
# Note: 'string' returns NavigableString objects, not tags.
# To get the parent tag, we can modify the search.
tags_with_para = soup.find_all(string=lambda text: "paragraph" in text.lower())
for string in tags_with_para:
print(f"Text: '{string}', Parent Tag: {string.parent.name}")
Text: 'First paragraph.', Parent Tag: p
Text: 'Second important paragraph.', Parent Tag: p
5. Using the limit Parameter
Stop the search after finding the first two anchor tags.
# Find only the first 2 links
first_two_links = soup.find_all('a', limit=2)
for link in first_two_links:
print(link.get('href'))
/home
/about
6. Combining Filters
You can combine name, attributes, and text for precise searches.
# Find all 'a' tags with class="active"
active_links = soup.find_all('a', class_='active')
for link in active_links:
print(link.text, link['href'])
About /about
Similarly, you can find specific tags like the strong tag within your results. See examples in our post about the BeautifulSoup find strong tag.
find_all() vs find() and find_parent()
It's important to distinguish find_all() from similar methods.
find() returns only the first matching element (or None). find_all() returns a list of all matches.
For moving up the HTML tree, use find_parent() and find_parents(). These are essential for navigating from a child element to its ancestors. Learn more about navigating upwards in our guides on find_parent() and find_parents().
Common Pitfalls and Tips
Remember It Returns a List: Even if one item matches, find_all() returns a list. Access items by index (e.g., results[0]).
Use 'class_' not 'class': Always use class_ as the keyword argument.
Check for Empty Results: Always check if the list is empty before processing to avoid errors.
results = soup.find_all('non-existent-tag')
if results:
# Process results
print("Found items.")
else:
print("No items found.") # This will execute
Conclusion
The find_all() function is the powerhouse of BeautifulSoup. It allows you to filter and extract any data from HTML with precision.
Start by finding tags by name. Then, refine your searches with attributes like class and ID. Use the string parameter to filter by content.
Combine these techniques to build robust web scrapers. Remember to handle cases where no data is found. With practice, find_all() will become an indispensable part of your Python toolkit.
For related parsing tasks, such as locating CSS stylesheets within HTML, you can explore our article on BeautifulSoup Find Stylesheets.