Last modified: Jan 10, 2026 By Alexander Williams

Handle Broken HTML with BeautifulSoup

Web scraping often means dealing with messy, broken HTML. Real-world web pages are rarely perfectly formed. Tags might be unclosed. Attributes may be missing quotes. The structure can be a nest of errors.

This broken code can crash a simple parser. It can also give you incorrect data. BeautifulSoup is a Python library built for this chaos. It can parse and clean malformed HTML documents with ease.

This guide will show you how. You will learn to use BeautifulSoup's forgiving parsers. You will also learn techniques to extract data from messy pages reliably.

Why HTML Breaks and Why It Matters

HTML on live websites is often invalid. Browsers are incredibly forgiving. They can render pages with many syntax errors.

Automated scripts are not as forgiving. A missing closing tag can break your data extraction. It can cause your script to fail or return empty results.

BeautifulSoup acts like a browser in this regard. It can take poorly written HTML and turn it into a parseable tree. This tree is called the Document Object Model (DOM).

Choosing the Right Parser

BeautifulSoup doesn't parse HTML by itself. It relies on external parsers. The choice of parser is crucial for handling broken HTML.

You specify the parser when creating the BeautifulSoup object. The main options are html.parser, lxml, and html5lib.

For a detailed comparison, see our guide on BeautifulSoup vs lxml: Which Python Parser to Use.

The html.parser Parser

This is Python's built-in parser. It is decent with well-formed HTML. It is not the best for severely broken pages.

It's a good default if you don't want extra installations. It comes with Python's standard library.


from bs4 import BeautifulSoup

broken_html = "This is a paragraph without a close tag"
soup = BeautifulSoup(broken_html, 'html.parser')
print(soup.prettify())


<p>
 This is a paragraph without a close tag
</p>

The lxml Parser

The lxml parser is very fast. It is also quite tolerant of errors. It is a great balance of speed and forgiveness.

You need to install it separately: pip install lxml. It is excellent for most scraping tasks.


soup = BeautifulSoup(broken_html, 'lxml')
print(soup.prettify())

The html5lib Parser

This is the most forgiving parser. It parses HTML the way a web browser does. It will fix even the most broken markup.

It is the slowest of the three. Use it when your HTML is a complete mess. Install it with pip install html5lib.


soup = BeautifulSoup(broken_html, 'html5lib')
print(soup.prettify())

Practical Example: Cleaning a Messy Page

Let's work with a realistic example of broken HTML. We will see how BeautifulSoup fixes it.


# Example of severely broken HTML
messy_html = """
First paragraph
            
Second paragraph
Item 1
                
Item 2


"""

# Parse with html5lib for maximum forgiveness
soup = BeautifulSoup(messy_html, 'html5lib')
print("Fixed HTML Structure:")
print(soup.prettify())


Fixed HTML Structure:
<html>
 <head>
 </head>
 <body>
  <div id="content">
   <p>
    First paragraph
   </p>
   <p>
    Second paragraph
   </p>
   <ul>
    <li>
     Item 1
    </li>
    <li>
     Item 2
    </li>
   </ul>
   <img alt="My Image" src="image.jpg"/>
  </div>
 </body>
</html>

Notice the fixes. The parser closed the first <p> tag. It closed the <li> tag for "Item 1". It also added quotes around the alt attribute and closed the <img> tag.

Extracting Data from the Repaired Tree

Once parsed, you can navigate the fixed tree normally. Use methods like find() and find_all().

For complex nested structures, our Parse Nested HTML with BeautifulSoup Guide can help.


# Extract all paragraph text
paragraphs = soup.find_all('p')
for p in paragraphs:
    print(p.get_text(strip=True))

# Extract image source
img_tag = soup.find('img')
if img_tag:
    print(f"Image source: {img_tag['src']}")


First paragraph
Second paragraph
Image source: image.jpg

When BeautifulSoup Isn't Enough

Sometimes the problem isn't just broken HTML. The content you need might be loaded dynamically with JavaScript.

BeautifulSoup alone cannot execute JavaScript. For those cases, you need a different tool. Consider using Scrape Dynamic Content with BeautifulSoup & Requests-HTML.

This combines BeautifulSoup with a library that can render JavaScript.

Best Practices for Robust Scraping

Follow these tips to make your scraping scripts more resilient.

Always use a tolerant parser. Start with lxml for speed. Switch to html5lib if you encounter parsing errors.

Use defensive coding. Always check if a tag was found before accessing its attributes.


tag = soup.find('some-rare-tag')
if tag and tag.has_attr('href'): # Check existence first
    link = tag['href']

Prettify for debugging. Use soup.prettify() to see how BeautifulSoup interpreted the HTML. This can reveal issues with your selectors.

Conclusion

Handling broken HTML is a core part of web scraping. BeautifulSoup excels at this task. Its flexible parser system turns messy web pages into clean data structures.

Remember to choose the right parser for your needs. Use html5lib for the worst HTML. Use lxml for a good mix of speed and tolerance.

Combine these techniques with defensive coding. Your scrapers will become much more reliable. They will handle the imperfect web with grace.