Last modified: Oct 24, 2026
Async Web Scraping with Playwright
Web scraping is a common way to collect data from websites. Modern sites load content with JavaScript. Simple HTTP requests often miss that data.
Playwright solves this problem. It drives a real browser. It also has a strong async API. This makes it fast and reliable.
In this guide, you will learn how async scraping works. You will see setup steps, code examples, and practical tips.
Why Use Async Scraping?
Scraping often waits. The browser waits for pages, network calls, and elements. Sync code blocks during every wait.
Async code does not block. It can run many tasks at once. Ten pages can load in parallel. This can cut run time by a large factor.
Playwright supports both sync and async APIs. The async API pairs well with asyncio. It is the better choice for larger jobs.
Installing Playwright
Start by installing the package. Then install the browser binaries.
# Install the library
pip install playwright
# Download the Chromium browser
playwright install chromium
Chromium is enough for most scraping tasks. You can also install Firefox or WebKit if a site needs them.
Your First Async Scraper
Let us scrape page titles. The code below opens a browser and fetches one page.
import asyncio
from playwright.async_api import async_playwright
async def main():
async with async_playwright() as p:
# Launch a headless browser
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
# Navigate and wait for the DOM
await page.goto("https://example.com", wait_until="domcontentloaded")
# Read the page title
title = await page.title()
print(title)
await browser.close()
asyncio.run(main())
Example Domain
Notice the pattern. The async_playwright context manager starts the driver. The launch method starts a browser. The goto method loads a URL.
Every Playwright call is awaited. This keeps the event loop free for other work.
Extracting Data from Elements
You rarely need just the title. You need text and attributes. Playwright locators make this easy.
import asyncio
from playwright.async_api import async_playwright
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch()
page = await browser.new_page()
await page.goto("https://books.toscrape.com", wait_until="domcontentloaded")
# Grab all product cards
cards = page.locator("article.product_pod")
count = await cards.count()
print(f"Found {count} books")
# Read the first five titles and prices
for i in range(5):
card = cards.nth(i)
title = await card.locator("h3 a").get_attribute("title")
price = await card.locator(".price_color").inner_text()
print(f"{title} - {price}")
await browser.close()
asyncio.run(main())
Found 20 books
A Light in the Attic - £51.77
Tipping the Velvet - £53.74
Soumission - £50.10
Sharp Objects - £47.82
Sapiens: A Brief History of Humankind - £54.23
The locator method targets elements with CSS selectors. The nth method picks one item. The get_attribute method reads attributes. The inner_text method reads visible text.
Scraping Many Pages Concurrently
This is where async shines. You can open many tabs at once. Each tab loads a page in parallel.
import asyncio
from playwright.async_api import async_playwright
URLS = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 6)]
async def scrape_page(context, url):
page = await context.new_page()
try:
await page.goto(url, wait_until="domcontentloaded")
titles = await page.locator("h3 a").all_inner_texts()
return titles
finally:
await page.close()
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch()
# One context shares cookies and settings
context = await browser.new_context()
# Run all page tasks at the same time
results = await asyncio.gather(*(scrape_page(context, url) for url in URLS))
all_titles = [t for page_titles in results for t in page_titles]
print(f"Collected {len(all_titles)} titles")
await browser.close()
asyncio.run(main())
Collected 100 titles
The gather function runs all tasks together. Each task opens its own page. The new_context method creates an isolated session. The close method frees each page when done.
Be careful with concurrency. Too many tabs can overload your machine or the target site. Start with five to ten tabs. Add a semaphore to cap the limit.
Controlling Concurrency with a Semaphore
A semaphore limits how many tasks run at once. This keeps memory and network use under control.
import asyncio
from playwright.async_api import async_playwright
URLS = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 21)]
LIMIT = asyncio.Semaphore(5) # Max 5 pages at a time
async def scrape_page(context, url):
async with LIMIT:
page = await context.new_page()
try:
await page.goto(url, wait_until="domcontentloaded")
return await page.locator("h3 a").all_inner_texts()
finally:
await page.close()
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch()
context = await browser.new_context()
results = await asyncio.gather(*(scrape_page(context, url) for url in URLS))
print(f"Pages scraped: {len(results)}")
await browser.close()
asyncio.run(main())
Pages scraped: 20
This pattern scales well. You can scrape hundreds of pages with a stable footprint.
Waiting for Dynamic Content
Many sites load data after the first paint. You must wait for the right elements. The wait_for_selector method pauses until an element appears.
import asyncio
from playwright.async_api import async_playwright
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch()
page = await browser.new_page()
await page.goto("https://quotes.toscrape.com/js/", wait_until="domcontentloaded")
# Wait for JS-rendered quotes
await page.wait_for_selector(".quote")
quotes = await page.locator(".quote .text").all_inner_texts()
print(quotes[0])
await browser.close()
asyncio.run(main())
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
Avoid fixed sleeps. The wait_for_timeout method works, but it is fragile. Prefer element-based waits. They are faster and more reliable.
If you also work with APIs, the same async mindset helps. For example, you can pair scraped data with models through OpenAI function calling in Python.
Handling Errors and Retries
Networks fail. Pages time out. Always wrap navigation in try blocks. Add a simple retry loop.
import asyncio
from playwright.async_api import async_playwright
async def fetch_with_retry(context, url, retries=3):
for attempt in range(1, retries + 1):
page = await context.new_page()
try:
await page.goto(url, wait_until="domcontentloaded", timeout=15000)
return await page.content()
except Exception as e:
print(f"Attempt {attempt} failed: {e}")
if attempt == retries:
return None
await asyncio.sleep(2 * attempt) # Backoff
finally:
await page.close()
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch()
context = await browser.new_context()
html = await fetch_with_retry(context, "https://example.com")
print("Page loaded" if html else "Gave up")
await browser.close()
asyncio.run(main())
Page loaded
The sleep function adds backoff between attempts. This gives the server time to recover.
Blocking Unneeded Resources
Images, fonts, and media slow down scraping. You can block them with route rules. This saves bandwidth and time.
import asyncio
from playwright.async_api import async_playwright
BLOCKED = {"image", "media", "font"}
async def block(route):
if route.request.resource_type in BLOCKED:
await route.abort()
else:
await route.continue_()
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch()
page = await browser.new_page()
await page.route("**/*", block) # Apply the filter
await page.goto("https://books.toscrape.com", wait_until="domcontentloaded")
print(await page.title())
await browser.close()
asyncio.run(main())
All products | Books to Scrape - Sandbox
The route method intercepts requests. The abort method cancels blocked ones. The continue_ method lets others through.
Scraping and Data Pipelines
Scraping is often the first step in a pipeline. You may clean text, split it, and store it. Text cleanup often starts with tokenization with NLTK in Python.
For structured output, consider binary formats. You can learn to compile and use .proto files in Python to define stable schemas.
Best Practices
Respect the site. Read the robots file and terms of service. Use a clear user agent.
Keep concurrency modest. Use a semaphore. Add retries with backoff.
Reuse one browser. Create a context per job. Close pages as soon as you finish them.
Store results as you go. Write to a file or database. This avoids losing work on a crash.
Log every failure. A failed page is easy to retry later.
Common Pitfalls
Do not mix sync and async APIs. Pick one style per project.
Do not forget await. A missing await returns a coroutine, not data.
Do not use long fixed sleeps. Wait for selectors instead.
Do not launch a new browser per URL. It is slow and heavy.
Conclusion
Async web scraping with Playwright is fast and clean. You get a real browser plus a modern async API.
Start small. Scrape one page. Then add concurrency with asyncio.gather. Cap it with a semaphore. Add retries and block heavy resources.
With these patterns, you can build reliable scrapers that handle modern JavaScript sites. Keep your code readable, respect target sites, and log your results. Your future self will thank you.