Web Scraping with Python in 2026: Requests, Playwright, and API Approaches

Python remains the top choice for web scraping. But the right tool depends on your target site. Here is when to use Requests, when to reach for Playwright, and when an API saves you days of work.

Tier 1 — Simple Static Pages: requests + BeautifulSoup

For plain HTML pages — government data, Wikipedia, news archives, simple directories — requests with BeautifulSoup is still the fastest path. No browser overhead, no JavaScript execution, sub-second response times. Install both with pip install requests beautifulsoup4.

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (compatible; MyBot/1.0)'}
resp = requests.get('https://example.com/data', headers=headers, timeout=10)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, 'html.parser')
rows = soup.select('table.data-table tbody tr')

for row in rows:
    cells = [td.get_text(strip=True) for td in row.find_all('td')]
    print(cells)

This pattern fails when: the page requires JavaScript to render, the site checks for headless browser signatures, or the content is loaded via XHR after the initial HTML response.

Tier 2 — JavaScript-Rendered Pages: Playwright

React, Vue, Angular, and Next.js apps render content client-side. The raw HTML response from the server is often just a shell with an empty <div id="root">. You need a real browser. Playwright is the current best choice over Selenium and Puppeteer: faster, more reliable, better async support, and actively maintained by Microsoft.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto('https://spa-example.com/products', wait_until='networkidle')

    # Wait for specific element to appear
    page.wait_for_selector('.product-card', timeout=10000)

    products = page.query_selector_all('.product-card')
    for prod in products:
        name = prod.query_selector('h3').inner_text()
        price = prod.query_selector('.price').inner_text()
        print(f'{name}: {price}')

    browser.close()

Playwright works well for medium-scale scraping. The challenge at scale: Chromium uses 150-200 MB RAM per instance, crashes under load, and requires careful browser pool management. Running 50 concurrent Playwright instances needs a dedicated, well-tuned machine.

Tier 3 — Protected Sites and Scale: API Approach

Sites with bot protection (Cloudflare, DataDome, PerimeterX) actively detect and block headless browsers. Playwright in default mode fails immediately. Stealth mode (playwright-stealth or undetected-chromedriver) works temporarily but requires constant maintenance as detection systems update. For commercial scraping workloads, an API that handles stealth and infrastructure is dramatically more reliable.

import requests

# SnapAPI handles stealth, proxies, and browser infrastructure
res = requests.post('https://api.snapapi.pics/v1/scrape',
    headers={'X-Api-Key': 'sk_live_xxx'},
    json={
        'url': 'https://protected-ecommerce-site.com/products',
        'waitFor': '.product-list',
        'stealth': True,
    }
)
data = res.json()
html = data['html']
print(f"Got {len(html)} bytes of rendered HTML")

# Parse normally with BeautifulSoup
soup = BeautifulSoup(html, 'html.parser')
prices = [el.text for el in soup.select('.price')]

Structured Data Extraction

For structured extraction — product details, article content, contact information — SnapAPI's extract endpoint returns JSON directly without you writing CSS selectors. Pass a schema describing the fields you want and the API returns structured data.

res = requests.post('https://api.snapapi.pics/v1/extract',
    headers={'X-Api-Key': 'sk_live_xxx'},
    json={
        'url': 'https://example.com/product/123',
        'schema': {
            'name': 'string',
            'price': 'number',
            'description': 'string',
            'images': 'array',
            'inStock': 'boolean'
        }
    }
)
product = res.json()['data']
print(product)  # {'name': 'Widget X', 'price': 49.99, 'inStock': True, ...}

SnapAPI's free tier gives 200 requests/month — enough for prototyping. Starter ($19/mo) covers 5,000 requests for small production workloads. All plans include scrape, extract, screenshot, PDF, video, and AI analysis under a single key. Get started free.