From CSS selectors to AI-powered extraction — techniques, tools, and code examples for extracting structured data from any website in 2026.
Content extraction is the process of taking unstructured web content and pulling out specific, structured data fields. Price from a product page. Headline from a news article. Job title from a company team page. The challenge is that every site has a unique HTML structure, many sites render content via JavaScript, and an increasing number of sites actively block automated access.
Extraction approaches range from simple to powerful, with increasing infrastructure requirements at each level.
Level 1 is static HTML parsing. Download the raw HTML with an HTTP GET and parse it with a DOM library like cheerio, BeautifulSoup, or php-html-parser. This works for sites that return full content in the initial HTML response, but fails entirely on JavaScript-rendered single-page applications.
Level 2 is headless browser extraction. Use Playwright or Puppeteer to render the page in a real browser, wait for JavaScript to finish, and extract using CSS selectors or evaluate JavaScript. This handles SPAs but requires managing a browser process, which is resource-intensive and operationally complex.
Level 3 is cloud extraction API. Send the URL and extraction schema to a managed service. The service handles rendering, anti-bot bypass, IP rotation, and returns clean structured JSON. Zero infrastructure on your side.
CSS selectors are the most portable way to define extraction schemas. The same selector that works in browser DevTools works in Playwright, cheerio, and cloud extraction APIs. Learning to write precise, robust selectors is the most valuable skill for content extraction work.
// Extraction schema using CSS selectors
const schema = {
// Simple element text
title: "h1",
price: ".product-price",
// Attribute extraction
canonical: { selector: 'link[rel="canonical"]', attr: "href" },
og_image: { selector: 'meta[property="og:image"]', attr: "content" },
author_link: { selector: ".author-name a", attr: "href" },
// Multiple values (returns array)
tags: { selector: ".post-tag", multiple: true },
images: { selector: ".article-body img", attr: "src", multiple: true },
// Nested schema
reviews: {
selector: ".review-item",
multiple: true,
schema: {
author: ".reviewer-name",
rating: ".star-count",
text: ".review-body",
}
}
};React, Vue, and Angular applications render content client-side. The raw HTML served by the server is an empty shell; the actual content is injected by JavaScript after the page loads. Static parsers see the shell, not the content. You need a real browser render to get extractable DOM.
Indicators that a page is JavaScript-rendered: the raw HTML contains only a div with an id like "root" or "app", the content appears in the browser but not in curl output, and the HTML source lacks the text you can see on the page.
// SnapAPI extract — handles JS rendering automatically
const { data } = await axios.post(
"https://api.snapapi.pics/v1/extract",
{
url: "https://reactapp.example.com/products/widget",
schema: { title: "h1", price: "[data-price]" },
wait_for: "[data-price]", // wait for this selector before extracting
stealth: true,
},
{ headers: { "X-Api-Key": process.env.SNAP_API_KEY } }
);
console.log(data); // { title: "Widget Pro", price: "$49.99" }For sites where the HTML structure is inconsistent or changes frequently, AI-powered extraction removes the need to maintain CSS selectors. Describe what you want in plain English and let the model find it on the rendered page.
const { data } = await axios.post(
"https://api.snapapi.pics/v1/analyze",
{
url: "https://example.com/product",
prompt: "Extract product name, current price, original price, discount percentage, and all listed features. Return as JSON.",
},
{ headers: { "X-Api-Key": process.env.SNAP_API_KEY } }
);
const product = JSON.parse(data.result);AI extraction is slower and more expensive per call than CSS selector extraction, but it handles irregular HTML structures, adapts to site redesigns without selector maintenance, and can interpret content semantically — for example, inferring whether a price is discounted even if there is no explicit discount class.
Websites change. A CSS class that exists today may be renamed or removed in a future deploy. Build resilience into your extraction schemas by using multiple fallback selectors and validating extracted values against expected formats.
function validateExtracted(data) {
const errors = [];
if (!data.title || data.title.length < 3) errors.push("title missing or too short");
if (!data.price || !/^\$?[\d,]+(\.\d{2})?$/.test(data.price)) errors.push("price format invalid");
if (errors.length > 0) {
console.warn("Extraction validation failed:", errors, data);
// Alert: selector may be broken
}
return errors.length === 0;
}Create a free SnapAPI account to get 200 monthly extractions. Build your extraction schema in the interactive API playground in the dashboard, verify it against your target URLs, then integrate the endpoint into your production code.
200 free extractions/month. JavaScript rendering, stealth mode, and AI analysis included.
Create Free AccountWebsites change. A product listing page that returns clean data today may add a wrapper div tomorrow that breaks your CSS selector. Build resilient schemas by specifying multiple fallback selectors for each field. SnapAPI's extract endpoint accepts a selectors array per field — it tries each in order and returns the first match.
{
"url": "https://shop.example.com/product/123",
"schema": {
"price": {
"selectors": [".price-current", ".product-price", "[data-price]", "meta[property='og:price:amount']"],
"attribute": "content",
"transform": "number"
},
"title": {
"selectors": ["h1.product-title", "h1", "meta[property='og:title']"],
"attribute": "content"
}
}
}The transform field accepts "number", "trim", "lowercase", and "url" — normalizing extracted values before they reach your application. A price field with transform: "number" strips currency symbols and thousands separators automatically, returning a float ready for database storage.
For paginated listings, combine the extract endpoint with a simple loop. Extract the "next page" link from each page using a selector, then queue the next URL. SnapAPI handles the JavaScript rendering and anti-bot bypass on every request, so pagination across protected e-commerce sites works without additional configuration.
Rate-limit your pagination loop to avoid triggering the target site's throttling. A delay of 1-2 seconds between requests is sufficient for most sites. For large scraping jobs, use SnapAPI's webhook parameter to receive each page's results asynchronously instead of blocking on each HTTP response.
When you don't know the site structure in advance — or when target sites change frequently enough that maintaining selectors isn't practical — SnapAPI's AI extraction mode uses a language model to identify and return the data you're asking for.
const resp = await fetch("https://api.snapapi.pics/v1/extract", {
method: "POST",
headers: { "X-Api-Key": "sk_live_YOUR_KEY", "Content-Type": "application/json" },
body: JSON.stringify({
url: "https://news.ycombinator.com",
mode: "ai",
prompt: "Return the top 10 story titles and their point counts as JSON array"
})
});
const data = await resp.json();
console.log(data.result); // Structured JSON from LLMAI mode is slower and consumes more quota than selector-based extraction, but it handles sites with dynamic class names, shadow DOM, or heavy JavaScript rendering where CSS selectors fail entirely. Use selector-based extraction for production pipelines where performance and cost matter, and AI mode for exploratory scraping or low-frequency one-off data collection.
Always validate extracted values before inserting them into your database. Use a schema validation library — Zod in TypeScript, Pydantic in Python, or Joi in plain JavaScript — to enforce types, ranges, and required fields. Log validation failures to a dead-letter queue for manual review rather than silently dropping records, so you can debug schema drift quickly when target sites change.
Set up a daily monitoring job that runs your extraction pipeline against a test URL with known expected values. Alert on discrepancies before customers notice the stale data in your product. SnapAPI's free tier provides 200 calls per month — more than enough for health-check monitoring alongside your paid extraction workload.
SnapAPI's extract endpoint combines headless browser rendering, anti-bot bypass, proxy rotation, and flexible schema definitions into a single API call. There's no browser fleet to maintain, no proxy pool to manage, and no anti-fingerprinting code to update when detection techniques evolve. Every plan includes JavaScript rendering and stealth mode at no extra cost.
Sign up at snapapi.pics and get 200 free extractions per month — no credit card required. The free tier supports all extraction parameters including schema definitions, AI mode, CSS injection, custom cookies, and geographic IP routing. This gives you full access to evaluate the API against your production use case before choosing a paid plan.
The API documentation includes complete extraction schema reference, language examples in Node.js, Python, PHP, Go, Swift, Kotlin, and C#, plus an interactive playground to test schemas directly in the browser. If your target site is particularly protected or you need help designing a resilient schema, our support team can help you get to production faster.