Why content extraction matters for AI workflows

Every AI agent, RAG pipeline, and LLM-powered application that needs to understand web content faces the same problem: raw HTML is noise. A typical web page is 90% nav bars, ads, footers, cookie banners, and JavaScript artifacts. The actual content — the article, the product description, the data — is buried inside.

Traditionally, developers solved this with libraries like Readability.js (Mozilla's parser), BeautifulSoup heuristics, or Trafilatura. These work for simple pages but break constantly on modern single-page applications, paywalled content, and JavaScript-rendered pages.

SnapAPI's /v1/extract endpoint takes a different approach: it actually renders the page in a real Chromium browser (with JavaScript execution), then extracts the semantic content from the rendered DOM — not the raw HTML source. This means it works on SPAs, dynamic content, and authenticated pages that static scrapers can't touch.

What the /v1/extract endpoint returns

The extract endpoint supports three output formats, configurable with the format parameter:

• markdown — Clean markdown: headings, lists, links, code blocks. Best for feeding to LLMs like Claude, GPT-4, or Llama.
• text — Plain text only. No markup. Best for embedding into vector databases or computing semantic similarity.
• json — Structured extraction: title, author, date, body, images, links. Best for building knowledge graphs or structured data pipelines.

Code examples: Using /v1/extract in AI pipelines

RAG pipeline with LangChain (Python)

import requests
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings

def extract_and_index(url: str, api_key: str):
    # Extract clean content from any URL
    response = requests.post('https://api.snapapi.pics/v1/extract',
        headers={'X-API-Key': api_key},
        json={'url': url, 'format': 'markdown', 'wait_until': 'networkidle'})
    
    content = response.json()['content']
    
    # Split and embed for RAG
    splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
    chunks = splitter.split_text(content)
    
    vectorstore = Chroma.from_texts(chunks, OpenAIEmbeddings(),
        metadatas=[{'source': url}] * len(chunks))
    return vectorstore

AI agent web research (JavaScript)

// Give your AI agent the ability to read any webpage
async function readWebpage(url) {
  const response = await fetch('https://api.snapapi.pics/v1/extract', {
    method: 'POST',
    headers: { 'X-API-Key': process.env.SNAPAPI_KEY, 'Content-Type': 'application/json' },
    body: JSON.stringify({ url, format: 'markdown', wait_until: 'networkidle' })
  });
  const { content, title, word_count } = await response.json();
  return { content, title, word_count };
}

// Use in an agent loop
const pageContent = await readWebpage('https://example.com/article');
const summary = await llm.complete(`Summarize this:

${pageContent.content}`);

Handling paywalled and authenticated pages

The /v1/extract endpoint supports custom cookies and headers, making it possible to extract content from authenticated pages. Pass session cookies to access content behind login walls, or use the custom_js parameter to run JavaScript before extraction (useful for clicking "accept" on cookie banners or expanding collapsed content).

Comparison: SnapAPI extract vs alternatives