Bulk Screenshot API

Bulk Screenshot API Architecture

Bulk screenshot workflows that process hundreds or thousands of URLs require a different architecture than single on-demand screenshot requests. Rather than making one synchronous HTTP call per screenshot from application code, bulk workflows use a job queue to distribute screenshot requests across time, a worker process that dequeues and processes jobs at a controlled rate, and a storage layer that persists generated screenshots for later retrieval. This architecture decouples the URL submission step from the screenshot generation step, allowing applications to accept large batches of URLs instantly and process them asynchronously at the API's rate limit of ten requests per second. A queue depth of 1000 URLs processes to completion in approximately 100 seconds at maximum throughput, while a depth of 10000 URLs completes in about 1000 seconds running continuously.

The simplest bulk screenshot queue for small to medium batches uses a Python or Node.js script with an in-memory list of URLs and a rate-limiting loop. Iterate over the URL list, call the SnapAPI screenshot endpoint for each URL, save the response to disk or cloud storage, and sleep for 100 milliseconds between requests to stay at or below ten requests per second. This script-based approach requires no queue infrastructure and works reliably for batches up to a few thousand URLs when run on a server or long-lived cloud function. For larger batches or batches that need to survive process restarts, use a Redis-backed job queue that persists pending URLs and tracks completed, failed, and retried jobs.

Bulk Screenshot API with Redis Queue

Redis-based job queues provide durable bulk screenshot processing with automatic retry for failed jobs. Bull for Node.js and RQ for Python are the most popular libraries for Redis-backed job queues. Define a screenshot job processor function that accepts a URL as the job data, calls the SnapAPI endpoint, uploads the screenshot to S3, and updates a database record with the S3 URL. Configure the queue concurrency to five to match a conservative throughput target below the API rate limit. Bull's job events allow tracking progress: listen for completed events to update a progress counter and failed events to log error details and trigger retry logic. The Redis queue survives server restarts with all pending jobs intact, making it appropriate for large bulk screenshot workloads that run overnight or over multiple days.

Webhook delivery from SnapAPI eliminates the need for active queue polling in bulk screenshot workflows. Instead of waiting for each HTTP response, submit screenshot requests with a webhook_url parameter pointing to your application's callback endpoint. SnapAPI calls the webhook URL when each screenshot completes, passing the job ID and screenshot data in the POST body. Your webhook handler stores the screenshot and marks the job as complete in your database. This pattern is particularly efficient for bulk workflows because the submitting application does not maintain open HTTP connections during screenshot generation, freeing connection pool resources for other work. Configure your webhook endpoint behind a queue itself — receive and acknowledge webhook requests quickly, then process the screenshot storage asynchronously to prevent SnapAPI from retrying webhook calls due to slow handler response times.

Bulk Screenshot API Storage Patterns

Storing thousands of screenshots from a bulk job requires object storage rather than local filesystem storage. Amazon S3, Google Cloud Storage, and Cloudflare R2 all provide the scale, durability, and HTTP serving capabilities needed for screenshot storage at scale. Use a consistent key naming scheme that encodes the target URL as a filesystem-safe hash or slug, the capture timestamp, and the requested parameters, so that screenshot files are uniquely identifiable and retrievable without a database lookup. Generate S3 presigned URLs with a one-week expiry for serving screenshots to users, rotating the URLs before expiry if long-term access is required. Store screenshot metadata — target URL, capture timestamp, file size, word count, HTTP status code — in a PostgreSQL table indexed by the target URL for efficient lookups.

Bulk Screenshot API Rate Limiting Strategy

Effective bulk screenshot processing stays consistently below the API rate limit to avoid 429 errors that trigger retry delays and reduce overall throughput. Implement token bucket rate limiting in your bulk processing code rather than relying on simple sleep intervals, because token bucket limiting correctly handles bursts and irregular processing times without accumulating rate limit violations. Libraries like aiolimiter for Python asyncio and Bottleneck for Node.js implement token bucket algorithms that integrate cleanly with async HTTP calls. Set the token bucket refill rate to eight requests per second — slightly below the API's ten request per second limit — to leave headroom for retry requests without exceeding the limit. Monitor the X-RateLimit-Remaining response header from SnapAPI to dynamically adjust processing speed when approaching the per-second limit.

Bulk Screenshot API Error Handling and Retry

Bulk screenshot jobs encounter several categories of errors that require different handling strategies. Target URLs that return 404 or 5xx status codes produce SnapAPI 422 errors indicating the target page is unavailable — these jobs should be marked as failed with the error reason recorded, not retried immediately, because the target URL is unlikely to become available within the job processing window. Network timeouts and SnapAPI 500 errors represent transient infrastructure failures that benefit from exponential backoff retry with a maximum of three attempts before marking the job as permanently failed. SnapAPI 429 rate limit errors indicate the queue is processing too fast and the rate limiter needs adjustment — pause the queue for one second and resume at a lower throughput target after a 429 response. Categorizing errors correctly prevents bulk jobs from spinning in retry loops on permanently failed URLs while still recovering automatically from transient failures.