AWS Product Scraper: Serverless Data Pipeline

The problem

I needed to collect product data from multiple websites regularly. Running scrapers on my local machine wasn't sustainable. Leaving a server running 24/7 for jobs that run a few times a day seemed wasteful.

This seemed like a perfect use case for serverless. Pay only when the scraper runs, scale automatically when needed, and not worry about server maintenance.

What I built

A fully serverless scraping pipeline on AWS:

• Lambda functions that scrape target websites
• EventBridge schedules to trigger scrapes automatically
• SQS queues to handle distributed processing
• DynamoDB for storing scraped data
• S3 for raw files and images

Tech stack

Layer	Technology	Why
Compute	AWS Lambda	Pay per execution, auto scaling
Scheduler	EventBridge	Cron expressions, serverless
Queue	SQS	Decoupled processing
Database	DynamoDB	Serverless NoSQL
Storage	S3	Raw file storage
Infrastructure	AWS CDK	TypeScript infra as code

What building this taught me

1. Cold starts are real and they hurt

My first Lambda functions took 3 to 5 seconds to start from cold. For scraping, that added up fast.

I learned to optimize bundle sizes, use provisioned concurrency for critical paths, and accept that some latency is just part of serverless life.

// Before: 3-5 second cold starts
import * as AWS from 'aws-sdk' // Huge bundle

// After: <1 second cold starts
import { DynamoDB } from '@aws-sdk/client-dynamodb' // Tree-shakeable

Resources:

• Lambda Cold Start Optimization
• AWS SDK v3 (Modular)

2. DynamoDB pricing is tricky

I assumed "pay per request" meant cheap. Then I saw my first bill.

Scan operations are expensive. Provisioned capacity vs on demand changes costs dramatically. I rewrote queries multiple times to reduce costs.

The lesson: design your access patterns first, then design your table. Not the other way around.

Resources:

• DynamoDB Pricing
• Single Table Design

3. Infrastructure as code saves your future self

I started clicking around the AWS console. It worked until I needed to replicate the setup for staging.

CDK seemed like overkill initially, but having version controlled infrastructure made everything reproducible and debuggable.

// Define entire scraping infrastructure in TypeScript
const scraperFunction = new lambda.Function(this, 'Scraper', {
  runtime: lambda.Runtime.NODEJS_18_X,
  handler: 'scraper.handler',
  code: lambda.Code.fromAsset('lambda'),
  timeout: Duration.minutes(5),
  memorySize: 1024,
})

// Schedule runs every 6 hours
new events.Rule(this, 'ScraperSchedule', {
  schedule: events.Schedule.rate(Duration.hours(6)),
  targets: [new targets.LambdaFunction(scraperFunction)],
})

Resources:

• AWS CDK Documentation
• CDK Patterns

4. Scrapers break constantly

Websites change their HTML structure without warning. I built monitoring and alerting so I know when scrapers fail, and I designed the system to gracefully handle partial failures without losing data.

// Graceful error handling with partial success
async function scrapeProducts(urls: string[]) {
  const results = await Promise.allSettled(urls.map(url => scrapeOne(url)))

  const successful = results.filter(r => r.status === 'fulfilled')
  const failed = results.filter(r => r.status === 'rejected')

  // Log failures but don't lose successful scrapes
  if (failed.length > 0) {
    await alertSlack(`${failed.length}/${urls.length} scrapes failed`)
  }

  return successful.map(r => r.value)
}

Resources:

• CloudWatch Alarms
• Dead Letter Queues

5. Memory equals speed equals cost

Lambda pricing ties memory and CPU together. Sometimes allocating more memory actually costs less because the function runs faster.

Sweet spot: 1024 MB costs the same ($0.0025) but runs 6x faster than 128 MB.

I experimented to find the sweet spot for each function.

Resources:

• Lambda Power Tuning
• Lambda Pricing

Polite scraping

I built in rate limiting, request delays, and respect for robots.txt. Partly that is the ethical thing to do, and partly it is self-preservation: aggressive scraping gets your IP banned fast.

const rateLimiter = new Bottleneck({
  minTime: 2000, // 2 second delay between requests
  maxConcurrent: 1, // One request at a time per domain
})

// Respect robots.txt
const robotsParser = await fetchRobotsTxt(domain)
if (!robotsParser.isAllowed(url)) {
  console.log(`Skipping ${url} - blocked by robots.txt`)
  return null
}

The bigger realization

Building this scraper taught me that serverless isn't just "functions in the cloud." It's a different way of thinking about architecture. You design around events, embrace eventual consistency, and accept that distributed systems have distributed problems.

What I would do differently

Start with Step Functions for orchestration. I cobbled together Lambda triggers and SQS queues manually. Step Functions would have given me better visibility into the pipeline and easier error handling.

References

• AWS Lambda Documentation
• DynamoDB Best Practices
• AWS CDK Documentation
• Serverless Patterns
• Lambda Power Tuning

Links

• GitHub: aws-product-scraper