<h2>The Problem</h2>
<p>I needed to collect product data from multiple websites regularly. Running scrapers on my local machine wasn't sustainable. Leaving a server running 24/7 for jobs that run a few times a day seemed wasteful.</p>
<p>This seemed like a perfect use case for serverless. Pay only when the scraper runs, scale automatically when needed, and not worry about server maintenance.</p>
<h2>What I Built</h2>
<p>A fully serverless scraping pipeline on AWS:</p>
<ul>
<li><strong><a href="https://aws.amazon.com/lambda/">Lambda</a></strong> functions that scrape target websites</li>
<li><strong><a href="https://aws.amazon.com/eventbridge/">EventBridge</a></strong> schedules to trigger scrapes automatically</li>
<li><strong><a href="https://aws.amazon.com/sqs/">SQS</a></strong> queues to handle distributed processing</li>
<li><strong><a href="https://aws.amazon.com/dynamodb/">DynamoDB</a></strong> for storing scraped data</li>
<li><strong><a href="https://aws.amazon.com/s3/">S3</a></strong> for raw files and images</li>
</ul>
<pre><code>flowchart LR
    subgraph Triggers["Scheduling"]
        EB[EventBridge&#x3C;br/>Cron Schedule]
    end

    subgraph Scraping["Data Collection"]
        L1[Lambda&#x3C;br/>Scraper]
        S3[(S3&#x3C;br/>Raw Data)]
    end

    subgraph Processing["Data Processing"]
        SQS[[SQS Queue]]
        L2[Lambda&#x3C;br/>Processor]
    end

    subgraph Storage["Persistence"]
        DDB[(DynamoDB&#x3C;br/>Clean Data)]
    end

    EB -->|"triggers"| L1
    L1 -->|"stores raw"| S3
    L1 -->|"queues jobs"| SQS
    SQS -->|"triggers"| L2
    L2 -->|"reads"| S3
    L2 -->|"writes clean"| DDB
</code></pre>
<h3>Tech Stack</h3>
<table>
<thead>
<tr>
<th>Layer</th>
<th>Technology</th>
<th>Why I Chose It</th>
</tr>
</thead>
<tbody>
<tr>
<td>Compute</td>
<td><a href="https://aws.amazon.com/lambda/">AWS Lambda</a></td>
<td>Pay per execution, auto scaling</td>
</tr>
<tr>
<td>Scheduler</td>
<td><a href="https://aws.amazon.com/eventbridge/">EventBridge</a></td>
<td>Cron expressions, serverless</td>
</tr>
<tr>
<td>Queue</td>
<td><a href="https://aws.amazon.com/sqs/">SQS</a></td>
<td>Decoupled processing</td>
</tr>
<tr>
<td>Database</td>
<td><a href="https://aws.amazon.com/dynamodb/">DynamoDB</a></td>
<td>Serverless NoSQL</td>
</tr>
<tr>
<td>Storage</td>
<td><a href="https://aws.amazon.com/s3/">S3</a></td>
<td>Raw file storage</td>
</tr>
<tr>
<td>Infrastructure</td>
<td><a href="https://aws.amazon.com/cdk/">AWS CDK</a></td>
<td>TypeScript infra as code</td>
</tr>
</tbody>
</table>
<h2>What Building This Taught Me</h2>
<h3>1. Cold starts are real and they hurt</h3>
<p>My first Lambda functions took 3 to 5 seconds to start from cold. For scraping, that added up fast.</p>
<p>I learned to optimize bundle sizes, use <a href="https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html">provisioned concurrency</a> for critical paths, and accept that some latency is just part of serverless life.</p>
<pre><code>// Before: 3-5 second cold starts
import * as AWS from 'aws-sdk' // Huge bundle

// After: &#x3C;1 second cold starts
import { DynamoDB } from '@aws-sdk/client-dynamodb' // Tree-shakeable
</code></pre>
<p>Resources:</p>
<ul>
<li><a href="https://docs.aws.amazon.com/lambda/latest/operatorguide/cold-starts.html">Lambda Cold Start Optimization</a></li>
<li><a href="https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/">AWS SDK v3 (Modular)</a></li>
</ul>
<h3>2. DynamoDB pricing is tricky</h3>
<p>I assumed "pay per request" meant cheap. Then I saw my first bill.</p>
<p><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html">Scan operations</a> are expensive. Provisioned capacity vs on demand changes costs dramatically. I rewrote queries multiple times to reduce costs.</p>
<pre><code>xychart-beta
    title "DynamoDB Cost: 1M Reads/Month"
    x-axis ["On-Demand", "Provisioned", "With GSI Scans"]
    y-axis "Cost ($)" 0 --> 16
    bar [1.25, 0.50, 15.00]
</code></pre>
<p>The lesson: design your access patterns first, then design your table. Not the other way around.</p>
<p>Resources:</p>
<ul>
<li><a href="https://aws.amazon.com/dynamodb/pricing/">DynamoDB Pricing</a></li>
<li><a href="https://www.alexdebrie.com/posts/dynamodb-single-table/">Single Table Design</a></li>
</ul>
<h3>3. Infrastructure as code saves your future self</h3>
<p>I started clicking around the AWS console. It worked until I needed to replicate the setup for staging.</p>
<p><a href="https://aws.amazon.com/cdk/">CDK</a> seemed like overkill initially, but having version controlled infrastructure made everything reproducible and debuggable.</p>
<pre><code>// Define entire scraping infrastructure in TypeScript
const scraperFunction = new lambda.Function(this, 'Scraper', {
  runtime: lambda.Runtime.NODEJS_18_X,
  handler: 'scraper.handler',
  code: lambda.Code.fromAsset('lambda'),
  timeout: Duration.minutes(5),
  memorySize: 1024,
})

// Schedule runs every 6 hours
new events.Rule(this, 'ScraperSchedule', {
  schedule: events.Schedule.rate(Duration.hours(6)),
  targets: [new targets.LambdaFunction(scraperFunction)],
})
</code></pre>
<p>Resources:</p>
<ul>
<li><a href="https://docs.aws.amazon.com/cdk/v2/guide/home.html">AWS CDK Documentation</a></li>
<li><a href="https://cdkpatterns.com/">CDK Patterns</a></li>
</ul>
<h3>4. Scrapers break constantly</h3>
<p>Websites change their HTML structure without warning. I built monitoring and alerting so I know when scrapers fail, and I designed the system to gracefully handle partial failures without losing data.</p>
<pre><code>// Graceful error handling with partial success
async function scrapeProducts(urls: string[]) {
  const results = await Promise.allSettled(urls.map(url => scrapeOne(url)))

  const successful = results.filter(r => r.status === 'fulfilled')
  const failed = results.filter(r => r.status === 'rejected')

  // Log failures but don't lose successful scrapes
  if (failed.length > 0) {
    await alertSlack(`${failed.length}/${urls.length} scrapes failed`)
  }

  return successful.map(r => r.value)
}
</code></pre>
<p>Resources:</p>
<ul>
<li><a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html">CloudWatch Alarms</a></li>
<li><a href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html">Dead Letter Queues</a></li>
</ul>
<h3>5. Memory equals speed equals cost</h3>
<p>Lambda pricing ties memory and CPU together. Sometimes allocating more memory actually costs less because the function runs faster.</p>
<pre><code>xychart-beta
    title "Lambda Memory vs Duration Tradeoff"
    x-axis ["128 MB", "512 MB", "1024 MB"]
    y-axis "Duration (seconds)" 0 --> 12
    bar [10, 3, 1.5]
</code></pre>
<p><em>Sweet spot: 1024 MB costs the same ($0.0025) but runs 6x faster than 128 MB.</em></p>
<p>I experimented to find the sweet spot for each function.</p>
<p>Resources:</p>
<ul>
<li><a href="https://github.com/alexcasalboni/aws-lambda-power-tuning">Lambda Power Tuning</a></li>
<li><a href="https://aws.amazon.com/lambda/pricing/">Lambda Pricing</a></li>
</ul>
<h2>Polite Scraping</h2>
<p>I built in rate limiting, request delays, and respect for robots.txt. Not just to be ethical, but because aggressive scraping gets your IP banned fast.</p>
<pre><code>const rateLimiter = new Bottleneck({
  minTime: 2000, // 2 second delay between requests
  maxConcurrent: 1, // One request at a time per domain
})

// Respect robots.txt
const robotsParser = await fetchRobotsTxt(domain)
if (!robotsParser.isAllowed(url)) {
  console.log(`Skipping ${url} - blocked by robots.txt`)
  return null
}
</code></pre>
<h2>The Bigger Realization</h2>
<p>Building this scraper taught me that serverless isn't just "functions in the cloud." It's a different way of thinking about architecture. You design around events, embrace eventual consistency, and accept that distributed systems have distributed problems.</p>
<h2>What I'd Do Differently</h2>
<p>Start with <a href="https://aws.amazon.com/step-functions/">Step Functions</a> for orchestration. I cobbled together Lambda triggers and SQS queues manually. Step Functions would have given me better visibility into the pipeline and easier error handling.</p>
<h2>References</h2>
<ul>
<li><a href="https://docs.aws.amazon.com/lambda/">AWS Lambda Documentation</a></li>
<li><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/best-practices.html">DynamoDB Best Practices</a></li>
<li><a href="https://docs.aws.amazon.com/cdk/v2/guide/home.html">AWS CDK Documentation</a></li>
<li><a href="https://serverlessland.com/patterns">Serverless Patterns</a></li>
<li><a href="https://github.com/alexcasalboni/aws-lambda-power-tuning">Lambda Power Tuning</a></li>
</ul>
<h2>Links</h2>
<ul>
<li><strong>GitHub:</strong> <a href="https://github.com/Cyvid7-Darus10/aws-product-scraper">aws-product-scraper</a></li>
</ul>


Projects.

AWS Product Scraper: Serverless Data Pipeline