The Scraper's Journey

15 May, 2025

Act 1

Eddie stared at his terminal in frustration as his simple script failed once again. What had started as innocent curiosity-just wanting to map out documentation sites without dealing with their bloated JavaScript interfaces-had turned into hours of frustration.

"All I wanted was a damn sitemap yo"

he muttered, watching another 403 Forbidden error flash on his screen. His basic requests.get() approach worked fine on his personal blog but crumbled against any site with even basic protection.

The Stack Overflow answers weren't helping:

Just add a User-Agent header.

Use Beautiful Soup.

None addressed the modern web's complexity. His CSV files sat nearly empty, missing the very data he needed.

Act 2

That night, Eddie discovered a post about headless browsers. The concept was elegant; instead of trying to parse raw HTML, why not let a browser render everything first?

"So that's how they do it," he whispered, reading about Selenium and Playwright. The code examples showed browsers loading pages silently, executing JavaScript, and then extracting the fully rendered content.

He spent days experimenting with API interception too, watching network requests in Chrome DevTools, noticing patterns in how sites loaded data. Sometimes the content wasn't in the HTML at all but fetched separately through JSON endpoints.

"The data was there all along!" he realized. "Just not where I was looking."

Act 3

Weeks passed. Eddie's simple sitemap script evolved into something more sophisticated; a hybrid system that could map a site through its sitemap.xml, then strategically render pages when needed, and even intercept API calls directly.

His code now included proper delays between requests. It rotated between different user agents. It respected robots.txt files religiously. It backed off when sites seemed overwhelmed.

"I'm not taking anything they wouldn't give a normal visitor," he reminded himself. "I'm just being more efficient about it."

The transformation wasn't just in his code but in his understanding. The web wasn't just HTML; it was a complex ecosystem of APIs, rendering engines, and protection systems. By understanding them, he could work with them rather than against them.

Act 4

Eddie sat in the glow of his monitor at 3 AM, watching as his crawler methodically indexed technical documentation across dozens of sites. Each page rendered, its content extracted, its links followed.

With great scrapes comes IP overdosage. Several times, he'd had to throttle back his scraper when he noticed it was hitting sites too hard. Once, he'd accidentally followed a pattern that generated infinite URLs, nearly taking down a small documentation site.

"This is a conversation, not a siege" he reminded himself, adding more safeguards to his code.

The pain of change wasn't technical but ethical. He'd built something powerful; Theoretically capable of scraping vast portions of the web. He chose to use it judiciously. He added filters to only collect what he actually needed. He cached aggressively to minimize repeat requests.

And when he finally looked at his local storage...exabytes of perfectly organized, searchable technical documentation; he felt not triumph over the systems he'd circumvented, but gratitude for the knowledge they shared.

"The internet isn't something to conquer" he thought, closing his laptop as dawn broke. "It's something to commune with."

His journey had come full circle: from a simple curiosity about sitemaps to a deeper understanding of the web itself. The power wasn't in having the internet in his local storage; it was in knowing how to become the AI yourself.

Act 1: This is me and it's not working

Validation

def check_robots_txt(url):
    parsed_url = urlparse(url)
    base_url = f"{parsed_url.scheme}://{parsed_url.netloc}"
    robots_url = f"{base_url}/robots.txt"

Sitemap

def get_sitemap(url):
    if not url.endswith('sitemap.xml'):
        if url.endswith('/'):
            url = url + 'sitemap.xml'
        else:
            url = url + '/sitemap.xml'

XML

soup = BeautifulSoup(response.content, 'xml')

# sitemaps of sitemaps
sitemapTags = soup.find_all('sitemap')
if sitemapTags:
    # Handle sitemap index
    ...
else:
    # Handle regular sitemap
    ...

This implements a crucial pattern in sitemap crawling - the ability to handle sitemap index files. Large websites often use a hierarchical structure.

Extraction

urls = []
for url_tag in soup.find_all('url'):
    loc = url_tag.find('loc').text
    lastmod = url_tag.find('lastmod').text if url_tag.find('lastmod') else None
    
    url_data = {'url': loc, 'lastmod': lastmod}
    urls.append(url_data)

The extraction focuses on:

• <loc> - The actual URL (required by the protocol)
• <lastmod> - When the content was last modified (optional)

Potentially useful fields:

• <changefreq> - How often the page changes
• <priority> - Relative importance within the site

Current Limitations

1. Depth: Only extracts URLs without visiting them
2. Completeness: Relies entirely on the sitemap (which may be incomplete)
3. Authentication: Can't access protected content
4. JavaScript Rendering: Can't render JavaScript-based content
5. Rate Limiting: No built-in throttling or delay mechanism

Blocking Mechanisms

1. Request patterns - Too many requests too quickly
2. Browser fingerprinting - Canvas fingerprinting, WebGL fingerprinting, etc.
3. Mouse movement analysis - Humans move mice in non-linear patterns
4. CAPTCHA systems - Especially for login-protected content
5. HTTP header inspection - Missing or incorrect headers

Act 2: Is there another way

1. Proper delays - Not just between requests but with randomized timing
2. Complete header sets - Including cookies, referrers, etc.
3. Proxy rotation - Using residential proxies (more expensive but less detectable)
4. Browser environment spoofing - Mimicking real browser fingerprints
5. CAPTCHA solving services - Either automated or human-powered (ethically questionable)

Content Fetching

def fetch_page_content(url):
    response = requests.get(url)
    if response.status_code == 200:
        soup = BeautifulSoup(response.content, 'html.parser')
        return soup
    return None

Throttling & Politeness

import time

def polite_request(url, delay=1.0):
    time.sleep(delay)  # Wait between requests
    return requests.get(url)

Breadth-First Crawling

def crawl_site(start_url, max_pages=100):
    visited = set()
    queue = [start_url]
    results = []
    
    while queue and len(visited) < max_pages:
        url = queue.pop(0)
        if url in visited:
            continue
            
        visited.add(url)
        page = fetch_page_content(url)
        if not page:
            continue
            
        content = extract_data(page)
        results.append(content)
        
        links = page.find_all('a', href=True)
        for link in links:
            absolute_url = urljoin(url, link['href'])
            if is_valid_url(absolute_url) and absolute_url not in visited:
                queue.append(absolute_url)
                
    return results

Content Parsing & Storage

def extract_data(soup):
    data = {}
    
    title_tag = soup.find('title')
    if title_tag:
        data['title'] = title_tag.text.strip()
    
    content_div = soup.find('div', {'class': 'main-content'})
    if content_div:
        data['content'] = content_div.text.strip()
        
    return data

Act 3: I have transformed

The JavaScript Challenge

Single Page Applications (SPAs) or heavy JavaScript applications:

1. You get a minimal HTML shell
2. JavaScript executes to fetch data and render the actual content
3. The DOM is built dynamically in your browser

Common solutions to modern problems:

1. Headless Browsers

   • Tools like Puppeteer, Playwright, or Selenium
   • They launch actual Chrome/Firefox instances without a UI
   • Let JavaScript fully execute before extracting content
   • Can interact with the page (click, scroll, type)
   • Resource-intensive, slow, complex to manage at scale

2. API Interception (Elegant but Requires Analysis)

   • Instead of scraping the rendered page, identify and call the same APIs the page uses
   • Much faster and lighter than headless browsers
   • Requires reverse engineering the site's API structure, which changes frequently

3. Best of Both

   • Use headless browsers to understand API patterns
   • Then switch to direct API calls for production scraping
   • Efficient once set up
   • More complex implementation

Selenium example

from selenium import webdriver

def render_javascript(url):
    driver = webdriver.Chrome()
    driver.get(url)
    time.sleep(3)
    
    html = driver.page_source
    driver.quit()
    
    return BeautifulSoup(html, 'html.parser')

Act 4: Can I handle the pain of change

Parallel Processing

from concurrent.futures import ThreadPoolExecutor

def parallel_crawl(urls, max_workers=5):
    results = []
    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        future_to_url = {executor.submit(fetch_page_content, url): url for url in urls}
        for future in as_completed(future_to_url):
            url = future_to_url[future]
            try:
                data = future.result()
                results.append(data)
            except Exception as exc:
                print(f'{url} generated an exception: {exc}')
    return results

Fingerprint Management

To avoid getting blocked:

import random

user_agents = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15',
] 

def get_headers(): # for rotating user agents
    return {
        'User-Agent': random.choice(user_agents),
        'Accept': 'text/html,application/xhtml+xml,application/xml',
        'Accept-Language': 'en-US,en;q=0.9',
    }

Incremental Crawling

To efficiently update your dataset:

def incremental_crawl(site_url, previous_data):
    sitemap_data = get_sitemap(site_url)
    
    urls_to_crawl = []
    for item in sitemap_data:
        url = item['url']
        lastmod = item.get('lastmod')
        
        if url not in previous_data or previous_data[url]['lastmod'] != lastmod:
            urls_to_crawl.append(url)
            
    new_data = crawl_pages(urls_to_crawl)
    
    for url, data in new_data.items():
        previous_data[url] = data
        
    return previous_data

✌️🔚

What Will Get You Blocked

1. Hammering the server - Rapid, repeated requests
2. Ignoring response codes - Not backing off when you get 429s or 503s
3. Spider traps - Following infinite URL patterns (like calendars that go to year 9999)
4. Scraping login-protected content - Especially with shared/stolen credentials
5. Scrapers from data center IPs - AWS, Digital Ocean, etc. are easily identified

The Ethical Dimension

1. Terms of Service - Many explicitly forbid scraping
2. Legal precedents - Cases like hiQ vs LinkedIn have mixed outcomes
3. Server load - Small sites can be overwhelmed by aggressive scraping
4. Content licensing - Just because it's visible doesn't mean it's free to reuse

The most sophisticated scrapers today blend in by behaving exactly like real users - they request pages at human speeds, move virtual mice in human-like patterns, and spread requests across many IP addresses to avoid detection.