3  Best Practices

3.1 Scraper Design

3.2 Developer Ergonomics

3.3 Being a Good Citizen

3.3.1 Scraping != Adversarial

3.3.2 Identifying Yourself

3.3.3 robots.txt

3.3.4 Rate Limiting

What is Rate Limiting?

Naive Rate Limiting

Proper Rate Limiting

Enhancements

3.3.5 Caching

HTTP Caching

Caching for Scrapers

Caveats