Collecting data the polite way: robots.txt first, then RSS and public APIs, and dates you can trust

My site is a pile of Python scripts that run every hour: collect, process, build static HTML, git push. The collecting part is where I made most of my early mistakes, so this post is about that: what