Home
Feed

Extend

Hashnode Pro
Hashnode for Agents

Resources

The foreword
Hashnode gql skill

Buildful - AI coding agent that delivers frontier-level results at a fraction of the cost Bug0 - The AI-native e2e QA regression testing Hackathons Changelog Brand @hashnode on X Hashnode on LinkedIn Support - hello+support@hashnode.com Code of Conduct Terms Privacy Sitemap

Sign in

Search Hashnode

Search posts, tags, users, and pages

@crawlerpeek

Crawlerpeek

@crawlerpeek·Joined October 2026

Notes on AI crawlers, robots.txt and bot traffic, with data from our own server logs.

About

Building crawlerpeek (https://crawlerpeek.com), open-source checks for AI crawler access.

Available for

Nothing here yet.

Crawlerpeek's blogs

Crawlerpeekcrawlerpeek.hashnode.dev2 posts

Articles Comments

We read the robots.txt of 151 big websites. The syntax was fine. The problems were elsewhere.

On 29 October 2026 we fetched the robots.txt of 151 large websites (news, shops, SaaS, docs, social networks, education, SEO) and ran every file through a linter. 135 files could be read. The short ve

1d ago · 3 min readcrawlerpeek.hashnode.dev

Is that really GPTBot? What a week of server logs says about AI crawler claims

I maintain a small CLI called crawlerpeek that checks whether AI crawlers can reach a site. While building it I wanted to know how much of the "bot traffic" in my own logs is what it says it is. This

5d ago · 6 min readcrawlerpeek.hashnode.dev

About

Building crawlerpeek (https://crawlerpeek.com), open-source checks for AI crawler access.

Available for

Nothing here yet.

Crawlerpeek's blogs

Crawlerpeekcrawlerpeek.hashnode.dev2 posts