Tag

#robotstxt

15 posts0 followers

Explore Hashnode

Alternatives

The New Shared robots.txt Token for AI Training, Explained in 5 Minutes

A single robots.txt group may soon tell every major AI provider not to train on your site. On Oct 30, 2026, Google said it will support a shared "policy token" from the IETF AIPREF working group. Micr

hafizwaleedahmed5.hashnode.dev

Jacob CampbellOct 23 · 2 min read

noindex vs robots.txt: Which to Use and the Mistake to Avoid

noindex is for keeping a page out of results; a robots.txt Disallow is for stopping crawlers from requesting URLs. The two aren't interchangeable. A disallowed URL can still appear in Google without a

built2win.hashnode.dev

Mayank SharmaOct 12 · 6 min read

robots.txt for the AI era: copy-paste templates for GPTBot, ClaudeBot, PerplexityBot and Google-Extended

Most robots.txt files I review were written for one crawler: Googlebot. Today the same file is read by a dozen AI agents, each run by a different company and each doing a different job. One line in th

mayank-sharma.hashnode.dev

Muhammed Veysel ErinOct 1 · 6 min read

Designing a test for the gap between a permission file and a server

Most write-ups of a crawler-access study lead with the finding. This one is about the design that had to be right before the finding meant anything. The question was narrow: when a website publishes a

citevio.hashnode.dev

Yogeshwar PeelaAug 7 · 11 min read

TryHackMe : Infinity Pool Writeup

Summary Recon on <MACHINE_IP> revealed a Gunicorn-hosted "Byte Lotus" hotel site with two paths disallowed in robots.txt - /internal/ and /status. The /status page exposes an internal staff tool ("Sis

1 exploitnotes.hashnode.dev

Jaxon ParrottJul 23 · 7 min read

The Two-Layer AI Crawler Access Stack Every Developer Should Audit

Most AI visibility advice starts with the wrong question. "Is your content structured for AI citations?" "Are you using the right schema markup?" "Is your prose extractable?" All of that is downstream

1 authoritytech.hashnode.dev

Yogeshwar PeelaJul 12 · 5 min read

BroncoCTF: Lovely Login

Executive Summary Lovely Login presents a minimal "Secure Database" login form backed by an Express API at /login. The obvious attack surface — NoSQL operator injection on username/password — turned o

exploitnotes.hashnode.dev

Bhavin ShethJun 22 · 14 min read

What is Robots.txt? How to Generate One for Free in 2026

A startup founder reached out to me in a panic. Three months after launching, their site had almost no organic traffic despite genuinely good content and a clean technical setup. We checked indexing s

allinonetools.hashnode.dev

Akash BijweDec 18, 2025 · 2 min read

Robots.txt Generator — Because Crawlers Should Respect Your Rules 🤖📜

Robots.txt — it sounds like ancient robot law… and honestly, sometimes it feels like it. You want search engines to crawl your site the right way — but writing robot rules by hand can quickly turn into a guessing game: User-agent: * Disallow: /admin/...

devutilx.hashnode.dev