# uzufrukt (azlaw.ulvialiyev.com) — robots.txt # # POLICY (reviewed 2026-08-04). One line: a fetch that ends in a CITATION is # welcome; a fetch that ends in TRAINING DATA is not. # # An assistant that retrieves this page to answer someone's question names # uzufrukt as the source and sends that reader here. A bulk training crawler # copies the corpus into model weights, after which the answer is given # without us and nobody arrives. The legislation + court-act corpus # (article-level citations across ~1.7M court acts) is the product; it comes # back from nobody once absorbed. # # Google-Extended and Applebot-Extended were BLOCKED here until 2026-08-04 and # are now allowed. Neither is a crawler: they are content-usage tokens that # Googlebot / Applebot consult after the ordinary crawl. Set to `Disallow: /` # they did not stop a single fetch — they only withheld permission for the # grounded, source-linking answers Gemini apps and Apple Intelligence produce. # That is the citation surface this whole file exists to win, so blocking them # cost the upside and protected nothing. GPTBot, ClaudeBot, CCBot, Bytespider, # Amazonbot and meta-externalagent are a different thing entirely — real bulk # fetchers feeding training corpora — and stay fully blocked. # # robots.txt group matching is EXCLUSIVE: a crawler that finds its own # `User-agent:` group obeys that group ONLY and never falls back to # `User-agent: *`. So every named allowed group below repeats the full # Disallow list (_SEO_ROBOTS_RESTRICTED). Naming a bot without repeating them # would hand it /admin, /account and /api — the opposite of what the line # looks like it does. tests/test_seo_discovery.py parses this file into groups # and asserts that property; do not "tidy" the repetition away. # # /admin, /account and /api are private/operational; /search and any # "?q="-bearing URL are live-DB search results (expensive, not canonical # content — the canonical article text lives at /law//maddə/). # # Bot classifications reviewed against each provider's own published crawler # docs where available. Provider bot names and stated purposes change over # time — re-verify periodically, don't trust this file blindly. # Default group: ordinary search engines + anything not named below. User-agent: * Disallow: /admin Disallow: /account Disallow: /api Disallow: /search Disallow: /*?q= Allow: / # --- Allowed, same limits as everyone else. Two kinds live in this section: # retrieval / user-initiated-fetch CRAWLERS (they fetch a page to answer a # question and cite it), and content-usage TOKENS (Google-Extended, # Applebot-Extended — not crawlers at all; they gate whether an already- # fetched page may be used in a grounded, source-linking answer). --- # OpenAI: ChatGPT-User fetches a single page live for a user's request; # OAI-SearchBot crawls to power ChatGPT's search/citation feature. Neither # is OpenAI's training crawler (that's GPTBot, blocked below). User-agent: ChatGPT-User Disallow: /admin Disallow: /account Disallow: /api Disallow: /search Disallow: /*?q= Allow: / User-agent: OAI-SearchBot Disallow: /admin Disallow: /account Disallow: /api Disallow: /search Disallow: /*?q= Allow: / # Anthropic: Claude-User fetches a page live for a user's Claude session # (e.g. its web-fetch/browsing tool); Claude-SearchBot supports Claude's # search/citation feature. ClaudeBot (blocked below) is the bulk training # crawler. User-agent: Claude-User Disallow: /admin Disallow: /account Disallow: /api Disallow: /search Disallow: /*?q= Allow: / User-agent: Claude-SearchBot Disallow: /admin Disallow: /account Disallow: /api Disallow: /search Disallow: /*?q= Allow: / # Perplexity: Perplexity-User fetches a page live for a user's query; # PerplexityBot crawls to build Perplexity's cited-answer index. User-agent: Perplexity-User Disallow: /admin Disallow: /account Disallow: /api Disallow: /search Disallow: /*?q= Allow: / User-agent: PerplexityBot Disallow: /admin Disallow: /account Disallow: /api Disallow: /search Disallow: /*?q= Allow: / # Meta: Meta-ExternalFetcher is the user-initiated fetch (someone shares or # asks about a link in Meta AI / WhatsApp / Instagram). meta-externalagent, # blocked below, is the training crawler. User-agent: Meta-ExternalFetcher Disallow: /admin Disallow: /account Disallow: /api Disallow: /search Disallow: /*?q= Allow: / # Amazon split one token into three. Amazonbot (blocked below) is # training-eligible per Amazon's own docs; these two are not — Amzn-SearchBot # indexes for Alexa/Rufus search surfaces and Amzn-User is a live user-triggered # fetch. Blocking all three would cut off Amazon's citation surfaces along with # the training we actually object to. User-agent: Amzn-SearchBot Disallow: /admin Disallow: /account Disallow: /api Disallow: /search Disallow: /*?q= Allow: / User-agent: Amzn-User Disallow: /admin Disallow: /account Disallow: /api Disallow: /search Disallow: /*?q= Allow: / # Google: Google-Extended is the permission Gemini apps consult before # grounding an answer in a page Googlebot already fetched. Allowing it is what # lets Gemini quote and link uzufrukt; it is not a separate crawl and it is not # a Search ranking signal either way. GPTBot-style bulk training is a different # token and is not this one. User-agent: Google-Extended Disallow: /admin Disallow: /account Disallow: /api Disallow: /search Disallow: /*?q= Allow: / # Apple: Applebot-Extended is the same shape for Apple Intelligence — a usage # permission checked after Applebot's ordinary crawl, not a crawler of its own. User-agent: Applebot-Extended Disallow: /admin Disallow: /account Disallow: /api Disallow: /search Disallow: /*?q= Allow: / # --- Bulk AI training crawlers: fully blocked --- User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / User-agent: meta-externalagent Disallow: / Sitemap: https://uzufrukt.com/sitemap.xml