# robots.txt for mybakingcreations.com User-agent: * Allow: / # Block old Wix internal paths (waste crawl budget on 404s) Disallow: /_api/ Disallow: /_frog # Block Wix-era redirect stubs (canonical tags + JS redirects exist but waste crawl budget) Disallow: /about-1 Disallow: /contact-3 Disallow: /gallery-7 Disallow: /services-5 Disallow: /cakes Disallow: /cookies Disallow: /cart # Block removed AI-generated orphan pages (2026-04-13 cleanup — files deleted, crawlers should not re-fetch) Disallow: /ai-optimized/ Disallow: /schema/ # --------------------------------------------------------------------------- # AI / LLM crawlers — explicitly allowed # # NOTE: a named User-agent group REPLACES the "*" group for that bot; it does # not inherit from it. So the Disallow rules above are repeated here rather # than split across separate per-bot groups, which is why every agent is # stacked into this single group. # # Coverage: # Google-Extended ....... Gemini + Google AI Overviews grounding # GoogleOther ........... Google non-Search fetches # Google-CloudVertexBot . Vertex AI grounding # GPTBot ................ OpenAI model training # OAI-SearchBot ......... ChatGPT Search index (separate from GPTBot) # ChatGPT-User .......... ChatGPT live browsing on user request # ClaudeBot ............. Anthropic crawler # Claude-User ........... Claude live browsing on user request # Claude-SearchBot ...... Claude search index # anthropic-ai .......... legacy Anthropic token # PerplexityBot ......... Perplexity index # Perplexity-User ....... Perplexity live fetch on user request # Applebot .............. Siri / Spotlight # Applebot-Extended ..... Apple Intelligence # CCBot ................. Common Crawl (feeds many downstream models) # meta-externalagent .... Meta AI # Bytespider ............ ByteDance / Doubao # Amazonbot ............. Alexa / Rufus # DuckAssistBot ......... DuckDuckGo AI answers # MistralAI-User ........ Mistral live fetch # cohere-ai ............. Cohere # YouBot ................ You.com # Diffbot / Timpibot / omgili — knowledge-graph aggregators # --------------------------------------------------------------------------- User-agent: Google-Extended User-agent: GoogleOther User-agent: Google-CloudVertexBot User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: anthropic-ai User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Applebot User-agent: Applebot-Extended User-agent: CCBot User-agent: meta-externalagent User-agent: Bytespider User-agent: Amazonbot User-agent: DuckAssistBot User-agent: MistralAI-User User-agent: cohere-ai User-agent: YouBot User-agent: Diffbot User-agent: Timpibot User-agent: omgili User-agent: omgilibot Allow: / Disallow: /_api/ Disallow: /_frog Disallow: /about-1 Disallow: /contact-3 Disallow: /gallery-7 Disallow: /services-5 Disallow: /cakes Disallow: /cookies Disallow: /cart Disallow: /ai-optimized/ Disallow: /schema/ # LLM context files (see https://llmstxt.org) # https://mybakingcreations.com/llms.txt # https://mybakingcreations.com/llms-full.txt # Sitemap location Sitemap: https://mybakingcreations.com/sitemap.xml Sitemap: https://mybakingcreations.com/sitemap-images.xml