User-agent: * Disallow: Allow: / User-agent: Googlebot Disallow: Allow: / User-agent: Bingbot Disallow: Allow: / User-agent: Slurp Disallow: Allow: / User-agent: DuckDuckGoBot Disallow: Allow: / User-agent: YandexBot Disallow: Allow: / User-agent: Baiduspider Disallow: Allow: / User-agent: SogouSpider Disallow: Allow: / User-agent: Exabot Disallow: Allow: / User-agent: FacebookExternalHit Disallow: Allow: / User-agent: Twitterbot Disallow: Allow: / User-agent: Applebot Disallow: Allow: / User-agent: LinkedInBot Disallow: Allow: / User-agent: Pinterestbot Disallow: Allow: / User-agent: ia_archiver Disallow: Allow: / User-agent: MJ12bot Disallow: Allow: / User-agent: AhrefsBot Disallow: Allow: / User-agent: SemrushBot Disallow: Allow: / User-agent: DotBot Disallow: Allow: / User-agent: Ryze Disallow: Allow: / User-agent: SeznamBot Disallow: Allow: / User-agent: CCBot Disallow: Allow: / User-agent: ZoominfoBot Disallow: Allow: / User-agent: Uptimebot Disallow: Allow: / User-agent: Gigablast Disallow: Allow: / User-agent: Qwantify Disallow: Allow: / User-agent: LTX71 Disallow: Allow: / User-agent: Ecosia Disallow: Allow: / User-agent: PetalBot Disallow: Allow: / User-agent: SMTBot Disallow: Allow: / User-agent: ToutiaoSpider Disallow: Allow: / User-agent: Seoscanners Disallow: Allow: / User-agent: Cliqzbot Disallow: Allow: / User-agent: IstellaBot Disallow: Allow: / User-agent: Screaming Frog SEO Spider Disallow: Allow: / User-agent: BLEXBot Disallow: Allow: / User-agent: NetcraftSurveyAgent Disallow: Allow: / User-agent: Dataprovider.com Disallow: Allow: / User-agent: MojeekBot Disallow: Allow: / User-agent: StackRambler Disallow: Allow: / User-agent: Woocommerce Disallow: Allow: / User-agent: SearchAtlas Disallow: Allow: / User-agent: Plukkie Disallow: Allow: / User-agent: Cocolyzebot Disallow: Allow: / User-agent: Dragonbot Disallow: Allow: / User-agent: NerdyBot Disallow: Allow: / User-agent: Facebot Disallow: Allow: / # --- AI/LLM-Specific Bots (Top 20) --- User-agent: GPTBot Disallow: User-agent: ChatGPT-User Disallow: User-agent: ChatGPT-OpenAI Disallow: User-agent: ClaudeBot Disallow: User-agent: anthropic-ai Disallow: User-agent: CCBot Disallow: User-agent: CommonCrawl Disallow: User-agent: Yeti Disallow: User-agent: cohere-ai Disallow: User-agent: PerplexityBot Disallow: User-agent: UpstreamBot Disallow: User-agent: Amazonbot Disallow: User-agent: NeevaBot Disallow: User-agent: YouBot Disallow: User-agent: ReacherBot Disallow: User-agent: Diffbot Disallow: User-agent: MetaBot Disallow: User-agent: Apple-MLBot Disallow: User-agent: AlephAlpha Disallow: User-agent: SearxBot Disallow: Sitemap: https://www.shelfperks.com/sitemap.xml