# Chive.com robots.txt # Platform: Shopify # Last updated: 2026-09-09 # # Structure: Shopify default base + Chive-specific parameter/duplicate URL blocks # Changelog 2026-09-09: # - Added Disallow: /customer_authentication/ (+ locale twin) to *, AhrefsBot, # and AhrefsSiteAudit groups - 885 param-permuted login URLs in GSC noindex # bucket, daily crawl-budget waste (Asana 1218074860106316). Expected side # effect: some URLs may show "Indexed, though blocked by robots.txt" in # GSC - cosmetic, do not re-task. # Changelog 2026-08-26: # - Added Allow exceptions for /products/*?variant=* (Google channel feed # landing pages) — fixes GMC "Unable to check product pages" disapprovals # caused by the 2026-06-12 &-param rules (Asana 1217862679462180) # Changelog 2026-06-12: # - Replaced [a-f0-9]/[0-9] character-class rules (not supported by robots.txt # spec; matched as literal text, i.e. dead rules) in *, AhrefsBot, and # AhrefsSiteAudit groups # - Added mid-position (&-prefixed) parameter twins for variant/pr_/jdgm/ref/ # utm/srsltid/sort_by/_sid/_ss # - Added /admin and /orders to facebookexternalhit and Facebot groups # Agent instructions: https://chive.com/agents.md # Terms of Service: https://www.shopify.com/legal/terms # Contact: bots@shopify.com User-agent: * # --- Base: allow all indexable content --- Allow: / # --- Exception allow rules (paths that overlap with blocked patterns) --- Allow: /products/account Allow: /products/orders Allow: /products/checkout Allow: /*/products/account Allow: /*/products/orders Allow: /*/products/checkout Allow: /collections/account Allow: /collections/orders Allow: /collections/checkout Allow: /*/collections/account Allow: /*/collections/orders Allow: /*/collections/checkout Allow: /pages/checkout Allow: /*/pages/checkout Allow: /blogs/*account Allow: /blogs/*orders Allow: /blogs/*checkout Allow: /*/blogs/*account Allow: /*/blogs/*orders Allow: /*/blogs/*checkout Allow: /account/login Allow: /*/account/login # --- Exception: Google channel feed landing pages (added 2026-08-26) --- # The Shopify Google & YouTube channel submits every product as # /products/X?variant=...&country=...¤cy=...&utm_...=sag_organic # The 2026-06-12 &-param twins blocked these URLs, so GMC's landing-page # checker cannot fetch them -> "Unable to check product pages" disapprovals # (first seen: triangular-watermelon-ornament, Aug 26). Longest-match: # these Allows beat the six shorter Disallows for feed URLs ONLY; all # ?variant=/utm/currency blocking stays intact everywhere else. # /collections/*/products/* block untouched. Allow: /products/*?variant=* Allow: /*/products/*?variant=* # --- Private / transactional paths --- Disallow: /admin Disallow: /cart/ Disallow: /*/cart/ Disallow: /checkout Disallow: /*/checkout Disallow: /checkouts/ Disallow: /*/checkouts/ Disallow: /orders Disallow: /*/orders Disallow: /account Disallow: /*/account Disallow: /customer_authentication/ Disallow: /*/customer_authentication/ Disallow: /37556879497/checkouts Disallow: /37556879497/orders Disallow: /carts # --- Shopify internal endpoints --- Disallow: /services Disallow: /sf_* Disallow: /cdn/wpm/*.js Disallow: /web-pixels Disallow: /apple-app-site-association Disallow: /.well-known/shopify/monorail Disallow: /search/suggest Disallow: /a/downloads/-/* # --- AJAX / agent endpoints (use UCP/MCP instead) --- Disallow: /cart.js Disallow: /*/cart.js Disallow: /recommendations/products Disallow: /*/recommendations/products # --- Shopify remote product variants (internal) --- # NOTE: robots.txt does not support [a-f0-9] character classes (only * and $). # Previous rules matched nothing. Wildcard versions below; the two # /collections/ variants are technically redundant with the broader # /collections/*/products/* rule but kept for explicitness. Disallow: /products/*-remote Disallow: /*/products/*-remote Disallow: /collections/*/products/*-remote Disallow: /*/collections/*/products/*-remote # --- Duplicate collection-scoped product URLs --- # Shopify generates /collections/X/products/Y as an alternate to /products/Y # Blocking prevents crawl waste; canonical tags already handle indexation correctly Disallow: /collections/*/products/* # --- Collection facets / sort / filters --- Disallow: /collections/*sort_by* Disallow: /*/collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /collections/*filter*&*filter* Disallow: /*/collections/*filter*&*filter* Disallow: /collections/*/size_ Disallow: /collections/*/color_ Disallow: /collections/*/color- Disallow: /collections/*/price_ Disallow: /collections/*/ceramic Disallow: /collections/*/1000 Disallow: /*?sort_by=* Disallow: /*&sort_by=* Disallow: /*?filter. Disallow: /*?filter_ Disallow: /*?q=*&options=* # --- Blog facets --- Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* # --- Shopify internal session / preview params --- Disallow: /*?*oseid=* Disallow: /*?*preview_theme_id=* Disallow: /*?*preview_script_id=* Disallow: /*?*ls=*&ls=* Disallow: /*?*ls%3*ls%3* # --- Shopify predictive search params --- Disallow: /*?_pos=* Disallow: /*?_sid=* Disallow: /*?_ss=* Disallow: /*?_psq=* Disallow: /*?_v=* Disallow: /*&_sid=* Disallow: /*&_ss=* # --- Variant & currency params --- Disallow: /*?variant=* Disallow: /*&variant=* Disallow: /*?currency=* Disallow: /*¤cy=* Disallow: /*?country=* Disallow: /*&country=* # --- Rebuy / product recommendation params --- Disallow: /*?pr_* Disallow: /*&pr_* # --- Judge.me referral params --- Disallow: /*?jdgm_referral_location=* Disallow: /*&jdgm_referral_location=* Disallow: /*?ref=* Disallow: /*&ref=* # --- UTM params --- Disallow: /*?utm_source=* Disallow: /*?utm_medium=* Disallow: /*?utm_campaign=* Disallow: /*&utm_source=* Disallow: /*&utm_medium=* Disallow: /*&utm_campaign=* # --- Pinterest params --- Disallow: /*?pins_campaign_id=* # --- Shop App / list generator --- Disallow: /*?list_generator=* # --- Google Merchant / Surfaces Across Google --- Disallow: /*?srsltid=* Disallow: /*&srsltid=* # --- Email / tracking params --- # NOTE: previous /*?t=[0-9]* used an unsupported character class (dead rule). # /*?t=* below blocks ANY URL whose first query param is t= — wider than the # original intent. Delete this line if any legitimate feature uses ?t=. Disallow: /*?t=* Disallow: /*?dt_id=* # --- Misc --- Disallow: /*.atom Disallow: /agents.md Sitemap: https://chive.com/sitemap.xml # --------------------------------------------------------------------------- # Google Ads bot: ignores User-agent: * so must be explicitly named. # Explicit Allow rules ensure product/collection/page/blog crawling for ad serving # (required for PMax, Shopping, and Dynamic Search campaigns). User-agent: adsbot-google Allow: /products/ Allow: /*/products/ Allow: /collections/ Allow: /*/collections/ Allow: /pages/ Allow: /*/pages/ Allow: /blogs/ Allow: /*/blogs/ Allow: /pages/checkout Allow: /*/pages/checkout Allow: /blogs/*checkout Allow: /*/blogs/*checkout Disallow: /checkout Disallow: /*/checkout Disallow: /checkouts/ Disallow: /*/checkouts/ Disallow: /orders Disallow: /*/orders Disallow: /services Disallow: /sf_* Disallow: /37556879497/checkouts Disallow: /37556879497/orders Disallow: /cdn/wpm/*.js # --------------------------------------------------------------------------- # Meta/Facebook ad crawlers: explicitly named to bypass User-agent: * blocks # Required for Dynamic Product Ads, catalogue crawling, and pixel attribution User-agent: facebookexternalhit Allow: /products/ Allow: /*/products/ Allow: /collections/ Allow: /*/collections/ Allow: /pages/ Allow: /blogs/ Disallow: /admin Disallow: /checkout Disallow: /*/checkout Disallow: /checkouts/ Disallow: /orders Disallow: /*/orders Disallow: /cart/ Disallow: /*/cart/ Disallow: /account Disallow: /*/account Disallow: /services User-agent: Facebot Allow: /products/ Allow: /*/products/ Allow: /collections/ Allow: /*/collections/ Allow: /pages/ Allow: /blogs/ Disallow: /admin Disallow: /checkout Disallow: /*/checkout Disallow: /checkouts/ Disallow: /orders Disallow: /*/orders Disallow: /cart/ Disallow: /*/cart/ Disallow: /account Disallow: /*/account Disallow: /services # --------------------------------------------------------------------------- User-agent: Nutch Disallow: / # --------------------------------------------------------------------------- User-agent: AhrefsBot Crawl-delay: 10 Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /37556879497/checkouts Disallow: /37556879497/orders Disallow: /carts Disallow: /account Disallow: /customer_authentication/ Disallow: /*/customer_authentication/ Disallow: /a/downloads/-/* Disallow: /collections/*sort_by* Disallow: /*/collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /collections/*filter*&*filter* Disallow: /*/collections/*filter*&*filter* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*oseid=* Disallow: /*?*preview_theme_id=* Disallow: /*?*preview_script_id=* Disallow: /*?*ls=*&ls=* Disallow: /*?*ls%3*ls%3* Disallow: /search Disallow: /apple-app-site-association Disallow: /.well-known/shopify/monorail Disallow: /cdn/wpm/*.js Disallow: /services/login_with_shop Disallow: /products/*-remote Disallow: /*/products/*-remote Disallow: /collections/*/products/*-remote Disallow: /*/collections/*/products/*-remote Disallow: /collections/*/products/* Sitemap: https://chive.com/sitemap.xml # --------------------------------------------------------------------------- User-agent: AhrefsSiteAudit Crawl-delay: 10 Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /37556879497/checkouts Disallow: /37556879497/orders Disallow: /carts Disallow: /account Disallow: /customer_authentication/ Disallow: /*/customer_authentication/ Disallow: /a/downloads/-/* Disallow: /collections/*sort_by* Disallow: /*/collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /collections/*filter*&*filter* Disallow: /*/collections/*filter*&*filter* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*oseid=* Disallow: /*?*preview_theme_id=* Disallow: /*?*preview_script_id=* Disallow: /*?*ls=*&ls=* Disallow: /*?*ls%3*ls%3* Disallow: /search Disallow: /apple-app-site-association Disallow: /.well-known/shopify/monorail Disallow: /cdn/wpm/*.js Disallow: /services/login_with_shop Disallow: /products/*-remote Disallow: /*/products/*-remote Disallow: /collections/*/products/*-remote Disallow: /*/collections/*/products/*-remote Disallow: /collections/*/products/* Sitemap: https://chive.com/sitemap.xml # --------------------------------------------------------------------------- User-agent: MJ12bot Crawl-delay: 10 User-agent: Pinterest Crawl-delay: 1