# Grahchaal - Astrology Platform # robots.txt # # A crawler obeys ONLY the group that names it and falls back to "*" only when # no group does; groups are never merged (RFC 9309 section 2.2.1). So every # group below has to carry the full rule list. There used to be Googlebot, # Bingbot and Yandex groups that said just "Allow: /", which opened /admin/, # /customer/, /api/ and /login to exactly the crawlers that matter. # Google, Bing and Yandex need no group of their own: "*" is what they get. # Everyone, and the crawlers we only ask to slow down (they honour Crawl-delay). User-agent: * Disallow: /admin/ Disallow: /customer/ Disallow: /astrologer/ Disallow: /api/ Disallow: /dev/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password/ Disallow: /email/verify Disallow: /2fa/ # An infinite surface of thin result pages. /search stays crawlable: it is # noindex,follow, and a crawler has to fetch it to read that. The blog # search also answers under each language (/hi/blog/search …). Disallow: /blog/search Disallow: /*/blog/search Disallow: /*/print$ Crawl-delay: 1 User-agent: SemrushBot User-agent: DotBot Disallow: /admin/ Disallow: /customer/ Disallow: /astrologer/ Disallow: /api/ Disallow: /dev/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password/ Disallow: /email/verify Disallow: /2fa/ Disallow: /blog/search Disallow: /*/blog/search Disallow: /*/print$ Crawl-delay: 5 # Block bad bots User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / # Sitemap location — host must match the canonical the pages declare # (non-www); www resolves too, and pointing crawlers at it splits the site # into two properties in Search Console. Sitemap: https://grahchaal.com/sitemap.xml