Tehuty Search Index

TehutyBot Web Crawler

TehutyBot is the official web crawler operated by Global Empire Group Ltd (United Kingdom) to build a fast, multilingual public search index with dedicated coverage of the United Kingdom and all 54 African nations.

Official User-Agent

Every request from our crawler identifies itself clearly with this HTTP header:

TehutyBot/1.0 (+https://tehuty.com/crawler; bot@tehuty.com)

What TehutyBot indexes

  • Publicly accessible HTML pages and XML sitemaps across English, French, Arabic, Portuguese and Swahili.
  • Page titles, meta descriptions, headings, canonical links and cleaned text snippets for Tehuty Search.
  • TehutyBot never crawls private networks, authenticated dashboards, or pages marked with noindex.

Politeness and crawl frequency

TehutyBot honours Crawl-delay directives, limits request concurrency per domain, and sends conditional HTTP headers (If-None-Match and If-Modified-Since) so unchanged pages are not re-downloaded.

Controls for site owners (robots.txt)

TehutyBot checks User-agent: TehutyBot rules first, then falls back to User-agent: *. To block TehutyBot from a directory or your entire site, add:

User-agent: TehutyBot
Disallow: /private/
Crawl-delay: 5

Contact and removal requests

Site owners can report crawl issues, suggest public UK or African domains, or request immediate URL removal by emailing bot@tehuty.com.