Tehuty Search Index
TehutyBot Web Crawler
TehutyBot is the official web crawler operated by Global Empire Group Ltd (United Kingdom) to build a fast, multilingual public search index with dedicated coverage of the United Kingdom and all 54 African nations.
Official User-Agent
Every request from our crawler identifies itself clearly with this HTTP header:
TehutyBot/1.0 (+https://tehuty.com/crawler; bot@tehuty.com)What TehutyBot indexes
- Publicly accessible HTML pages and XML sitemaps across English, French, Arabic, Portuguese and Swahili.
- Page titles, meta descriptions, headings, canonical links and cleaned text snippets for Tehuty Search.
- TehutyBot never crawls private networks, authenticated dashboards, or pages marked with
noindex.
Politeness and crawl frequency
TehutyBot honours Crawl-delay directives, limits request concurrency per domain, and sends conditional HTTP headers (If-None-Match and If-Modified-Since) so unchanged pages are not re-downloaded.
Controls for site owners (robots.txt)
TehutyBot checks User-agent: TehutyBot rules first, then falls back to User-agent: *. To block TehutyBot from a directory or your entire site, add:
User-agent: TehutyBot
Disallow: /private/
Crawl-delay: 5Contact and removal requests
Site owners can report crawl issues, suggest public UK or African domains, or request immediate URL removal by emailing bot@tehuty.com.