Last updated: 2026-11-08
gotoai-general-bot is the web crawler operated by GotoAI Inc. (GotoAI). It collects publicly available information, such as government publications and court decisions. It follows each site’s robots.txt and terms of use.
GotoAI uses the collected data for research and to build search and retrieval datasets for its products and services.
How to recognise it
The crawler sends this User-Agent:
gotoai-general-bot/0.1 (+https://gotoai.com/en/references-en/gotoai-general-bot-en/)
- The
robots.txttoken isgotoai-general-bot. - The version number after the
/changes as the crawler is updated. The token does not. - Anyone can copy a User-Agent string. If traffic using this name behaves differently from what this page describes, please contact us.
Crawling policy
- Respects
robots.txt. The crawler readsrobots.txtbefore crawling a site and checks every URL against it. It also honoursCrawl-delay. - Public pages only. It does not log in, pass paywalls, solve CAPTCHAs or submit forms.
- Slow and sequential. It makes one request at a time. For most sites, GotoAI General Bot should not access the site more than once every few seconds on average. A longer
Crawl-delaytakes precedence. - Backs off on errors. On HTTP 429 or 5xx it waits, honouring
Retry-After. After repeated failures it stops the run. - Stops when refused. On HTTP 403 it stops immediately. It never retries from another address.
- Avoids unnecessary load. Unchanged pages are not downloaded again (conditional requests). The crawler avoids sites’ known peak periods, for example 確定申告 (tax-filing) season on nta.go.jp.
- Follows terms of use. It respects each site’s licence and attribution requirements, such as the attribution required by 公共データ利用規約 (the Public Data Terms of Use).
How to block or limit it
Add rules for gotoai-general-bot to your robots.txt.
Block the whole site:
User-agent: gotoai-general-bot
Disallow: /
Block part of the site:
User-agent: gotoai-general-bot
Disallow: /private/
Slow it down (seconds between requests):
User-agent: gotoai-general-bot
Crawl-delay: 10
The crawler picks up changes within one hour, because it re-reads robots.txt at the start of every run and at least hourly during a run.
If you would rather contact us directly, or want data that was already collected to be removed, see below.
Contact
- https://gotoai.com/en/contact-us-en/ — please mention
gotoai-general-botin your message.
Changes
- 2026-11-08 — First version.