About iWebScan, our scanner
iWebScan is the crawler behind the free AI Readiness Scan by iWeb & SEO. It runs only when someone asks for a scan of a specific website, and it is not a general web crawler.
What it requests
For each scan it makes a small number of requests (usually 15–25) to the scanned site:
- the homepage, once as a normal browser and once each as ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot, to test whether AI assistants are let through;
/robots.txt,/llms.txt, the XML sitemap, and a few well-known files such as/.well-known/mcpand/openapi.json;- the About, Contact and Privacy pages linked from the homepage, plus up to five links listed in llms.txt.
It only reads public pages. It never submits forms, logs in, or follows links beyond those above. Results are cached for an hour, so repeated scans of the same site don't trigger new requests.
User agent
Our own requests identify themselves as:
iWebScan/1.0 (+https://scan.iwebandseo.com/bot)
When testing assistant access we send the assistant's published user agent with iWebScan/1.0 (+https://scan.iwebandseo.com/bot) appended, so you can always tell these requests apart from the real assistants.
Opting out
To stop scans of your site, block the iWebScan user agent at your firewall, or contact us and we'll exclude your domain.