13. Letting AuditAble Scan Your Website
13.1 Why your scan was blocked
Many websites use a security service, such as a firewall or bot protection from Cloudflare, Akamai, or your hosting company. These services try to stop automated programs from reading the site. They usually let people browsing the site through, but they can block AuditAble's scanner, because the scanner is an automated program too.
When that happens, the website answers the scanner with "access denied" (HTTP 403), "too many requests" (HTTP 429), or a "verify you are human" check. AuditAble cannot check pages it is not allowed to read, so the scan stops and tells you the site blocked it. Nothing is wrong with your account, and running the scan again will not help until the setting changes.
Your IT department, or the company that hosts your website, can fix this with a small, narrow exception. You do not need to turn off any protection.
13.2 What to send your IT department
Copy the message below and send it to your IT department or website host. Fill in your website address on the first line.
Request: please allow AuditAble's accessibility scanner to read our public website.
Website: [our website address]
Why: We use AuditAble to check our website for accessibility problems that
affect residents with disabilities. Our website's security settings are
blocking the scanner, so no pages can be checked.
Scanner name (User-Agent):
AuditAble Accessibility Scanner (+https://auditable.co/scanner)
What the scanner does:
- Loads public pages the same way a web browser does and reads them.
- Does not sign in, and does not change anything on the website.
- Requests at most two pages at a time from our site, with a pause of
1 to 3 seconds between requests.
- Follows robots.txt, including Crawl-delay.
What we are asking for:
1. In our firewall, bot protection, or content delivery network (CDN),
find the rule that blocked the scanner. The security log will show
requests from the scanner name above returning 403, 429, or a
challenge page.
2. Add a narrow exception that lets requests matching AuditAble's
scanner through to our public pages. Match both the scanner name and
AuditAble's verified source addresses, because a scanner name alone
can be copied by other programs. Email [email protected] for the
current verified source addresses.
3. Keep every other protection turned on.
4. If robots.txt blocks the public pages we want checked, allow the
scanner to read them.
Cloudflare note: ordinary WAF Skip rules do not bypass Bot Fight Mode.
Check which Cloudflare feature issued the challenge before adding the
exception:
https://developers.cloudflare.com/bots/get-started/bot-fight-mode/
13.3 After IT makes the change
Run the scan again from the same place you started it. If the scan is blocked again, your IT department can check the security log for the new scan time to find which rule still applies. If your team cannot change the website's security settings, email [email protected] with the website address and the time of the scan.