My Crawl Is Stuck or Failed# Link to this article

First, distinguish slow from stuck. If the fetched-page count is still rising, the crawl is working — see Why a Site Audit May Take Longer Than Expected.

1. The crawler is blocked# Link to this section

Open Crawl Logs. Repeated 403 or 429 responses mean something is refusing the crawler — usually a WAF, a bot-protection service, or a rate limiter. Ask whoever manages your infrastructure to allow Rankato's crawler user agent. This is the most common cause by a wide margin.

2. robots.txt blocks the crawl# Link to this section

A Disallow: / rule affecting our user agent stops the crawl at the first request. The audit will report zero or very few pages. Rankato obeys robots.txt and will not override it.

3. Server errors# Link to this section

A run of 5xx responses means the site was failing during the crawl. Rankato backs off and retries, but a sustained outage ends the run. Check whether a deploy or an incident overlapped the crawl window, and re-run afterwards.

4. Allowance exhausted# Link to this section

If the crawl allowance ran out mid-run, the audit stops at the limit and reports what it collected. The usage panel shows this. Results already gathered remain valid.

5. JavaScript rendering timeouts# Link to this section

If rendering is enabled and pages are heavy, individual pages can time out. The crawl continues past them and they are reported as failures. Try a run with rendering off to confirm whether rendering is the cause.

6. Start URL unreachable# Link to this section

If the project's start URL redirects off-domain, requires authentication, or does not resolve, nothing can be crawled. Open the URL in a private browser window exactly as configured.

Still stuck?# Link to this section

Contact support with the project name and the approximate time the crawl started. Including the crawl log excerpt gets to an answer fastest.