SmartSEOBot is the crawler behind the SMARTSEO app. It reads a website’s pages so the app can show that site’s owner what is broken and what to fix. It is not a search engine, it does not build an index of the web, and it does not crawl on its own: every visit starts with a SMARTSEO user.
Its User-Agent
Every request SmartSEOBot makes sends exactly this User-Agent:
Mozilla/5.0 (compatible; SmartSEOBot/1.0; +https://smartseo.dev/bot)The token to match on is SmartSEOBot.
When it visits
- A user adds their website to SMARTSEO, or presses Rescan. Rescans are limited by the user’s plan (a few a day at most).
- A user names a website as a competitor. SmartSEOBot then reads up to 50 of that site’s pages once, and again only if that read failed.
There is no schedule that crawls sites by itself.
What it fetches
/robots.txtand the sitemap it names (or the usual sitemap addresses).- The homepage, the pages in the sitemap and, when there is room, pages found by following links from the homepage, up to the user’s plan limit for one scan (100 or 1,000 pages) and a 15-minute time limit, whichever comes first.
- A handful of checks per scan: the
http://address, the other ofwww/ no-www, one made-up address to see how the site answers a missing page, the HTTPS certificate, and the site icon.
It asks for at most two pages at a time, and only when your robots.txt sets no Crawl-delay; otherwise it asks for one at a time. If your site answers 429 or 503, sends Retry-After, or stops answering twice in a row, it goes back to one at a time for the rest of that scan. It waits up to 15 seconds for each answer, or 8 seconds after two pages in a row didn’t answer, until one does. It runs no JavaScript and never submits forms. It keeps the cookies your site sets only for the length of one scan, as a browser would, and then forgets them. It reads HTML pages; links to other kinds of files are not followed.
Page-speed measurements in the app come from Google’s PageSpeed Insights, so those requests come from Google, not from SmartSEOBot.
robots.txt
SmartSEOBot fully obeys robots.txt. It reads yours before it asks for anything else, and then checks every address it requests on your site against it: the homepage, the sitemap, every page in the sitemap, every page found by following links, the handful of checks above, and the address at the end of every redirect. That is the same whether the site is a SMARTSEO user’s own or one a user named as a competitor.
It obeys the rules in a User-agent: SmartSEOBot group, or in the User-agent: * group if there is none, the way RFC 9309 describes (the longest matching rule wins; * and $ are understood). A page it is told not to read is skipped, never fetched. If robots.txt disallows your homepage, it reads nothing at all.
If your robots.txt answers with a server error, or doesn’t answer, it reads no page, as RFC 9309 asks. A missing robots.txt (a 404) means everything is allowed.
Crawl-delay is honoured too: if your group sets Crawl-delay: 5, SmartSEOBot asks for one page at a time and waits at least 5 seconds between requests to your site. A scan still ends after 15 minutes, so a long delay means fewer pages are read, never a longer visit.
To keep SmartSEOBot off your site completely:
User-agent: SmartSEOBot
Disallow: /Or to slow it down:
User-agent: SmartSEOBot
Crawl-delay: 10When SmartSEOBot can’t read a site
When SmartSEOBot is turned away (a refusal, a bot challenge, or “slow down” that lasts past its own back-off), finds only a JavaScript shell, or can read only a handful of the pages a site plainly has, SMARTSEO may ask a crawling service, DataForSEO, to read the same public pages once for that scan. It is never asked first, never when robots.txt says no, and never for a site that only timed out or moved.
Its crawler sends the same SmartSEOBot User-Agent, never a browser’s. It obeys your robots.txt together with the rules SmartSEOBot obeys, waits at least your Crawl-delay (and never less than 2 seconds between pages), and reads no more pages than SmartSEOBot would. It doesn’t try a different network to get past a block. If it is turned away too, it isn’t asked again for that site for 30 days, unless the site’s own owner starts a scan. Blocking SmartSEOBot in robots.txt blocks both.
How to allow it
Some firewalls block crawlers they don’t know. SmartSEOBot runs on cloud hosting and has no fixed list of IP addresses, so allow it by its User-Agent.
Cloudflare
Security → WAF → Custom rules → Create rule. Set the field to User Agent, the operator to contains and the value to SmartSEOBot, then choose the action Skip and tick the protections to skip (for example the managed rules and Bot Fight Mode, where your plan allows it). Put the rule first.
Wordfence
Wordfence allowlists by IP address, which SmartSEOBot doesn’t have, so check the two things that usually stop it: the firewall’s rate limiting for crawlers (allow more page views per minute, or set the action to throttle rather than block), and any blocking rule that matches its User-Agent. Wordfence’s Live Traffic view shows each request’s User-Agent, so you can see what happened to SmartSEOBot’s.
Sucuri
In the Sucuri firewall, make sure SmartSEOBot is not on your list of blocked User-Agents. If the firewall still challenges it, ask Sucuri support to allow requests whose User-Agent contains SmartSEOBot; an IP allowlist won’t work, because it has no fixed IP.
If your host’s firewall has no User-Agent rule, ask your host to allow requests whose User-Agent contains SmartSEOBot.
Contact
Questions, or want SmartSEOBot to stop visiting your site? Email info@smartseo.dev and include your domain.