Skip to content

See where you stand

Change it

Works withWordPressShopifyWebflowGitHubGoogleSlackAll integrations
FixesFixes now ship to WordPress, Shopify, Webflow and GitHub

By team

By goal

Done for you

Not sure where to start? Take a guided tour.Book a demo
Customer storyHow the voice AI company Futuro grew its traffic3×AI traffic in 60 days

Learn

Data and research

FreeAI Site GraderAI Query Scannerllms.txt GeneratorAll free tools
Open dataWhere AI sends people, across more than 6,000 sites

ApifyOther crawler

ApifyBot

ApifyBot is a token associated with crawlers run on the Apify scraping platform by its customers. This record has no current Apify source, so treat a matching request as unverified.

Not tied to answers in an AI assistant.

Get the robots.txt ruleRecord source
Is ApifyBot allowed on your site?
  • ApifyBotOther
    Not checked

Reads your live robots.txt and applies the same rules these crawlers use. Nothing is stored.

robots.txt token
ApifyBot
Follows robots.txt
Not documented
Seen on
Not measured
Last checked
11 August 2026

Should you block ApifyBot?

You can, but it may not listen. Apify has not documented ApifyBot, so treat any rule as a request. Blocking it affects only crawlers that send this token. Apify states its main content crawler sends no identifying user-agent.

About robots.txtApify has not documented whether ApifyBot follows robots.txt, so a rule is a request, not a guarantee.

robots.txt
User-agent: ApifyBot
Disallow: /

Asks ApifyBot not to crawl any page. Add it to the robots.txt file at the root of your site, above any wider rules.

Ignored robots.txt?

Spot it in your logs.

Match the name, then check where the request came from. Anyone can copy a user-agent.

Reported user-agentsource
ApifyBot

This string comes from a public crawler registry or observed naming, not current operator documentation.

  1. 1

    Match the name

    Search your server or CDN log for ApifyBot, case-insensitive. A match is a clue, not proof.

  2. 2

    Check the signed agent key directory

    Compare the request’s source with the list Apify publishes. Anything outside it is not Apify. api.apify.com/.well-known/http-message-signatures-directory

Is this really ApifyBot?

Signed requests

Apify signs each ApifyBot request with HTTP Message Signatures (RFC 9421), so its address proves nothing. A genuine request carries Signature, Signature-Input and Signature-Agent headers, and the signature checks out against Apify’s published keys.

Many CDNs, Cloudflare included, check these signatures for you and mark the request as a verified bot. Without a signature, it isn’t ApifyBot.

api.apify.com/.well-known/http-message-signatures-directory

Serve the content in HTML

The cited operator material does not verify JavaScript rendering. Serve useful HTML before client-side JavaScript where possible.

Robots.txt behaviour is set per crawler by the Apify customer running it, so it varies.

Questions, answered.

ApifyBot is a token associated with crawlers run on the Apify scraping platform by its customers. This record has no current Apify source, so treat a matching request as unverified.

Sources.

Reported, not operator verified. Operator facts and Trakkr’s measurements keep separate dates.

  1. [1]
    Reported crawler reference

    Reported, not operator verified · checked 11 August 2026

    apify.com/apify/website-content-crawler/input-schema

Last source check 11 August 2026.

All AI crawlers

See which AI crawlers read your site.

Trakkr shows which bots, ApifyBot included, visit which pages each day, and whether the answers they feed mention you.

14-day free trial. Cancel anytime.