Skip to content

See where you stand

Change it

Works withWordPressShopifyWebflowGitHubGoogleSlackAll integrations
FixesFixes now ship to WordPress, Shopify, Webflow and GitHub

By team

By goal

Done for you

Not sure where to start? Take a guided tour.Book a demo
Customer storyHow the voice AI company Futuro grew its traffic3×AI traffic in 60 days

Learn

Data and research

FreeAI Site GraderAI Query Scannerllms.txt GeneratorAll free tools
Open dataWhere AI sends people, across more than 6,000 sites

LyrenthAI search crawler

AIWebIndex

AIWebIndex is a crawler that builds Lyrenth, an index of public web pages served to AI systems and developers as extracted, attributed documents.

Needed to be found in its search index.

Get the robots.txt ruleLyrenth documentation
Is AIWebIndex allowed on your site?
  • AIWebIndexSearch
    Not checked

Reads your live robots.txt and applies the same rules these crawlers use. Nothing is stored.

robots.txt token
AIWebIndex
Follows robots.txt
Yes, documented
How to verify it
Published IP list
Seen on
Not measured
Last checked
17 September 2026

Should you block AIWebIndex?

Not if you want to be found. Blocking it keeps your pages out of the Lyrenth index. Lyrenth says the block covers both scheduled crawls and on-demand fetches.

robots.txt
User-agent: AIWebIndex
Disallow: /

Asks AIWebIndex not to crawl any page. Add it to the robots.txt file at the root of your site, above any wider rules.

Ignored robots.txt?

Spot it in your logs.

Match the name, then check where the request came from. Anyone can copy a user-agent.

Scheduled crawlsource
AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)

Version numbers change. Match the highlighted name, not the whole string.

On-demand index fetchsource
AIWebIndex-Agent/2.0 (+https://lyrenth.com/bot; on-demand index fetch)

Lyrenth says on-demand fetches follow the same robots.txt rules as scheduled crawls.

  1. 1

    Match the name

    Search your server or CDN log for AIWebIndex, case-insensitive. A match is a clue, not proof.

  2. 2

    Check the published IP list

    Lyrenth also signs requests with Web Bot Auth (RFC 9421) and says every crawler IP has forward-confirmed reverse DNS under lyrenth.com. lyrenth.com/bot/ip-ranges.json

Is this really AIWebIndex?

Addresses and reverse DNS

Paste an address that claims to be AIWebIndex. It is checked against Lyrenth’s published list and its reverse DNS. Nothing is stored.

Check any crawler’s address

Serve the content in HTML

The cited operator material does not verify JavaScript rendering. Serve useful HTML before client-side JavaScript where possible.

Lyrenth, operated by Aleksma Ai, Inc., says it does not train foundation models on crawled content. Verifying your domain with Lyrenth grants it the right to license that domain’s content to AI labs.

Questions, answered.

AIWebIndex is a crawler that builds Lyrenth, an index of public web pages served to AI systems and developers as extracted, attributed documents.

Sources.

Officially documented. Operator facts and Trakkr’s measurements keep separate dates.

  1. [1]
    Lyrenth crawler documentation

    Officially documented · checked 17 September 2026

    lyrenth.com/bot

First verified 17 September 2026. Last source check 17 September 2026.

All AI crawlers

See which AI crawlers read your site.

Trakkr shows which bots, AIWebIndex included, visit which pages each day, and whether the answers they feed mention you.

14-day free trial. Cancel anytime.