Not sure where to start? Take a guided tour.
AI Site Grader
AI Query Scanner
llms.txt GeneratorZyteOther crawler
Scrapy
Scrapy is a Scrapy framework user-agent commonly used for web scraping, including AI and machine learning data extraction. This record has no current Zyte source, so treat a matching request as unverified.
Not tied to answers in an AI assistant.
- ScrapyOtherNot checked
Reads your live robots.txt and applies the same rules these crawlers use. Nothing is stored.
- robots.txt token
Scrapy- Follows robots.txt
- Not documented
- How to verify it
- Not published
- Seen on
- 33% of sites · #16
- Last checked
- 11 June 2026
Should you block Scrapy?
You can, but it may not listen. Zyte has not documented Scrapy, so treat any rule as a request. Blocking Scrapy catches many generic scraper deployments, but may also block benign research or customer-configured crawls.
About robots.txtZyte has not documented whether Scrapy follows robots.txt, so a rule is a request, not a guarantee.
User-agent: Scrapy
Disallow: /Asks Scrapy not to crawl any page. Add it to the robots.txt file at the root of your site, above any wider rules.
Allowing Scrapy only creates the possibility of retrieval. To find out whether your pages are selected, measure which answer engines actually cite your pages using a stable query set.
Spot it in your logs.
Match the name, then check where the request came from. Anyone can copy a user-agent.
ScrapyThis string comes from a public crawler registry or observed naming, not current operator documentation.
- 1
Match the name
Search your server or CDN log for
Scrapy, case-insensitive. A match is a clue, not proof. - 2
Check the source
Zyte publishes no IP ranges, reverse DNS or signatures for Scrapy in the sources here, so it cannot be verified. Treat matches as unconfirmed.
Serve the content in HTML
The cited operator material does not verify JavaScript rendering. Serve useful HTML before client-side JavaScript where possible.
How active is Scrapy?
Measured across the sites connected to Trakkr, 27 July 2026 to 25 October 2026.
of connected sites saw Scrapy in the four weeks to 25 October 2026. That is 34 of 104.
Share of connected sites it visited each week
- Rank by reach
- 16 of 26
- Typical site, per week
- 12 requests
- Sites reached, 13 weeks
- 42 of 135
Most widespread AI crawlers
Share of connected sites each one visited, last four weeks.
- 1
OAI-SearchBot84%
- 2
ClaudeBot84%
- 3
- 4
Amazonbot61%
- 5
- 6
PerplexityBot56%
- 7
Applebot51%
- 16
Scrapy33%
How this is measured
Finalized daily crawler summaries from sites connected to Trakkr, grouped by the crawler named in each request. Trakkr’s own site is excluded. A site counts as reached in a week when it logged at least one request from Scrapy. The typical figure is the median weekly requests among the sites it reached.
- This is a sample of connected sites, not the whole web.
- A matching user-agent does not prove the named operator sent the request.
- Requests are not unique pages, training use or citations.
- Crawlers seen on fewer than 10 sites are not shown. Classic search crawlers such as Googlebot are not collected on every site, so they are left out.
Questions, answered.
Scrapy is a Scrapy framework user-agent commonly used for web scraping, including AI and machine learning data extraction. This record has no current Zyte source, so treat a matching request as unverified.
Sources.
Reported, not operator verified. Operator facts and Trakkr’s measurements keep separate dates.
- [1]
Last source check 11 June 2026. Activity measured to 25 October 2026.
Crawlers like Scrapy.
All AI crawlersSee which AI crawlers read your site.
Trakkr shows which bots, Scrapy included, visit which pages each day, and whether the answers they feed mention you.
14-day free trial. Cancel anytime.