Install crawler tracking

Crawler tracking must see the request before a browser loads. Pick the deepest layer you can connect: a CDN or hosting integration sees cache hits and requests that never reach your app, while an origin integration sees only requests that reach the origin.

The source picker in Traffic → Crawlers matches the current catalog. Choose your platform there to create the connection, then use the matching section here for the full setup.

Cloudflare {#cloudflare}

Cloudflare is a practical path when your website hostname is proxied through Cloudflare. It reads available server-side analytics, but cache behavior, sampling, the chosen dataset and WAF rule order affect what appears. Cloudflare documents that WAF rules can block traffic before AI Crawl Control sees it.

Before you start

You need a Cloudflare account with an active zone and permission to create API tokens. In DNS, the website record must show the orange cloud. A grey, DNS-only record does not pass traffic through Cloudflare, so this method cannot see crawler visits.

Create the token

1. Open Cloudflare API Tokens.
2. Choose Create Token, then Create Custom Token. The link in Trakkr opens a pre-filled template with the same permissions.
3. Add these three read permissions:

1. Under Zone Resources, allow all zones or only the domain you want to track.
2. Create the token and copy it. Cloudflare only shows it once.

Connect and verify

1. In Trakkr, open Traffic → Crawlers and choose Cloudflare.
2. Paste the token and choose Verify token.
3. Pick the zone, then choose Connect zone.
4. Use Send verification in the Crawlers header. Verified rows should appear in the Live feed.

Cloudflare analytics are aggregated. Busy zones can sample or truncate low-volume bot traffic. The optional Worker can add per-request delivery for matching requests that reach it, but it cannot recover a request blocked earlier by a WAF or upstream system.

Cloudflare troubleshooting

• Token verification fails: Check for all three permissions, remove trailing spaces, and confirm the token has not expired.
• No zones appear: Add Zone → Zone → Read and include the target zone in the token resources.
• Analytics is denied: Add Zone → Analytics → Read. Account Analytics alone is not enough on every account.
• Verification works but real bots do not appear: Confirm the record is proxied, then check Bot Fight Mode and custom WAF rules. See Trakkr behind a WAF.

Vercel {#vercel}

Vercel sends production request logs to Trakkr through a Log Drain. This path requires Vercel Pro or Enterprise.

1. Open Traffic → Crawlers, choose Vercel, then Continue to Vercel.
2. Authorize the Vercel team that owns the website project.
3. Back in Trakkr, pick one project. Trakkr creates and scopes the Log Drain for you.
4. Send verification and look for the Verified rows in the Live feed.

The connection covers production traffic on the main domain and production aliases. Preview and branch deployments are excluded. One Vercel project can be connected per brand; choosing another replaces the existing source.

If no data arrives, confirm the project is on Pro or Enterprise and open the connection health message. Reconnect if the Log Drain is missing, disabled, points at the wrong endpoint, or is scoped to a different project. Disconnecting in Trakkr removes the drain; you can also remove it from Vercel's team integrations.

Netlify {#netlify}

Netlify uses OAuth to select the site, then a one-file Edge Function forwards matching AI crawler events as requests reach that function. Trakkr lists your sites but does not read site content, deploy code, or change DNS.

Connect the site

1. Open Traffic → Crawlers, choose Netlify, then Continue to Netlify.
2. Approve access for the right Netlify team and pick the site after Netlify sends you back.
3. Trakkr creates a pending connection and shows the Edge Function file plus its credentials.

Add the Edge Function

Create this file in the connected site's repository:

netlify/edge-functions/trakkr-crawler.ts

Copy the current template from the Trakkr connection card into that file. The template lets every request continue unchanged and sends a small background request only when it matches a known AI crawler.

Add the three values shown by Trakkr in Site configuration → Environment variables:

TRAKKR_CONNECTION_ID=<your connection id>
TRAKKR_WEBHOOK_SECRET=<your webhook secret>
TRAKKR_INGEST_URL=https://api.trakkr.ai/crawler-connect/ingest/netlify

Commit and push, or trigger Deploys → Trigger deploy → Clear cache and deploy site. A new deploy is required before the function can receive traffic.

Netlify troubleshooting

• OAuth does not finish: Allow trakkr.ai and app.netlify.com, check the selected team, and retry.
• The function is live but the feed is empty: Check Netlify deploy logs for missing TRAKKR_ values, then redeploy after fixing them.
• Real visits are quiet: Synthetic verification only checks Trakkr's test-row path. Check the deployed function, environment values and Netlify logs, then wait for the source's expected delivery delay and a real matching request.
• The function fails: Open the site's Edge Functions logs and use the first error to fix the file or environment values.

To remove the connection, disconnect it in Trakkr, delete the Edge Function, deploy again, and optionally revoke Trakkr under Netlify's authorized apps.

WordPress {#wordpress}

WordPress tracking can use a crawler-only account. It does not need access to Trakkr's separate publishing features in Sites.

1. In Traffic → Crawlers, choose WordPress and download the Trakkr plugin. The permanent download is https://api.trakkr.ai/crawler-connect/wordpress/plugin.
2. On standard WordPress, open Plugins → Add New → Upload Plugin, upload the ZIP, and activate it. On WordPress VIP, add the unpacked plugin folder to the application's /plugins directory through its Git repository, deploy it to a non-production environment, then activate it.
3. In WordPress, create a dedicated user with the Trakkr Crawler Sync role.
4. Create an Application Password for that user.
5. Return to Trakkr, enter the HTTPS site URL, username, and Application Password, then choose Enable tracking.
6. Send verification.

The crawler-only role can retrieve crawler rows and mark received rows as synced. It cannot publish content, edit posts, upload media, change site options, or write files. If the site is already connected in Sites for publishing, you can reuse that connection instead.

If the site uses Wordfence, Solid Security, Sucuri, or another WAF, allow authenticated access to /wp-json/ and /wp-json/trakkr/*. If verification works but real visits stay empty, check the security tool for blocked AI user agents. The WAF guide has exact steps.

Next.js self-hosted {#nextjs}

Choose this path for a Next.js app you host yourself. Trakkr creates a unique endpoint, bearer token, and a Proxy or middleware template.

1. Choose Next.js in the Crawlers source picker and name the connection.
2. Copy the endpoint, bearer token, and current template.
3. Send the sample event in Trakkr. This checks authentication before you deploy.
4. Add the template to your Proxy or middleware path, deploy it, then send verification.

The template forwards matching crawler hits in the background and does not wait before returning the page. Keep the bearer token in a server-only secret. If your app sits behind another CDN, prefer connecting that CDN so cache hits are not missed.

Node and Express {#node}

The Node path provides Express middleware that reports a crawler request after the response finishes.

1. Choose Node / Express and create the connection.
2. Copy the endpoint, bearer token, and middleware from Trakkr.
3. Send the built-in sample event.
4. Add the middleware before your routes, set the token as a server secret, and deploy.

If you use a reverse proxy, configure Express trust proxy correctly so the reported IP comes from the real client header. The IP is used for analytics only, never as proof that a bot is genuine.

Nginx and OpenResty {#nginx}

This path uses an asynchronous OpenResty log hook.

1. Choose Nginx / OpenResty, then copy the generated endpoint, bearer token, and configuration.
2. Send the sample event in Trakkr.
3. Add the hook to the request log phase and reload Nginx after validating the configuration.
4. Send verification from Crawlers.

Set real_ip_header and trusted proxy ranges before using the client IP when Cloudflare, Sucuri, or another proxy sits in front. A plain Nginx origin does not see requests served or blocked at the CDN.

AWS CloudFront {#cloudfront}

CloudFront uses a Lambda@Edge function on Viewer Request, so it can see requests before the cache decision.

1. Choose AWS CloudFront and create the connection.
2. Copy the generated Lambda@Edge template, including its endpoint and bearer token.
3. Send the sample event in Trakkr.
4. Create the function in us-east-1, publish a numbered version, and attach it to the distribution's Viewer Request event.
5. Wait for the association to deploy, then send verification.

Lambda@Edge does not support environment variables, so the generated values are kept in the function source. Restrict access to the function and rotate the connection if the token is exposed.

Akamai {#akamai}

Choose Akamai for DataStream 2 or another Akamai HTTPS forwarder. Trakkr generates a unique endpoint and Basic Authentication credentials.

Create the connection, copy the Akamai template, and send the sample event. Then configure DataStream 2 to send matching request batches to the endpoint with the generated username and password. Use the connection health view to confirm events arrive.

Fastly {#fastly}

Choose Fastly to send log streaming data over HTTPS. Trakkr generates an endpoint, bearer token, and a ready-to-adapt template.

Test the sample event first, add the HTTPS log endpoint in Fastly, pass the bearer token in the Authorization header, then activate the service version. Filter at the edge when possible so only known crawler rows are forwarded.

Other or custom edge {#custom}

Use the generic webhook for another CDN, reverse proxy, or log forwarder. The setup screen gives you a POST endpoint, bearer token, sample payload, and curl example.

Events use a small JSON list with the time, URL, user agent, IP, status code, and country when available. Send the built-in sample first, then adapt your platform's HTTPS forwarder. Keep the token secret and send only server-side request data.

Webflow {#webflow}

Webflow does not expose edge request logs, so the product path places Cloudflare in front of it.

1. In Webflow, open Project Settings → Publishing and copy the custom-domain DNS records.
2. Add the domain to Cloudflare and recreate those records.
3. Change the nameservers at your registrar to the two Cloudflare gives you.
4. Keep the website record proxied, then connect the Cloudflare zone in Trakkr.

Webflow still serves the site. Cloudflare only sits in front and supplies the server-side traffic signal.

Shopify {#shopify}

Shopify does not expose raw request logs on standard plans. Use Cloudflare in front of the store.

1. In Shopify Admin → Settings → Domains, confirm the primary domain is connected by DNS.
2. Add the domain to Cloudflare and recreate the Shopify A and CNAME records.
3. Change the registrar nameservers to Cloudflare and proxy the website record.
4. Choose Shopify in Trakkr and finish the Cloudflare token and zone steps.

If the domain was bought through Shopify, use the current records Shopify provides when building the Cloudflare zone.

HubSpot {#hubspot}

HubSpot CMS does not expose raw edge logs. Keep HubSpot as the host and proxy the domain through Cloudflare.

1. In HubSpot, open Settings → Website → Domains & URLs and note the connected-domain records.
2. Recreate the HubSpot CNAME records in Cloudflare.
3. Point the registrar nameservers at Cloudflare and proxy the site record.
4. Choose HubSpot in Trakkr and connect the Cloudflare zone.

Squarespace {#squarespace}

Squarespace needs an externally managed domain before Cloudflare can sit in front.

1. In Squarespace → Settings → Domains, use Use a domain I own.
2. Add the domain to Cloudflare and recreate the A and CNAME records Squarespace gives you.
3. Change the registrar nameservers to Cloudflare.
4. Choose Squarespace in Trakkr and connect the zone.

If Squarespace manages the domain and does not allow external nameservers, transfer the domain to a registrar that does before changing DNS.

Wix {#wix}

Wix does not expose raw request logs, and a browser script misses most AI crawlers.

1. In Wix, open Settings → Domains and note the A and CNAME records.
2. Add the domain to Cloudflare with the same records and proxy the website hostname.
3. Change nameservers if Cloudflare is not already authoritative.
4. Choose Wix in Trakkr and connect the Cloudflare zone.

Framer {#framer}

Framer does not expose request logs or third-party edge code.

1. In Framer → Site Settings → Domains, copy the CNAME that points to framer.website.
2. Recreate that record in Cloudflare and set it to Proxied.
3. Point the registrar nameservers at Cloudflare.
4. Choose Framer in Trakkr and connect the zone.

Ghost {#ghost}

Ghost(Pro) does not provide raw access logs or third-party edge code.

1. In Ghost Admin, open Settings → Custom Domain and confirm the domain record.
2. Add the domain to Cloudflare and recreate the A or CNAME record that points to Ghost.
3. Change the registrar nameservers to Cloudflare and proxy the website record.
4. Choose Ghost in Trakkr and connect the zone.

AI Pages {#ai-pages}

If AI Pages is enabled for the brand and already has traffic data, the source picker can connect it automatically. Choose AI Pages and Trakkr creates the crawler source without another token, DNS change, or code file.

This option only appears when AI Pages has enough data and is not already connected. If it is missing, use the platform that actually receives your site's requests.

Verify any connection {#verify}

After setup, use Send verification in the Crawlers header. It creates labelled synthetic events and confirms that the dashboard can store and query those test rows. It does not prove that the provider source is configured, that a real request reached the source, or that the live ingest path delivered it.

Verification does not prove that real bots can reach your site. If test rows appear but real traffic remains empty after the source's expected delivery delay, check these in order:

1. The connection health message and platform logs.
2. Whether the connected source sees the production hostname.
3. robots.txt rules for the missing bot.
4. WAF, bot-management, rate-limit, and security-plugin logs.
5. Whether two sources watch the same origin and should be reduced to one.

Start monitoring

Review the AI crawler monitoring overview if you still need to choose a source or explain the measurement model. When you are ready, create a Trakkr account or book a demo to plan the install with the team.