How to control which AI crawlers access your content

AI companies routinely send automated bots to crawl the web. Some use what they find to train their language models. Others power AI-driven search results, or serve answers inside AI assistant products. AI Crawl Control lets you decide which of those bots can access your beehiiv website, so you stay in control of how your content is used.

This article explains how the AI Crawl Control dashboard works, the types of crawlers you can manage, and how to configure your settings.

beehiiv already gives you a global switch called Discoverable on the web in your SEO settings. 

To access the Discoverable on web toggle: From the left panel in your beehiiv account, go to Website > Builder > Settings > All general settings > Search engine optimization, then toggle on or off the setting. 

When you turn that off, it updates your site's robots.txt file to ask all search engines and AI crawlers not to index your content. That is an all-or-nothing signal, and it relies on convention: some bots read robots.txt and honor it, but there is no technical enforcement stopping a bot from ignoring it.

The Search engine indexing toggle can also be configured at the individual page level, not just site-wide. The two settings work additively: if either the site-level or page-level toggle has discoverability turned off, that page will be excluded from indexing. This means a page can independently opt out of discoverability even when the global toggle is on, but a page cannot opt back in to discoverability when the global toggle is off. For more, refer to SEO settings for your website.

AI Crawl Control works differently. It enforces your settings at the network level via Cloudflare, so requests from blocked crawlers are stopped before they ever reach your site, regardless of whether the bot respects robots.txt.

It is also more granular. Instead of blocking every bot, you see exactly which crawlers are hitting your site and choose which ones to allow or block, one at a time or by category. For example, you might block all AI model training bots while still allowing traditional search engines to index your posts for organic traffic.

1. From the panel on the left in your account, go to Website > AI Crawlers.
2. The AI crawler dashboard will open. You’ll be presented with a dashboard detailing AI crawler activity on your website. 
   
   
   

3. Under your analytics, you’ll see a list of crawlers, each with its name, operator, category, and a description of what the bot does. For each crawler, toggle access on or off based on your preferences.
   
   

   
   Tech Note: Toggling ON indicates that you want the crawler to be blocked. By default, the crawler is able to access your site.
4. You will be presented with a confirmation modal to confirm that you want to block the crawler. If you want to proceed, click on Block crawler. 
   
   
   
   
   If a crawler has a search visibility warning, you will be reminded that blocking the crawler will reduce your visibility in search results. Review it and click on the Block crawler button to confirm your choice.
   
   
   
5. Your settings will save automatically. 

Your preferences are applied to your custom web domain. Changes take effect as Cloudflare processes the update, typically within a short period after saving.

The dashboard is split into two views: a summary at the top showing overall activity and the pages bots are hitting most, and a crawlers list below where you configure your blocking settings.

A filter bar at the top of the page lets you narrow the data by specific crawler and time range. The default view shows all crawlers over the last 4 weeks.

Four cards at the top of the dashboard give you an at-a-glance read on crawler activity for the selected period.

Crawler requests is the total number of requests made to your site by AI crawlers and bots during the selected period. This count covers all bots in the catalog, whether blocked or not.

Blocked requests is the number of requests blocked based on your current settings. If you have not blocked any crawlers yet, this will be 0.

AI crawler share is the percentage of your total site traffic that comes from AI crawlers rather than human visitors. This gives you a sense of scale: a higher percentage means bots make up a larger share of your bandwidth.

Unique crawlers is the count of distinct crawler types detected on your site in the selected period. beehiiv tracks 22 known AI and AI-adjacent crawlers; you may see fewer if not all of them have visited your site.

A line chart shows served and blocked requests over time for the selected period. Two lines run in parallel: served requests (purple fill) and blocked requests (red line). Hover over any point on the chart to see the exact served and blocked counts for that day.

Use this chart to:

• See whether crawler activity is increasing, decreasing, or spiking around specific dates.
• Confirm that blocking rules are taking effect after you change your settings.
• Spot unusual traffic patterns that may not be visible in your standard website analytics.

A ranked list of the pages on your site that received the most crawler requests during the selected period, with a request count next to each page title.

Use this to:

• Identify which content AI companies are most interested in accessing.
• Spot whether high-value posts are being crawled at disproportionately high rates.

A breakdown of how your server responded to crawler requests during the selected period.

Common codes you may see:

• 200 OK: The request succeeded and the crawler received the page content.
• 204 No content: The server accepted the request but returned no content.
• 301 Moved permanently: The crawler followed a redirect to another URL.
• 404 Not found: The crawler requested a URL that no longer exists.
• 422 Unprocessable: The request was understood but could not be completed.
• 429 Too many requests: The crawler was rate-limited.

Below the summary, a table lists every AI crawler detected on your site, sorted by request volume. Each row shows the crawler name, its operator, its category, the total number of requests in the selected period, and a toggle to block it.

AI Crawl Control organizes crawlers into four categories. Knowing which category a crawler belongs to helps you make decisions that match your goals.

AI training crawlers: collect content to build or improve machine learning models. Blocking these prevents your writing from being used as training data.

AI search crawlers: power AI-driven search engines that synthesize results instead of linking directly to your site. Blocking these means your content may not appear in those AI-generated summaries.

AI assistant crawlers: retrieve content to answer questions inside AI assistant products in real time. Blocking these prevents those assistants from surfacing your content in responses.

Search engine crawlers: traditional indexing bots from established search engines. This category includes some crawlers with an AI component (such as Google's extended crawler). Blocking these can reduce your organic search visibility and SEO, and over time, remove your content from search results entirely. 

beehiiv tracks 22 crawlers across four categories. The table below lists every crawler in the catalog.

The catalog is maintained by beehiiv and updated as new crawlers become relevant.