agents.json: The New robots.txt for AI

In 1994, a file called robots.txt changed the internet. It was simple: a plain text file at the root of your website that told search engines what to crawl and what to skip. Every website has one. Every crawler respects it.

Thirty years later, we need the same thing for AI agents. That's agents.json.

The Problem: AI Agents Are Guessing

When ChatGPT or Gemini needs to interact with a website, it guesses. It scrapes HTML. It tries different URL patterns. It parses whatever structured data it can find. There's no standard way for a website to say: "Here's what I can do, and here's how to do it."

This is where we were with search engines in 1993 — before robots.txt, before sitemaps, before any kind of machine-readable protocol. AI agents are crawling the web blind.

What agents.json Does

agents.json is a JSON file served at /.well-known/agents.json that tells AI agents:

• What your website is (a store, a service, a content site)
• What actions are available (search products, check prices, add to cart)
• What endpoints exist and how to call them
• What authentication is needed (if any)
• What data format to expect in responses

Instead of guessing, the agent reads one file and knows everything it can do.

A Real Example

Here's what GEOstack Shadow generates for a WooCommerce store:

{
  "schema_version": "1.0",
  "name_for_human": "Your Store Name",
  "name_for_model": "your_store",
  "description": "Online store selling handcrafted goods",
  "api": {
    "type": "openapi",
    "url": "/shadow/openapi.json"
  },
  "capabilities": {
    "search": {
      "endpoint": "/shadow/search/",
      "method": "GET",
      "params": ["q", "category", "min_price", "max_price"]
    },
    "product_details": {
      "endpoint": "/shadow/products/{slug}/",
      "method": "GET"
    },
    "variants": {
      "endpoint": "/shadow/products/{slug}/variants/",
      "method": "GET"
    },
    "cart": {
      "endpoint": "/shadow/cart/add/",
      "method": "POST"
    }
  },
  "auth": {
    "type": "none"
  },
  "contact": "support@yourstore.com"
}

An AI agent reads this and immediately knows: I can search this store's catalog, get product details, check variants, and add items to cart. No guessing, no HTML parsing, no trial and error.

The Family of Discovery Files

agents.json isn't alone. GEOstack Shadow generates four discovery files, each targeting different AI systems:

agents.json

The primary machine-readable discovery file. Structured JSON describing all capabilities. Served at /.well-known/agents.json.

ai.txt

A human-and-machine-readable file (like robots.txt but for AI). Plain text that describes what the store offers and how AI agents should interact with it. Served at /ai.txt.

llms.txt

Specifically designed for large language models. Contains a concise summary of the store in a format optimized for LLM context windows. Served at /llms.txt.

ai-plugin.json

Compatible with the OpenAI plugin specification. Allows ChatGPT and compatible systems to discover and interact with your store as a plugin. Served at /.well-known/ai-plugin.json.

Why This Matters for Your Store

Right now, AI agents discover stores through web search and HTML scraping. It's inefficient, error-prone, and favors stores with clean markup — regardless of product quality.

Discovery files flip this. When an AI agent visits your store, the first thing it checks is /.well-known/agents.json. If it finds one, it knows exactly what your store can do. If it doesn't, it falls back to guessing.

Stores with discovery files get found. Stores without them get skipped. It's that simple.

How to Add agents.json to Your Store

You have two options:

• Automatic: Install GEOstack Shadow (free WooCommerce plugin). It generates all four discovery files automatically based on your store's actual catalog and capabilities.
• Manual: Create a .well-known/agents.json file on your server. Follow the schema above. You'll need to build and maintain the endpoints yourself.

The plugin approach takes 2 minutes. The manual approach takes days and requires ongoing maintenance. For most stores, the plugin is the right choice.