Let's talk
WritingAI Search · SEO · Next.js · llms.txt

Make Your Next.js Site Visible to ChatGPT, Claude & Perplexity

By Srikanth V5 min read

More and more people find developers by asking an assistant — "find me a frontend developer in Bangalore who knows Next.js" — instead of typing keywords into a search box. So when I rebuilt my portfolio I wanted it to be as easy for ChatGPT, Claude, Perplexity and Gemini to read as it is for Google.

This is what I shipped, with real code, and an honest note on what's actually proven.

1. Let the crawlers in (properly)

None of this matters if robots.txt keeps the bots out. My first version looked fine:

User-Agent: GPTBot
Allow: /
Disallow: /admin
Disallow: /api

An AI-visibility checker still rated every bot as "Partial". Technically that's correct — they were restricted from some paths. Nothing useful was behind those paths: /admin needs a login and sends noindex, and /api is two auth endpoints. But "partial" is a signal I didn't need to send.

The fix is to give AI crawlers their own group that allows everything, and keep the restrictions in the * group for everyone else. The rule that makes this work: a crawler obeys only the most specific group that names it. GPTBot reads its own group and ignores * entirely.

// app/robots.js
const AI_AGENTS = [
    'GPTBot', 'OAI-SearchBot', 'ChatGPT-User',
    'ClaudeBot', 'Claude-User', 'Claude-SearchBot', 'Claude-Web', 'anthropic-ai',
    'PerplexityBot', 'Perplexity-User',
    'Google-Extended', 'Applebot-Extended', 'DuckAssistBot',
    'Meta-ExternalAgent', 'Amazonbot', 'CCBot', 'Bytespider',
    'cohere-ai', 'MistralAI-User', 'YouBot',
]

export default function robots() {
    return {
        rules: [
            { userAgent: AI_AGENTS, allow: '/' },
            { userAgent: '*', allow: '/', disallow: ['/admin', '/api', '/md/'] },
        ],
        sitemap: 'https://iamsrikanth.com/sitemap.xml',
    }
}

Worth knowing what these names mean. There are search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) that decide whether you can be cited in answers. There are user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) that load a page when someone pastes your link. And there are training crawlers and opt-out tokens (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended). For a portfolio whose whole job is to be found, I allow all three kinds.

Blocking Google-Extended doesn't affect normal Google Search ranking. It only controls whether Google uses your content for Gemini. The same goes for Applebot-Extended and Apple's models.

2. An llms.txt (and llms-full.txt)

llms.txt is a proposal by Jeremy Howard from 2024. It's a markdown file at the root of your site with a one-paragraph summary and a curated list of links, written for language models instead of browsers. Mine is a Next.js route handler that builds the file from the same Supabase data as the pages:

# Srikanth V — Frontend Developer in Bangalore

> Srikanth V is a frontend engineer (frontend developer) based in Bangalore, India…

## Projects
- [Hyqoo](https://iamsrikanth.com/projects/hyqoo.md): Migrated Hyqoo's site off WordPress…

## Optional
- [Full context](https://iamsrikanth.com/llms-full.txt): Profile, every case study and every article in one file

llms-full.txt is the long version: the profile, every case study and every article in full, in one file. An agent that wants everything makes one request instead of crawling twenty pages.

Both are force-static, so they cost nothing at request time. The admin calls revalidatePath('/', 'layout') after every edit, which refreshes them along with everything else.

3. A markdown version of every page

HTML is a noisy format for a model: navigation, animation markup, inline SVG. So every page also has a plain markdown version:

  • /index.md — home
  • /projects.md and /projects/<slug>.md
  • /blogs.md and /blogs/<slug>.md

There's one route handler, app/md/[[...path]]/route.js, and a few rewrites that map the .md URLs onto it. A rewrite on the Accept header means an agent that asks for markdown gets it on the normal URL:

// next.config.mjs
async rewrites() {
    const wantsMarkdown = [{ type: 'header', key: 'accept', value: '.*text/markdown.*' }]
    return {
        beforeFiles: [
            { source: '/', has: wantsMarkdown, destination: '/md' },
            { source: '/:section(projects|blogs)/:slug?', has: wantsMarkdown, destination: '/md/:section/:slug?' },
            { source: '/index.md', destination: '/md' },
            { source: '/:section(projects|blogs).md', destination: '/md/:section' },
            { source: '/:section(projects|blogs)/:slug.md', destination: '/md/:section/:slug' },
        ],
    }
}

curl -H 'Accept: text/markdown' https://iamsrikanth.com/projects/hyqoo

Two details keep this from hurting normal SEO:

  • Every markdown response sends Link: <https://iamsrikanth.com/projects/hyqoo>; rel="canonical". Google respects canonical links in HTTP headers, so the HTML page stays the one that's indexed.
  • Every HTML page advertises its markdown version with <link rel="alternate" type="text/markdown" href="….md">. In Next.js that's one line in the metadata: alternates.types['text/markdown'].

4. Structured data that answers the obvious questions

Assistants and search engines both lean on JSON-LD to work out who a page is about. The home page describes a Person with the details someone hiring would ask about: name and alternate names, job title, location (with "Bengaluru" as an alternate name for Bangalore), an Occupation with skills, current employer, and sameAs links to GitHub and LinkedIn. There's also a ProfilePage. Case studies are CreativeWork, articles are BlogPosting, and every page has BreadcrumbList.

What's proven and what isn't

I want to be straight about this, because a lot of "GEO" content online isn't:

  • robots.txt is real. If a crawler is blocked, it doesn't read your site. That part is binary.
  • Structured data and clean, server-rendered HTML are real. They help classic search, and AI search is built on top of classic search indexes.
  • llms.txt is a proposal, not a standard. I haven't seen a major AI provider publicly commit to reading it for search. Claims like "30% more AI citations" don't come with studies behind them. It's cheap to add, so I have one. But it's a bet, not a guarantee.
  • Content still wins. No file makes a thin site citable. What gets an engineer recommended is specific, first-hand writing, like the Core Web Vitals case study or the WordPress migration on this site.

The good news is that all of this is a few small files on top of a site you're building anyway. Get the crawlers in, give them a clean version of your content, describe yourself in structured data, and spend the rest of your time writing things worth citing.

More writing