--- title: "Make Your Next.js Site Visible to ChatGPT, Claude & Perplexity" description: "How I set up my portfolio for AI search: an llms.txt and llms-full.txt, a markdown version of every page (via .md URLs or Accept: text/markdown), robots.txt groups that fully allow AI crawlers, and JSON-LD. Plus an honest look at what's proven and what isn't." url: "https://iamsrikanth.com/blogs/llms-txt-nextjs-ai-search-visibility" author: "Srikanth V" published: "2026-10-20" updated: "2026-10-24" tags: "AI Search, SEO, Next.js, llms.txt" --- # Make Your Next.js Site Visible to ChatGPT, Claude & Perplexity By Srikanth V · 2026-10-20 · 5 min read > How I set up my portfolio for AI search: an llms.txt and llms-full.txt, a markdown version of every page (via .md URLs or Accept: text/markdown), robots.txt groups that fully allow AI crawlers, and JSON-LD. Plus an honest look at what's proven and what isn't. More and more people find developers by asking an assistant — "find me a frontend developer in Bangalore who knows Next.js" — instead of typing keywords into a search box. So when I rebuilt my portfolio I wanted it to be as easy for **ChatGPT, Claude, Perplexity and Gemini** to read as it is for Google. This is what I shipped, with real code, and an honest note on what's actually proven. ## 1. Let the crawlers in (properly) None of this matters if `robots.txt` keeps the bots out. My first version looked fine: ```txt User-Agent: GPTBot Allow: / Disallow: /admin Disallow: /api ``` An AI-visibility checker still rated every bot as **"Partial"**. Technically that's correct — they were restricted from some paths. Nothing useful was behind those paths: `/admin` needs a login and sends `noindex`, and `/api` is two auth endpoints. But "partial" is a signal I didn't need to send. The fix is to give AI crawlers their own group that allows everything, and keep the restrictions in the `*` group for everyone else. The rule that makes this work: **a crawler obeys only the most specific group that names it**. GPTBot reads its own group and ignores `*` entirely. ```js // app/robots.js const AI_AGENTS = [ 'GPTBot', 'OAI-SearchBot', 'ChatGPT-User', 'ClaudeBot', 'Claude-User', 'Claude-SearchBot', 'Claude-Web', 'anthropic-ai', 'PerplexityBot', 'Perplexity-User', 'Google-Extended', 'Applebot-Extended', 'DuckAssistBot', 'Meta-ExternalAgent', 'Amazonbot', 'CCBot', 'Bytespider', 'cohere-ai', 'MistralAI-User', 'YouBot', ] export default function robots() { return { rules: [ { userAgent: AI_AGENTS, allow: '/' }, { userAgent: '*', allow: '/', disallow: ['/admin', '/api', '/md/'] }, ], sitemap: 'https://iamsrikanth.com/sitemap.xml', } } ``` Worth knowing what these names mean. There are **search** crawlers (`OAI-SearchBot`, `PerplexityBot`, `Claude-SearchBot`) that decide whether you can be cited in answers. There are **user-triggered** fetchers (`ChatGPT-User`, `Claude-User`, `Perplexity-User`) that load a page when someone pastes your link. And there are **training** crawlers and opt-out tokens (`GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`, `Applebot-Extended`). For a portfolio whose whole job is to be found, I allow all three kinds. Blocking `Google-Extended` doesn't affect normal Google Search ranking. It only controls whether Google uses your content for Gemini. The same goes for `Applebot-Extended` and Apple's models. ## 2. An `llms.txt` (and `llms-full.txt`) [`llms.txt`](https://llmstxt.org) is a proposal by Jeremy Howard from 2024. It's a markdown file at the root of your site with a one-paragraph summary and a curated list of links, written for language models instead of browsers. Mine is a Next.js route handler that builds the file from the same Supabase data as the pages: ```txt # Srikanth V — Frontend Developer in Bangalore > Srikanth V is a frontend engineer (frontend developer) based in Bangalore, India… ## Projects - [Hyqoo](https://iamsrikanth.com/projects/hyqoo.md): Migrated Hyqoo's site off WordPress… ## Optional - [Full context](https://iamsrikanth.com/llms-full.txt): Profile, every case study and every article in one file ``` `llms-full.txt` is the long version: the profile, every case study and every article in full, in one file. An agent that wants everything makes one request instead of crawling twenty pages. Both are `force-static`, so they cost nothing at request time. The admin calls `revalidatePath('/', 'layout')` after every edit, which refreshes them along with everything else. ## 3. A markdown version of every page HTML is a noisy format for a model: navigation, animation markup, inline SVG. So every page also has a plain markdown version: - `/index.md` — home - `/projects.md` and `/projects/.md` - `/blogs.md` and `/blogs/.md` There's one route handler, `app/md/[[...path]]/route.js`, and a few rewrites that map the `.md` URLs onto it. A rewrite on the `Accept` header means an agent that asks for markdown gets it **on the normal URL**: ```js // next.config.mjs async rewrites() { const wantsMarkdown = [{ type: 'header', key: 'accept', value: '.*text/markdown.*' }] return { beforeFiles: [ { source: '/', has: wantsMarkdown, destination: '/md' }, { source: '/:section(projects|blogs)/:slug?', has: wantsMarkdown, destination: '/md/:section/:slug?' }, { source: '/index.md', destination: '/md' }, { source: '/:section(projects|blogs).md', destination: '/md/:section' }, { source: '/:section(projects|blogs)/:slug.md', destination: '/md/:section/:slug' }, ], } } ``` ```bash curl -H 'Accept: text/markdown' https://iamsrikanth.com/projects/hyqoo ``` Two details keep this from hurting normal SEO: - Every markdown response sends `Link: ; rel="canonical"`. Google respects canonical links in HTTP headers, so the HTML page stays the one that's indexed. - Every HTML page advertises its markdown version with ``. In Next.js that's one line in the metadata: `alternates.types['text/markdown']`. ## 4. Structured data that answers the obvious questions Assistants and search engines both lean on JSON-LD to work out *who* a page is about. The home page describes a `Person` with the details someone hiring would ask about: name and alternate names, job title, location (with "Bengaluru" as an alternate name for Bangalore), an `Occupation` with skills, current employer, and `sameAs` links to GitHub and LinkedIn. There's also a `ProfilePage`. Case studies are `CreativeWork`, articles are `BlogPosting`, and every page has `BreadcrumbList`. ## What's proven and what isn't I want to be straight about this, because a lot of "GEO" content online isn't: - **robots.txt is real.** If a crawler is blocked, it doesn't read your site. That part is binary. - **Structured data and clean, server-rendered HTML are real.** They help classic search, and AI search is built on top of classic search indexes. - **`llms.txt` is a proposal, not a standard.** I haven't seen a major AI provider publicly commit to reading it for search. Claims like "30% more AI citations" don't come with studies behind them. It's cheap to add, so I have one. But it's a bet, not a guarantee. - **Content still wins.** No file makes a thin site citable. What gets an engineer recommended is specific, first-hand writing, like the [Core Web Vitals case study](/blogs/core-web-vitals-65-to-99-nextjs) or the [WordPress migration](/blogs/migrating-wordpress-to-nextjs) on this site. The good news is that all of this is a few small files on top of a site you're building anyway. Get the crawlers in, give them a clean version of your content, describe yourself in structured data, and spend the rest of your time writing things worth citing.