Sitemap and Robots.txt for Non-Technical Founders
Learn sitemap and robots.txt setup for founders. Copy-paste examples, avoid breaking your site, and get organic visibility in 30 minutes.
The Problem You're Facing
You shipped. Your product works. But Google can't find half your pages, and you're bleeding organic traffic you didn't even know existed.
The culprit? Two files sitting in the root of your domain that you probably haven't touched since launch: robots.txt and your XML sitemap.
These aren't optional. They're the difference between Google crawling your site efficiently and wasting crawl budget on pages nobody needs. They're the difference between your new feature landing in search results in days versus months.
The brutal truth: most founders don't understand what these files do. So they either ignore them (and watch their crawl efficiency tank) or they misconfigure them (and accidentally block Google from indexing their money pages).
This guide is for non-technical founders. No jargon. No fluff. Just copy-paste examples you can implement in 30 minutes, plus the reasoning behind each directive so you don't accidentally nuke your organic visibility.
Prerequisites: What You Need Before Starting
Before you touch anything, make sure you have:
Access to your domain's root directory. This means you can upload or edit files in the top-level folder of your website (where example.com/robots.txt would live, not example.com/blog/robots.txt). If you're on a platform like Webflow, Squarespace, or Wix, you'll need to use their built-in SEO settings instead of manually uploading files.
A text editor. Notepad, VS Code, Sublime—it doesn't matter. You're just writing plain text.
Access to Google Search Console. This is where you'll verify your sitemap and robots.txt are working. If you don't have it set up, go here and add your domain. It takes five minutes.
A basic understanding of your site structure. You should know roughly how many pages you have, what the main sections are, and whether you have pages you don't want Google to index (like login pages, thank you pages, or staging environments).
If you're using a platform like WordPress, Shopify, or Next.js, these files might already exist. Don't assume they're correct. We'll audit them in the next section.
What Robots.txt Actually Does (And Doesn't Do)
Let's clear up the biggest misconception first: robots.txt doesn't prevent Google from indexing your pages.
That's what the noindex meta tag does. Robots.txt is a crawl directive. It tells search engines (and other bots) which parts of your site they're allowed to crawl. It's a suggestion, not a law. Google usually respects it. But if you block a page in robots.txt and link to it from elsewhere, Google might still index it.
Think of robots.txt as a traffic cop in your server. It says: "Hey, crawler, you can go down this street, but not that one."
Why does this matter? Because crawl budget is finite. Google allocates a certain number of requests per day to your domain. If you're wasting crawl budget on pages that don't matter (like duplicate test pages, old blog drafts, or admin dashboards), you're losing crawls that could be spent on your revenue-generating pages.
Robots.txt is also the place where you tell Google where your XML sitemap lives. This is critical. We'll get to that in a moment.
What an XML Sitemap Does
Your XML sitemap is a map of your site's structure. It's a list of all your important pages, organized in a way that search engines can parse instantly.
Unlike robots.txt, a sitemap doesn't block anything. It suggests what Google should crawl and index. It also tells Google when each page was last updated and how important it is relative to other pages on your site.
For a founder with a small site (under 500 pages), a sitemap is less critical than robots.txt. But it becomes essential as you scale. And if you're doing any kind of programmatic SEO (generating hundreds of pages automatically), a dynamic sitemap is non-negotiable.
The key difference: robots.txt is about control. Sitemaps are about guidance.
Step 1: Audit Your Current Robots.txt
First, check if you already have a robots.txt file.
Go to yourdomain.com/robots.txt in your browser. If you see a text file with directives, you have one. If you get a 404, you don't.
If you have one, copy the entire contents and paste it into a text editor. We're going to review it line by line.
Here's what to look for:
User-agent: * — This applies to all bots.
Disallow: /admin — This blocks crawlers from accessing /admin and anything under it.
Allow: /admin/public — This allows crawlers to access /admin/public even though /admin is disallowed. (Order matters here; more specific rules override broader ones.)
Crawl-delay: 5 — This tells crawlers to wait 5 seconds between requests. This is outdated. Don't use it.
Request-rate: 10/1s — This limits crawlers to 10 requests per second. Also outdated. Ignore it.
Sitemap: https://yourdomain.com/sitemap.xml — This tells crawlers where your sitemap is. You probably don't have this. We'll add it.
If your robots.txt is blocking important pages (like /blog or /pricing), that's a problem we'll fix in the next step.
Step 2: Create a Non-Technical Robots.txt (Copy-Paste Ready)
Here's a robots.txt that works for 90% of founder sites. Copy this exactly:
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /draft/
Disallow: /test/
Disallow: /staging/
Disallow: /?utm_
Disallow: /?ref=
Disallow: /*?sort=
Disallow: /*?page=
Disallow: /checkout/thank-you/
Disallow: /api/
Disallow: /tmp/
User-agent: AdsBot-Google
Allow: /
User-agent: Googlebot
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
Let's break down what each section does:
User-agent:* — This applies to all crawlers (Google, Bing, OpenAI, Perplexity, etc.).
Allow: / — By default, allow everything. (This is the baseline.)
Disallow: /admin/ — Block the admin dashboard. You don't want this indexed.
Disallow: /private/ — Block any private folders. Adjust this path based on your site.
Disallow: /draft/ — Block draft posts. If your CMS uses a different path (like /blog/draft), change this.
Disallow: /test/ and Disallow: /staging/ — Block test and staging environments. If you're running staging at staging.yourdomain.com instead of a subfolder, this doesn't apply.
Disallow: /?utm_ — Block URLs with UTM parameters. These are tracking parameters (utm_source, utm_medium, etc.). They create duplicate content and waste crawl budget. Google usually ignores this, but it's good practice.
Disallow: /?ref= — Block referral parameter URLs. Same reason.
Disallow: /*?sort=andDisallow: /*?page= — Block pagination and sorting parameters. If your site has a lot of filtered product pages or paginated archives, this prevents Google from crawling infinite variations of the same content.
Disallow: /checkout/thank-you/ — Block thank-you pages. These aren't meant to be indexed and won't convert.
Disallow: /api/ — Block API endpoints. These aren't web pages; they're for machines.
Disallow: /tmp/ — Block temporary files.
User-agent: AdsBot-Google and User-agent: Googlebot — These are special rules for Google's ad and organic crawlers. We're allowing everything for them. (This overrides the general rules above for these specific bots.)
Sitemap: https://yourdomain.com/sitemap.xml — This tells Google where your sitemap is. Replace yourdomain.com with your actual domain.
Important: This robots.txt assumes you want Google to crawl most of your site. If you have pages you don't want indexed, add a Disallow: line for them. But remember: robots.txt doesn't prevent indexing. For critical pages (like login or password reset), use a noindex meta tag instead.
Step 3: Customize Your Robots.txt for Your Site
Now, customize the template above for your specific site.
If you're on WordPress: You likely have a robots.txt already. Check your WordPress SEO plugin settings (Yoast, Rankmath, etc.) to see if you can edit it there. If not, you can upload a custom one via FTP or your hosting control panel.
If you have pages you want to block: Add them as Disallow: lines. For example, if you have a /members-only/ section, add Disallow: /members-only/. If you have multiple sections, add multiple lines.
If you have a lot of filtered or paginated content: Add parameter-blocking rules. For example, if you have a product filter by color at /products?color=red, add Disallow: /*?color=. This prevents Google from crawling every color variation as a separate page.
If you're on a subdomain (like app.yourdomain.com): Create a separate robots.txt for that subdomain. Subdomains are treated as separate sites by Google. The robots.txt at app.yourdomain.com/robots.txt won't affect yourdomain.com/robots.txt.
If you have a staging environment: If it's on a subdomain (like staging.yourdomain.com), create a robots.txt that blocks everything:
User-agent: *
Disallow: /
This prevents Google from indexing your staging site.
Once you've customized your robots.txt, save it as a plain text file named robots.txt (no .txt extension in the filename itself—the filename is robots.txt).
Step 4: Upload Your Robots.txt
Now you need to upload this file to the root of your domain.
If you're using a traditional host (Bluehost, GoDaddy, etc.): Use FTP or your hosting control panel's file manager. Navigate to the public_html folder (or equivalent) and upload robots.txt there.
If you're using Vercel (Next.js): Add a public/robots.txt file to your project. Vercel automatically serves files in the public folder at the root of your domain.
If you're using Netlify: Same as Vercel. Add robots.txt to your public folder.
If you're using Webflow, Squarespace, or Wix: These platforms have built-in SEO settings. Don't upload a file. Instead, go to your SEO settings and find the robots.txt editor. Paste your robots.txt content there.
If you're using WordPress: Use a plugin like Yoast SEO or Rankmath to edit your robots.txt. Or, if your hosting allows, upload it via FTP.
Once uploaded, verify it's live by going to yourdomain.com/robots.txt in your browser. You should see your robots.txt file displayed as plain text.
Step 5: Create Your XML Sitemap
Now let's create your XML sitemap.
First, check if you already have one. Go to yourdomain.com/sitemap.xml. If you see XML code, you have one. If you get a 404, you don't.
If you already have a sitemap: Skip to Step 6.
If you don't have one: Here's a basic template:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://yourdomain.com/</loc>
<lastmod>2024-01-15</lastmod>
<changefreq>weekly</changefreq>
1.0</priority>
</url>
<url>
<loc>https://yourdomain.com/about</loc>
<lastmod>2024-01-10</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
<url>
<loc>https://yourdomain.com/pricing</loc>
<lastmod>2024-01-12</lastmod>
<changefreq>weekly</changefreq>
<priority>0.9</priority>
</url>
<url>
<loc>https://yourdomain.com/blog/first-post</loc>
<lastmod>2024-01-08</lastmod>
<changefreq>never</changefreq>
<priority>0.7</priority>
</url>
</urlset>
<p>Let's break down each element:
— This is the XML declaration. Don't change it.
<url> block.
https://.
always, hourly, daily, weekly, monthly, yearly, never. This is a hint to Google, not a command. Use weekly for blog posts, monthly for static pages, never for old posts.
For a small site (under 100 pages), you can manually create a sitemap using this template. For larger sites, use a tool like Screaming Frog (free for up to 500 URLs) or XML Sitemap Generator.
Important: Your sitemap can have a maximum of 50,000 URLs. If you have more, create multiple sitemaps (sitemap1.xml, sitemap2.xml, etc.) and reference them in a sitemap index file. But if you're at 50,000 URLs, you have bigger problems to solve first.
Step 6: Upload Your Sitemap
Save your sitemap as sitemap.xml (plain text, XML format) and upload it to the root of your domain, just like you did with robots.txt.
If you're on Vercel, Netlify, or similar: Add it to your public folder.
If you're on WordPress: Use Yoast, Rankmath, or your hosting's file manager.
If you're on Webflow, Squarespace, or Wix: These platforms generate sitemaps automatically. You don't need to create one manually. But verify it's enabled in your SEO settings.
Once uploaded, verify it's live by going to yourdomain.com/sitemap.xml in your browser. You should see XML code.
Step 7: Submit Your Sitemap to Google
Now tell Google about your sitemap.
Go to Google Search Console and sign in with the account that owns your domain.
Select your property (your domain).
In the left sidebar, click Sitemaps.
Click Add a sitemap.
Enter sitemap.xml (just the filename, not the full URL).
Click Submit.
Google will validate your sitemap and start crawling the URLs in it. This usually takes a few hours to a few days.
Pro tip: If you have multiple sitemaps, you can create a sitemap index and submit that instead. A sitemap index is just a file that lists all your sitemaps. It looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://yourdomain.com/sitemap1.xml</loc>
</sitemap>
<sitemap>
<loc>https://yourdomain.com/sitemap2.xml</loc>
</sitemap>
</sitemapindex>
Submit this as sitemap_index.xml in Google Search Console.
Step 8: Verify Your Robots.txt in Google Search Console
While you're in Google Search Console, let's make sure your robots.txt is correct.
Click Settings (gear icon in the top right).
Click Crawl settings.
You should see your robots.txt file. If there are any errors, Google will flag them here.
Also, in the left sidebar, click URL Inspection. Pick a page on your site and check if Google can crawl it. If you see a warning like "Blocked by robots.txt," that means your robots.txt is preventing Google from crawling that page. You need to fix it.
Advanced: Robots.txt Directives for AI Crawlers
Here's where things get interesting. Google isn't the only crawler anymore. OpenAI, Anthropic, Perplexity, and other AI companies are crawling the web to train their models.
If you want your content cited by ChatGPT, Claude, or Perplexity, you need to allow their crawlers. If you don't want your content used for AI training, you need to block them.
Here's how to add AI-specific rules to your robots.txt:
To allow AI crawlers:
User-agent: GPTBot
Allow: /
User-agent: CCBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: anthropic-ai
Allow: /
To block AI crawlers:
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
If you're doing AEO (AI Engine Optimization), you want to allow these crawlers. If you're protecting proprietary content, block them.
For more on this, check out The AEO Playbook: Getting Cited by Claude, ChatGPT, and Gemini to understand how to optimize for AI citations.
Common Mistakes to Avoid
Mistake 1: Blocking your entire site accidentally.
If your robots.txt says Disallow: /, you're blocking everything. Google won't index anything. This is a common mistake when someone copies a robots.txt from a staging environment.
Fix: Make sure your robots.txt starts with Allow: / or only has specific Disallow: rules.
Mistake 2: Blocking CSS, JavaScript, or images.
Some old robots.txt files have rules like Disallow: /*.css or Disallow: /*.js. This breaks Google's ability to render your pages. Google needs to see your CSS and JavaScript to understand your site.
Fix: Remove any rules that block static assets. If you're concerned about crawl budget, block parameter-based URLs instead.
Mistake 3: Using robots.txt to hide sensitive pages.
If you have a login page or admin dashboard, don't rely on robots.txt to hide it. Use a noindex meta tag or password protection instead. Robots.txt is public; anyone can read it and see what you're trying to hide.
Mistake 4: Forgetting to update your sitemap when you add new pages.
If you're adding blog posts regularly, your sitemap becomes outdated. Google will crawl old URLs and miss new ones.
Fix: If you're on WordPress or a platform with automatic sitemap generation, this is handled for you. If you're manually maintaining a sitemap, set a reminder to update it weekly.
Mistake 5: Creating a sitemap with broken links.
If your sitemap includes URLs that return 404s, Google wastes crawl budget on them. This also signals to Google that your site is poorly maintained.
Fix: Before submitting your sitemap, spot-check a few URLs. Make sure they're all live.
Pro Tips for Founders Scaling Fast
If you're adding 100+ pages per week (programmatic SEO): Use a dynamic sitemap. Instead of manually updating a static XML file, generate your sitemap from your database on the fly. Here's a simple Next.js example:
// pages/sitemap.xml.js
export default function Sitemap() {}
export async function getServerSideProps({ res }) {
const posts = await fetch('your-api/posts').then(r => r.json());
const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
${posts.map(post => `
<url>
<loc>https://yourdomain.com/blog/${post.slug}</loc>
<lastmod>${post.updatedAt}</lastmod>
</url>
`).join('')}
</urlset>`;
res.setHeader('Content-Type', 'text/xml');
res.write(sitemap);
res.end();
return { props: {} };
}
This regenerates your sitemap every time it's requested, so it's always up to date. Learn more about scaling with Programmatic SEO for Startups: A 30-Day Playbook.
If you have multiple domains or subdomains: Create separate sitemaps and robots.txt files for each. Google treats subdomains as separate sites. A robots.txt at yourdomain.com/robots.txt doesn't apply to blog.yourdomain.com/robots.txt.
If you're doing client-side rendering (React, Vue, etc.): Make sure your robots.txt isn't blocking anything. Client-side rendered sites are harder for Google to crawl. You need to give Google every advantage. Check out The Hidden Cost of Client-Side Rendering in 2026 to understand the crawl implications.
If you want to optimize for AI citations: Allow the AI crawlers (GPTBot, CCBot, PerplexityBot) in your robots.txt and ensure your sitemap includes your best content. AI models prioritize fresh, authoritative content. Learn more in Perplexity Now Cites Schema-Marked Pages 3× More.
Monitoring and Maintenance
Once your robots.txt and sitemap are live, you're not done. You need to monitor them.
Weekly: Check Google Search Console for crawl errors. If Google can't crawl a page, you'll see a warning. Fix these immediately.
Monthly: Review your sitemap. Are you adding new pages? Remove old ones? Update your sitemap accordingly.
Quarterly: Audit your robots.txt. Are you blocking anything you shouldn't be? Are you allowing anything you should block?
Set reminders for these. It takes 10 minutes per week and saves you from silent organic traffic loss.
The Difference This Makes
Here's what happens when you get robots.txt and sitemaps right:
Google crawls your site faster. It spends less time on junk pages and more time on revenue-generating pages.
New pages get indexed in days instead of weeks. You can ship a new feature or blog post and have it in search results by the next day.
You stop wasting crawl budget on duplicate content, parameter variations, and pages nobody needs.
Your crawl efficiency score in Google Search Console goes up. This is a signal that your site is well-organized.
For founders shipping fast, this matters. If you're adding 100 AI-generated blog posts (like what SEOABLE delivers in under 60 seconds), a proper robots.txt and sitemap means Google crawls and indexes all of them within a week. Without it, you're looking at months.
Look at Solo Founder Hits 50K Organic/mo in Four Months for a real example of how content at scale requires proper crawl infrastructure.
Troubleshooting
Q: I uploaded my robots.txt, but Google says it can't find it.
A: Make sure it's in the root of your domain (yourdomain.com/robots.txt, not yourdomain.com/blog/robots.txt). Also, check that your file is named exactly robots.txt with no extension. If you're on a platform like Webflow or Squarespace, use their built-in SEO settings instead of uploading a file.
Q: My robots.txt has an error in Google Search Console. What does it mean?
A: Google will tell you which line has the error. Common issues: typos in directives, incorrect syntax, or invalid characters. Copy the exact line that's causing the error and compare it to the examples in this guide.
Q: I blocked a page in robots.txt, but it's still showing up in Google search results.
A: Robots.txt doesn't prevent indexing; it only prevents crawling. If the page is linked from elsewhere, Google might still index it. Use a noindex meta tag instead to prevent indexing.
Q: How often does Google crawl my sitemap?
A: It depends on your site's authority and update frequency. New sites might be crawled weekly. Established sites might be crawled daily. You can't control this directly, but you can make Google's job easier by keeping your sitemap fresh and your content high-quality.
Q: Can I have multiple sitemaps?
A: Yes. You can have up to 50,000 URLs per sitemap. If you have more, create multiple sitemaps and submit them via a sitemap index file.
Q: Should I include pagination pages in my sitemap?
A: No. Block them in robots.txt with Disallow: /*?page=. Include only the canonical versions of your content.
Next Steps: Get Your Full SEO Foundation in 60 Seconds
Robots.txt and sitemaps are the foundation. But they're just the beginning.
You also need:
- A technical SEO audit (broken links, crawl errors, indexation issues)
- A keyword roadmap (what to write about to actually get traffic)
- Content (100 AI-generated blog posts optimized for your keywords)
- Schema markup (so Google understands what your pages are about)
- Backlink strategy (so Google trusts you)
Doing this manually takes weeks. SEOABLE does it in under 60 seconds for $99. You get a domain audit, brand positioning, keyword roadmap, and 100 AI-generated blog posts ready to ship.
For founders who've shipped but lack organic visibility, this is the fastest path to traction.
If you want to dive deeper into technical SEO, check out Google's March 2026 Core Update: What Changed for Startups to see what actually moves the needle for small sites.
Summary: Key Takeaways
Robots.txt is a crawl directive. It tells search engines which parts of your site to crawl. It doesn't prevent indexing. Use it to block junk pages and parameters, and to point Google to your sitemap.
Your sitemap is a roadmap. It lists all your important pages and tells Google how often they change and how important they are. For small sites, it's optional. For scaling sites, it's essential.
The template in this guide works for 90% of founder sites. Copy it, customize it for your site structure, and upload it to the root of your domain.
Monitor your crawl health in Google Search Console. Check weekly for errors. Fix them immediately. This prevents silent organic traffic loss.
If you're scaling fast (adding 100+ pages per week), automate your sitemap. Generate it from your database instead of manually maintaining it.
Allow AI crawlers if you want AI citations. Add rules for GPTBot, CCBot, and PerplexityBot to your robots.txt. Block them if you're protecting proprietary content.
Don't use robots.txt to hide sensitive pages. Use noindex meta tags or password protection instead. Robots.txt is public.
Get these two files right, and you've eliminated a major blocker to organic visibility. Your pages will crawl faster, index quicker, and rank sooner.
For founders shipping at speed, that's the difference between staying invisible and getting discovered.
Get the next
dispatch on Monday.
One email per week with the most important SEO and AEO moves for founders. Unsubscribe in one click.