Shopify Robots.txt: Editing It Without an App

Why You Need to Edit Your Shopify Robots.txt

Your robots.txt file is a direct instruction set to Google, Bing, and every other search engine crawler. It tells them what to crawl, what to skip, and how to prioritize your site.

Most Shopify stores ship with the default robots.txt. This is a mistake.

The default file doesn't account for your specific business needs. It doesn't block duplicate product pages, filter parameters, or admin routes that waste crawl budget. It doesn't protect your staging environment or prevent indexing of search result pages that dilute your organic visibility.

Worse, you don't need an app to fix this. Shopify lets you edit robots.txt natively—directly in your theme code. No monthly subscription. No vendor lock-in. No waiting for support tickets.

This guide walks you through exactly what to change, what to leave alone, and why. By the end, you'll have a robots.txt file that works for your store, not against it.

Prerequisites: What You'll Need

Before you start, confirm you have the right access and setup:

• Shopify Admin Access: You need permission to edit theme code. This typically means you're the store owner or have the "Manage themes" permission.
• A Live Theme: You'll be editing the active theme that's currently live on your store. If you want to test first, create a duplicate theme and work there.
• Basic Understanding of Robots.txt: You don't need to be a developer, but you should know that robots.txt controls crawler access. If you're completely new to the concept, read our guide on writing your first robots.txt file first.
• Access to Shopify's Theme Code Editor: This is built into Shopify admin. No external tools required.
• A Text Editor (Optional): You can write your robots.txt changes in Notepad, VS Code, or any plain-text editor before pasting them into Shopify. This helps you avoid typos.

If you're unsure about your permission level, ask your Shopify store owner or check your admin settings under "Apps and sales channels" → "Apps" → "App and sales channel settings."

Step 1: Access Your Shopify Theme Code Editor

Start in your Shopify admin dashboard.

1. Log in to your Shopify store at admin.shopify.com.
2. From the left sidebar, click Sales channels (or Apps in older admin versions).
3. Find and click Themes.
4. You'll see your active theme listed. Click the ... (three dots) next to it, then select Edit code.
5. You're now in the theme code editor. This is where all theme files live—liquid templates, CSS, JavaScript, and more.

The file browser on the left shows your theme structure. You're looking for a file called robots.txt.liquid or robots.txt.

Important: Shopify stores that have never customized robots.txt won't have this file visible yet. That's normal. You'll create it in the next step.

If you do see robots.txt.liquid already, click it to open it. Skip to Step 3.

Step 2: Create the robots.txt.liquid File (If It Doesn't Exist)

If you don't see a robots.txt file in your theme code editor, you need to create one.

1. In the theme code editor, look for the Add a new file button (usually a + icon or labeled "Add file").
2. Click it.
3. A dialog box appears asking for a filename. Type: robots.txt.liquid
4. Make sure you're creating this file in the root directory (not inside a folder). Shopify will automatically place it correctly.
5. Click Create file.

You now have a blank robots.txt.liquid file. This is where you'll paste your custom robots.txt rules.

Why .liquid? Shopify uses Liquid templating language. The .liquid extension tells Shopify to process this file with Liquid before serving it as robots.txt. This lets you use dynamic content if needed (though most robots.txt files are static).

Step 3: Understand What to Keep, What to Change

Before you paste code, you need to understand the anatomy of a robots.txt file and what Shopify's defaults do.

Every robots.txt file has this basic structure:

User-agent: [crawler name]
Disallow: [path to block]
Allow: [path to allow]
Crawl-delay: [seconds]

A User-agent is the name of a crawler (Googlebot, Bingbot, etc.). A single robots.txt can have rules for multiple crawlers.

Disallow tells a crawler not to visit a path. For example, Disallow: /admin/ blocks crawlers from accessing your admin panel.

Allow overrides a Disallow rule for specific paths. This is useful when you want to block a directory but allow certain pages within it.

Crawl-delay tells a crawler how many seconds to wait between requests. This prevents your server from being hammered.

Shopify's default robots.txt typically looks like this:

User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /?*
Disallow: /*?*
Disallow: /*%
Disallow: /search
Disallow: /pages/*/edit
Disallow: /products/*/edit
Disallow: /apps/
Disallow: /cdn/
Disallow: /carts/
Disallow: /account
Disallow: /addresses
Disallow: /password
Disallow: /policy/
Disallow: /policies/
Disallow: /index.php
Disallow: /*?*sort_by*
Disallow: /*?*filter*
Disallow: /*?*page*
Disallow: /*?*preview_id*
Disallow: /*?*variant*
Disallow: /*?*_=*
Disallow: /cdn-cgi/
Disallow: /*.json
Disallow: /*.xml
Disallow: /*?*fb_action_ids*
Disallow: /*?*fb_source*
Disallow: /*?*fbclid*
Disallow: /*?*gclid*
Disallow: /*?*msclkid*
Disallow: /*?*mc_cid*
Disallow: /*?*mc_eid*
Disallow: /*?*utm_*
Disallow: /*?*_ga*

Sitemap: https://example.com/sitemap.xml

This is actually pretty good. It blocks admin pages, cart, checkout, search, and tracking parameters. But it might not be perfect for your store.

What you should keep:

• All the /admin/, /cart, /orders, /checkouts/ blocks. These are non-public pages.
• The /?* and /*?* rules that block query strings. These prevent duplicate content from filters and sorting.
• The Sitemap line pointing to your sitemap.

What you might change:

• If you have a staging environment, add Disallow: /staging/ or whatever your staging path is.
• If you have a login page at /account but want to keep it public, remove that line.
• If you have internal pages that shouldn't be indexed (like thank-you pages), add Disallow: /thank-you/.
• If you have collections with duplicate content, add rules to block low-value filter combinations.

For most Shopify stores, the default is 80% there. You just need to add 3–5 custom rules for your specific business.

Step 4: Write Your Custom Robots.txt

Now you'll create your actual robots.txt content.

Here's a solid template for most Shopify stores:

User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /?*
Disallow: /*?*sort_by*
Disallow: /*?*filter*
Disallow: /*?*page*
Disallow: /*?*preview_id*
Disallow: /*?*variant*
Disallow: /*?*_=*
Disallow: /cdn-cgi/
Disallow: /*.json
Disallow: /*.xml
Disallow: /*?*fb_action_ids*
Disallow: /*?*fb_source*
Disallow: /*?*fbclid*
Disallow: /*?*gclid*
Disallow: /*?*msclkid*
Disallow: /*?*mc_cid*
Disallow: /*?*mc_eid*
Disallow: /*?*utm_*
Disallow: /*?*_ga*
Disallow: /search
Disallow: /pages/*/edit
Disallow: /products/*/edit
Disallow: /apps/
Disallow: /cdn/
Disallow: /carts/
Disallow: /account
Disallow: /addresses
Disallow: /password
Disallow: /policy/
Disallow: /policies/
Disallow: /index.php
Disallow: /collections/*/products
Disallow: /*/collections/
Disallow: /discount/*

User-agent: AdsBot-Google
Disallow: 

User-agent: Googlebot-Image
Disallow: 

Sitemap: https://yourdomain.com/sitemap.xml

Let me break down the additions:

• Disallow: /collections/*/products: Blocks the paginated product listing pages within collections. This prevents duplicate content when you have 100 products across 10 pages.
• Disallow: /*/collections/: Blocks nested collection pages (some Shopify themes create these).
• Disallow: /discount/*: Blocks discount code pages. These are often auto-generated and create duplicate content.
• User-agent: AdsBot-Google and Disallow: (blank): Allows Google's ad bot to crawl everything. This helps with Google Shopping and ads performance.
• User-agent: Googlebot-Image and Disallow: (blank): Allows Google's image crawler to index your product images.

Customize for your store:

• Replace yourdomain.com with your actual domain.
• If you have a staging environment, add Disallow: /staging/ before the Sitemap line.
• If you have a blog at /blog/, consider whether you want it indexed. If yes, don't add a Disallow rule. If no, add Disallow: /blog/.
• If you have internal pages (like /thank-you/ after purchase), add Disallow: /thank-you/.
• If you have a /test/ or /dev/ folder, block it: Disallow: /test/ and Disallow: /dev/.

Don't overthink this. The goal is to block duplicate and non-public pages, not to hide your entire store.

Step 5: Paste Your Robots.txt Into Shopify

Now you're back in the theme code editor with your robots.txt.liquid file open (or newly created and blank).

1. Click inside the code editor window.
2. Select all existing content (Ctrl+A or Cmd+A) and delete it.
3. Paste your custom robots.txt content.
4. Do not wrap it in Liquid tags or HTML. Just paste the raw robots.txt rules.
5. Click Save (top right, or Cmd+S / Ctrl+S).

That's it. Your robots.txt is now live.

Shopify automatically serves this file at yourdomain.com/robots.txt. You can verify by visiting that URL in your browser—you should see your robots.txt rules displayed as plain text.

Step 6: Verify Your Robots.txt Is Live

Don't assume it worked. Verify.

1. Open a new browser tab.
2. Go to https://yourdomain.com/robots.txt (replace yourdomain.com with your actual domain).
3. You should see your robots.txt file displayed in plain text.
4. Check that all your custom rules are there and correctly formatted.

If you see a 404 error or the old robots.txt, clear your browser cache (Ctrl+Shift+Delete or Cmd+Shift+Delete) and try again. Sometimes browsers cache the old version.

If you still don't see your new robots.txt after 5 minutes, go back to the theme code editor and confirm the file is saved. Check for typos in the filename—it must be exactly robots.txt.liquid.

Step 7: Submit Your Updated Robots.txt to Google

Once your robots.txt is live, tell Google about it.

1. Go to Google Search Console.
2. Select your property (your domain).
3. In the left sidebar, click Settings.
4. Scroll down to Crawlers and indexing → robots.txt tester.
5. Click robots.txt tester.
6. You'll see your robots.txt file displayed. Google has already fetched it.
7. Review the rules. If you see any errors (red X marks), fix them in your Shopify theme editor.

Google doesn't require you to "submit" robots.txt—it fetches it automatically. But checking the tester confirms everything is formatted correctly.

You should also submit your sitemap to Google if you haven't already. Your robots.txt file includes a Sitemap line, but explicitly submitting it in Search Console speeds up crawling.

What NOT to Do: Common Mistakes

These will break your SEO or waste crawl budget:

Don't Block Your Entire Site

User-agent: *
Disallow: /

This tells all crawlers to go away. Your site won't be indexed. Only do this if you're taking your store offline.

Don't Use robots.txt to Hide Sensitive Data

Robots.txt is public. Anyone can read it at yourdomain.com/robots.txt. If you want to truly hide something (like an admin panel or customer data), use HTTP authentication, not robots.txt. For more on this decision, read our guide on noindex vs. robots.txt.

Don't Block Your Homepage or Main Collections

If your robots.txt says Disallow: / or Disallow: /collections/, you're invisible to Google. Don't do this.

Don't Use Wildcards Incorrectly

Robots.txt uses * as a wildcard, but it works differently than you might expect. /*?*filter* blocks any URL with / followed by anything, then ?, then anything, then filter. This is correct for blocking filter parameters. But Disallow: * doesn't block everything—it blocks nothing. Use Disallow: / to block everything.

Don't Forget the Sitemap Line

Your robots.txt should always end with a Sitemap line pointing to your sitemap.xml. This tells Google where to find all your pages. If you haven't created a sitemap yet, follow this guide to generate one.

Don't Change robots.txt Every Week

Once you've set it, leave it alone. Constantly changing robots.txt confuses crawlers and can cause indexing issues. Only update it if you're adding a new staging environment, blocking a new path, or restructuring your site.

Pro Tips: Advanced Customizations

If you want to go deeper, here are some advanced moves:

Block Specific Collections to Preserve Crawl Budget

If you have 50 collections but only 10 drive revenue, consider blocking the low-value ones:

Disallow: /collections/clearance/
Disallow: /collections/archive/

This tells Google to skip those collections and focus on your money-makers. Use this carefully—only block collections you truly don't want indexed.

Set a Crawl-Delay for High-Traffic Stores

If your server gets hammered by crawlers, you can slow them down:

User-agent: Googlebot
Crawl-delay: 2

User-agent: Bingbot
Crawl-delay: 2

This tells Google and Bing to wait 2 seconds between requests. Use this only if your server is struggling. For most stores, it's unnecessary.

Allow Google to Crawl Specific Pages Even If They Match a Disallow Rule

Disallow: /collections/*/
Allow: /collections/bestsellers/

This blocks all collections except bestsellers. Google respects Allow rules even when they contradict Disallow rules.

Block Specific Query Parameters

If you have custom URL parameters that create duplicate content, block them:

Disallow: /*?ref=*
Disallow: /*?source=*

This prevents Google from crawling different versions of the same page with different tracking parameters.

Understanding Shopify's robots.txt in Context

Your robots.txt is one piece of a larger SEO foundation. It works alongside other files and settings. For instance, robots.txt, sitemaps, and canonical tags are the three files most founders misconfigure. Robots.txt controls crawl access, but sitemaps tell Google which pages exist, and canonical tags prevent duplicate content issues.

If you're running a Shopify store, you should also ensure you have proper canonical tags set up to handle www vs. non-www versions of your domain. You should also set up HTTPS correctly so all traffic is encrypted—Google rewards this.

For e-commerce specifically, there's another layer: AI Engine Optimization (AEO) for e-commerce. While robots.txt controls traditional search engine crawlers, AEO helps your products get cited by ChatGPT, Perplexity, and other AI search tools. Your robots.txt should allow these crawlers to access your product pages.

Monitoring and Maintenance

Once your robots.txt is live, you don't need to babysit it. But check it occasionally:

Monthly: Review in Google Search Console

Go to the robots.txt tester in Google Search Console and confirm your file is still there and correctly formatted. Google should show zero errors.

Quarterly: Check for Indexing Issues

In Google Search Console, go to Pages and look at your indexed pages. If you see pages you meant to block (like /search or /cart), your robots.txt might not be working. Check the formatting and resubmit.

When You Make Site Changes: Update robots.txt

If you add a new staging environment, create a new internal tool, or restructure your collections, update robots.txt accordingly. For example, if you launch a new /blog/ section, decide whether you want it indexed. If yes, don't add a Disallow rule. If no, add Disallow: /blog/.

If you're doing a major restructure like a domain migration, you'll need to set up 301 redirects alongside your robots.txt changes.

Testing Your Robots.txt Rules

Before you go live with aggressive blocking, test your rules. Google Search Console has a built-in tester:

1. Go to Search Console.
2. Select your property.
3. Click Settings → Crawlers and indexing → robots.txt tester.
4. You'll see your current robots.txt.
5. Click Test blocked URLs and enter a URL you want to block (like /admin/).
6. Google will tell you whether that URL is blocked or allowed by your robots.txt.
7. Test a few URLs to confirm your rules work as intended.

If a URL should be blocked but isn't, check your robots.txt syntax. Common issues:

• Missing trailing slashes (Disallow: /admin vs. Disallow: /admin/).
• Typos in paths.
• Incorrect use of wildcards.

When to Use Robots.txt vs. Other Methods

Robots.txt is powerful, but it's not the only tool. You should also understand when to use other methods:

• robots.txt: Block crawlers from accessing pages. Best for admin pages, duplicate content, and low-value pages.
• noindex meta tag: Tell Google not to index a page, but allow crawlers to access it. Best for pages you want crawlers to see but don't want in search results. Learn the decision tree for robots.txt vs. noindex.
• Canonical tags: Tell Google which version of a page is the "main" version. Best for duplicate content. Learn how to set up canonicals correctly.
• HTTP authentication: Truly hide pages from the public. Best for admin panels and internal tools.

For most Shopify stores, robots.txt is your primary tool. Use it to block crawlers from non-public pages and duplicate content. Use noindex for pages you want crawlers to see but not index (rare for e-commerce). Use canonical tags to handle duplicate product pages with different parameters.

Shopify-Specific Considerations

Shopify has some quirks worth knowing:

Shopify Automatically Generates robots.txt

If you don't create a custom robots.txt.liquid file, Shopify generates a default one automatically. This default is decent, but it's generic. By creating a custom file, you override the default and get full control.

The robots.txt.liquid File Is Liquid-Templated

Because it's a .liquid file, you can use Liquid logic if you need dynamic robots.txt rules. For example:

User-agent: *
{% if shop.password_enabled %}
Disallow: /
{% else %}
Disallow: /admin/
{% endif %}

This would block all crawlers if your store is password-protected, but only block /admin/ if it's public. Most stores don't need this, but it's possible if you need conditional rules.

Shopify Respects robots.txt for Image Crawlers

If you block a path in robots.txt, Shopify respects that for image crawlers too. So if you Disallow: /collections/clearance/, Google Image won't crawl product images in that collection either. This is usually what you want, but be aware of it.

Verifying Your Domain in Search Console

Before Google can use your robots.txt, you need to verify your domain in Google Search Console. Follow this step-by-step guide to verify your domain using DNS, HTML file, meta tag, or Analytics verification.

Once verified, you can access the robots.txt tester and see exactly how Google interprets your rules.

Faster Crawling with IndexNow

Robots.txt controls what crawlers can access, but IndexNow lets you ping search engines when you update content. If you frequently update product pages or add new collections, IndexNow tells Bing and Yandex about changes in minutes instead of weeks. It's a complementary tool to robots.txt.

Open Graph Tags for AI Search

While you're optimizing crawl rules, remember that AI search engines like ChatGPT and Perplexity also crawl your site. Set up Open Graph tags to improve how your content appears in AI search results. These tags don't affect traditional robots.txt, but they improve how AI systems understand and display your content.

One More Thing: Check Your Sitemap

Your robots.txt includes a Sitemap line. Make sure that sitemap actually exists and is valid. Generate a sitemap.xml for your Shopify store if you haven't already. Your sitemap should list all your product pages, collections, and blog posts (if you have one). Google uses the sitemap to discover pages faster, even if robots.txt allows crawling.

Summary: Key Takeaways

Your robots.txt is a critical file. Here's what you need to do:

1. Access your Shopify theme code editor through Admin → Sales channels → Themes → Edit code.
2. Create or edit robots.txt.liquid with custom rules for your store. Block /admin/, /cart, /checkout, duplicate product pages, and any staging environments.
3. Keep the default Disallow rules for query strings, JSON files, and tracking parameters. These prevent duplicate content.
4. Add a Sitemap line pointing to your sitemap.xml.
5. Test your robots.txt in Google Search Console's robots.txt tester.
6. Verify your domain in Google Search Console so Google can crawl and index your site.
7. Don't block your homepage, main collections, or product pages. Only block non-public and duplicate content.
8. Update robots.txt when you restructure your site, but don't change it constantly.
9. Monitor in Search Console monthly to ensure Google isn't reporting errors.

Robots.txt is one of three critical files for SEO. If you haven't already, audit your robots.txt, sitemap, and canonical tags to ensure they're all working together. These three files form the foundation of technical SEO. Get them right, and Google will crawl and index your store efficiently. Get them wrong, and you'll waste crawl budget and miss rankings.

You've now got a custom robots.txt live on your Shopify store. You didn't need an app. You didn't need to hire an agency. You edited it natively in under 10 minutes. That's how it should be.

Next, make sure your sitemap is submitted to Google, Bing, and Yandex. Then, check if your pages are actually indexed using the site: operator in Google Search.

If you're running a Shopify store and want a full SEO audit in under 60 seconds—including domain analysis, brand positioning, keyword roadmap, and 100 AI-generated blog posts—check out Seoable. It's built specifically for founders who ship.