Skip to content
Every tutorial here is 100% free — and always will be
LSLovableStudent

How do I create robots.txt?

Last checked 31 Aug 2026

Short answer

Ask Lovable to add a robots.txt file in the public folder that allows crawling of public pages, blocks private/admin routes, and points to your sitemap.xml.

What robots.txt does

robots.txt tells well-behaved crawlers which parts of your site they may or may not crawl, and where to find your sitemap. It's a plain text file, not enforcement — it won't stop a determined bot, but Google and other major crawlers respect it.

Asking Lovable to create it

Add a robots.txt file at the root allowing all crawlers to index public pages, disallowing /admin and /account, and referencing my sitemap.xml.

A minimal example:

User-agent: *
Allow: /
Disallow: /admin
Disallow: /account

Sitemap: https://yourdomain.com/sitemap.xml

Where it lives

robots.txt must be served at the root of your domain — https://yourdomain.com/robots.txt — usually by placing the file directly in public/.

What to block (and what not to)

  • Block: admin panels, account/settings pages, checkout success pages with sensitive tokens, staging/duplicate routes.
  • Don't block: your CSS/JS assets — blocking these can stop Google rendering your pages properly, which hurts rather than helps.
  • Don't rely on it for security — anything genuinely sensitive needs actual auth/RLS, not just a Disallow line. robots.txt is public and readable by anyone.

What usually goes wrong

  • Accidentally blocking the whole site with Disallow: / left over from a staging setup — always double check this before going live.
  • Forgetting to add the Sitemap: line, missing an easy win for crawl discovery.
  • Blocking asset folders (/assets, /_next, etc.), which can prevent Google from rendering pages correctly.

Sources

Was this answer helpful?

Related questions

Website made, maintained and hosted by PromptSites.com

PromptSites logo