> ## Content Index > Fetch the complete content index at: https://kk-lab.uk/llms.txt > Use this file to discover other available public pages before exploring further. # How I Keep My Homelab Subdomains Out of Search Results with Cloudflare - URL: https://kk-lab.uk/how-i-keep-my-homelab-subdomains-out-of-search-results-with-cloudflare/ - Published: 2026-09-11T17:40:06.000Z - Updated: 2026-09-11T17:40:06.000Z - Author: Ryan Arief - Tags: Homelab I've been putting more of my homelab services behind Cloudflare lately. That's convenient, but it also made me think about something I hadn't really considered when I started self-hosting: **I don't want every service I publish to become part of the public search ecosystem.** My main website is one thing. That's content I actually want people to find. The other services are different. They're there because I need them. I don't need a search engine to tell the world that I have a self-hosted file server, media server, URL shortener, or whatever else happens to be running on my machine this week. So I decided to make a small change at the Cloudflare level. ## First, a Small Terminology Correction I originally approached this as: > "How do I prevent bots from crawling my subdomains?" But that's not quite what this setup does. There is an important difference between **crawling** and **indexing**. Crawling is when a bot visits a URL. Indexing is when a search engine stores that page in its search database and potentially shows it in search results. Adding a `noindex` directive doesn't stop a bot from making the request. It tells compliant search engines: > Don't put this page in the index. That's actually what I wanted. I'm not trying to build a magical invisibility cloak for my homelab. I'm just trying to keep unnecessary service pages out of search results. ## The Setup I Wanted Conceptually, my setup looks like this: ```text example.com | +-- public content +-- okay to index service-a.example.com service-b.example.com service-c.example.com | +-- homelab services +-- don't need to be indexed ``` I could configure this individually inside every application. But that gets annoying pretty quickly. Every new Docker container would mean remembering another application-specific SEO setting. Since the traffic is already passing through Cloudflare, I decided to handle it there instead. ## Using `X-Robots-Tag` The header I'm using is: ```text X-Robots-Tag: noindex, nofollow, noarchive ``` The important part is `noindex`. It tells search engines that the response shouldn't be added to their search index. The other directives are just additional instructions: ```text noindex ``` Don't index the page. ```text nofollow ``` Don't use links on the page as crawling instructions. ```text noarchive ``` Don't create an archived search result version. Again, none of these are security controls. They're crawler/search-engine directives. ## Why Cloudflare? Because I don't want to repeat the configuration across every application. My homelab is basically a collection of Docker containers. One week I'm deploying one service. The next week I'm removing it. Then I'm probably deploying something else because apparently I enjoy creating problems for myself. Having the rule at the Cloudflare layer means the applications don't need to know about it. The traffic flow becomes: ```text Internet | v Cloudflare | +---- main domain | +---- homelab services ``` And Cloudflare can modify the response before it goes back to the client. ## Creating the Cloudflare Rule In Cloudflare, I went to: **Rules → Transform Rules → Response Header Transform Rules** Then I created a rule that matches requests where the hostname isn't my main domain. The expression is conceptually: ```text (http.host ne "example.com") ``` Obviously, `example.com` here is just an example. I don't put my actual service hostnames into the article, because that would defeat the entire point of keeping unnecessary public references to them. The rule then adds: ```text X-Robots-Tag: noindex, nofollow, noarchive ``` That's basically it. ## What Happens Now? The main domain is treated differently. ```text example.com ``` doesn't match the rule. A service hostname such as: ```text something.example.com ``` does. So the response from the service gets the additional header: ```text HTTP/1.1 200 OK X-Robots-Tag: noindex, nofollow, noarchive ``` A search engine that respects the directive should then avoid indexing that content. ## This Does NOT Hide My Services This is probably the most important part. Adding this rule doesn't make the service private. Someone who already knows the URL can still visit it. A scanner can still potentially discover it. A bot can still make requests. And a subdomain can still be discovered through other means. For example, `noindex` doesn't protect against: - DNS enumeration - certificate transparency - URL guessing - external links - automated scanning - someone simply knowing the hostname So I don't consider this a security layer. It's more like cleaning up the public search footprint of my homelab. ## If I Actually Wanted to Stop Bots That's a different problem. If I wanted to prevent arbitrary clients from reaching a service, I'd need actual access controls. Depending on the application, that could mean: - Cloudflare Access - WAF rules - rate limiting - authentication - IP restrictions - VPN/Tailscale-only access - keeping the service completely private That's where the real security controls belong. The `X-Robots-Tag` is only about what search engines should do with the content. ## Why I Bothered This is probably one of those things that doesn't matter to most people. But once you start exposing multiple homelab services to the internet, I prefer having a clear separation between: **things I want people to discover** and **things I merely need to access remotely.** My website belongs in the first category. My random Docker containers belong in the second. I don't need a search engine to become the directory of my homelab. So this is a small Cloudflare rule that takes care of one small part of that problem. Not security. Not anonymity. Not a way to disappear from the internet. Just one less piece of my infrastructure that needs to show up in search results.