By Ryan Arief in Homelab —

How I Keep My Homelab Subdomains Out of Search Results with Cloudflare

How I Keep My Homelab Subdomains Out of Search Results with Cloudflare

I've been putting more of my homelab services behind Cloudflare lately.

That's convenient, but it also made me think about something I hadn't really considered when I started self-hosting:

I don't want every service I publish to become part of the public search ecosystem.

My main website is one thing. That's content I actually want people to find.

The other services are different.

They're there because I need them.

I don't need a search engine to tell the world that I have a self-hosted file server, media server, URL shortener, or whatever else happens to be running on my machine this week.

So I decided to make a small change at the Cloudflare level.

First, a Small Terminology Correction

I originally approached this as:

"How do I prevent bots from crawling my subdomains?"

But that's not quite what this setup does.

There is an important difference between crawling and indexing.

Crawling is when a bot visits a URL.

Indexing is when a search engine stores that page in its search database and potentially shows it in search results.

Adding a noindex directive doesn't stop a bot from making the request.

It tells compliant search engines:

Don't put this page in the index.

That's actually what I wanted.

I'm not trying to build a magical invisibility cloak for my homelab.

I'm just trying to keep unnecessary service pages out of search results.

The Setup I Wanted

Conceptually, my setup looks like this:

example.com
    |
    +-- public content
    +-- okay to index


service-a.example.com
service-b.example.com
service-c.example.com
    |
    +-- homelab services
    +-- don't need to be indexed

I could configure this individually inside every application.

But that gets annoying pretty quickly.

Every new Docker container would mean remembering another application-specific SEO setting.

Since the traffic is already passing through Cloudflare, I decided to handle it there instead.

Using X-Robots-Tag

The header I'm using is:

X-Robots-Tag: noindex, nofollow, noarchive

The important part is noindex.

It tells search engines that the response shouldn't be added to their search index.

The other directives are just additional instructions:

noindex

Don't index the page.

nofollow

Don't use links on the page as crawling instructions.

noarchive

Don't create an archived search result version.

Again, none of these are security controls.

They're crawler/search-engine directives.

Why Cloudflare?

Because I don't want to repeat the configuration across every application.

My homelab is basically a collection of Docker containers.

One week I'm deploying one service.

The next week I'm removing it.

Then I'm probably deploying something else because apparently I enjoy creating problems for myself.

Having the rule at the Cloudflare layer means the applications don't need to know about it.

The traffic flow becomes:

Internet
   |
   v
Cloudflare
   |
   +---- main domain
   |
   +---- homelab services

And Cloudflare can modify the response before it goes back to the client.

Creating the Cloudflare Rule

In Cloudflare, I went to:

Rules → Transform Rules → Response Header Transform Rules

Then I created a rule that matches requests where the hostname isn't my main domain.

The expression is conceptually:

(http.host ne "example.com")

Obviously, example.com here is just an example.

I don't put my actual service hostnames into the article, because that would defeat the entire point of keeping unnecessary public references to them.

The rule then adds:

X-Robots-Tag: noindex, nofollow, noarchive

That's basically it.

What Happens Now?

The main domain is treated differently.

example.com

doesn't match the rule.

A service hostname such as:

something.example.com

does.

So the response from the service gets the additional header:

HTTP/1.1 200 OK
X-Robots-Tag: noindex, nofollow, noarchive

A search engine that respects the directive should then avoid indexing that content.

This Does NOT Hide My Services

This is probably the most important part.

Adding this rule doesn't make the service private.

Someone who already knows the URL can still visit it.

A scanner can still potentially discover it.

A bot can still make requests.

And a subdomain can still be discovered through other means.

For example, noindex doesn't protect against:

  • DNS enumeration
  • certificate transparency
  • URL guessing
  • external links
  • automated scanning
  • someone simply knowing the hostname

So I don't consider this a security layer.

It's more like cleaning up the public search footprint of my homelab.

If I Actually Wanted to Stop Bots

That's a different problem.

If I wanted to prevent arbitrary clients from reaching a service, I'd need actual access controls.

Depending on the application, that could mean:

  • Cloudflare Access
  • WAF rules
  • rate limiting
  • authentication
  • IP restrictions
  • VPN/Tailscale-only access
  • keeping the service completely private

That's where the real security controls belong.

The X-Robots-Tag is only about what search engines should do with the content.

Why I Bothered

This is probably one of those things that doesn't matter to most people.

But once you start exposing multiple homelab services to the internet, I prefer having a clear separation between:

things I want people to discover

and

things I merely need to access remotely.

My website belongs in the first category.

My random Docker containers belong in the second.

I don't need a search engine to become the directory of my homelab.

So this is a small Cloudflare rule that takes care of one small part of that problem.

Not security.

Not anonymity.

Not a way to disappear from the internet.

Just one less piece of my infrastructure that needs to show up in search results.