Published: by Lucas Rolff
Gatekeeper Cloudflare: Breaking DNS standard
First a little background
We're a hosting provider based in The Netherlands. We primarily focus on shared hosting solutions powered by cPanel, as well as our own in-house control panel.
Whenever customers provision domains on their hosting accounts under our Grid Hosting product, which is what uses our in-house panel, we will provision a DNS zone containing an _acme-challenge
subdomain, Which in reality points to a CNAME managed by us (e.g. acme-example.nl
). This acme-challenge domain uses acme-dns to manage the whole lifecycle of dealing with ACME DNS challenges.
We chose DNS-01 challenges for ACME because they allow us to issue wildcard certificates for our customers' domains. This means we only have one certificate to manage, synchronize, and renew for each primary domain and all its subdomains.
This approach simplifies certificate issuance/renewal using any ACME client. For internal DNS, the required _acme-challenge
CNAME configuration is automatic. For external DNS providers, it's a matter of verifying the correct CNAME setup.
Switching to HTTP-01 challenges would introduce significant complexity. We'd have to constantly track when customers add or remove subdomains to correctly start and stop certificate issuance and renewal processes, resulting in more overhead and more things that can go wrong.
So how does Cloudflare affect this?
Roughly 20% of the internet traffic passes through Cloudflare, whether they use their proxy service, or simply relying on Cloudflare DNS. It's a lot of the internet. So obviously for us as a hosting provider, we also see a significant portion of customers using Cloudflare, either for DNS or full on proxy-mode. That's fine, it's the customer's choice.
Over the past years, we've never really had issues in this regard, certificate issuance and renewals have worked fine, even when people used Cloudflare DNS or proxied their site(s) through Cloudflare.
For Cloudflare's proxying to work, they also have to issue SSL certificates themselves, also as wildcard certificates, meaning they too rely on DNS-01 challenges when issuing or renewing certificates.
The way Cloudflare does this is by publishing their own TXT records for the _acme-challenge
subdomain. This lets them pass the validations that are being done, and a certificate is issued.
If a customer configures the _acme-challenge
subdomain as a CNAME pointing to their special subdomain under acme-example.nl
then Cloudflare should return this CNAME, even when doing a query for TXT
records, why? Because that's what the RFC says!
Breaking the RFCs
The big, beautiful internet is built on a lot of rules about how to do things correctly. These rules ensure that people who make software can build it so that systems can communicate properly. This results in a list of expectations on how software should function in a given scenario. These rules are also known as RFCs and they usually undergo a significant amount of discussion and work before they're published.
As with anything, there's a set of RFCs defined for DNS, and how DNS resolvers, recursors and authoritative nameservers are supposed to work.
One of the expectations here, is that if you have a CNAME configured on a given route-record (RR), no other DNS type can exist. That means, if _acme-challenge.example.com
is a CNAME towards example.com.acme-example.nl
- then you cannot have a TXT, MX, A, AAAA, SRV or any other record type existing on this _acme-challenge
subdomain. A CNAME works a bit like a redirect, telling resolvers to "go look here for what you need".
The specific clause can be found in RFC1912 section 2.4:
2.4 CNAME records
A CNAME record is not allowed to coexist with any other data. In
other words, if suzy.podunk.xx is an alias for sue.podunk.xx, you
can't also have an MX record for suzy.podunk.edu, or an A record**, or
even a TXT record.**
However, it seems that with a recent change at Cloudflare, they continue to publish these TXT records, even if they're not actively issuing a certificate:
# dig txt _acme-challenge.example.nl +short @alexa.ns.cloudflare.com
"TVYEySVHJN-7efzGmLjGIqaNKWXk1fPVSoFTmuKoUpw"
"CnQaAQybkd6bu-Ci75wRuPkn6dmU1chGR2FwswELHZM"
So when a customer opts for using Cloudflare, and configures the CNAME as instructed. Cloudflare opts to return their TXT records when you query for TXT
but return the configured CNAME when you query for CNAME
. This is prohibited by the RFC, and it breaks quite a few resolvers.
This also means, when we try to issue (or renew) a wildcard certificate using Let's Encrypt or Google Trust Services (GTS), then validation fails because what Let's Encrypt and Google Trust Services see, are simply a TXT record. Because Cloudflare decides to do continued hijacking of TXT records, the "redirect" in the form of a CNAME is never seen by these certificate issuers, and thus we cannot issue certificates.
Just disable "Universal SSL"
Cloudflare support suggests to simply disable "Universal SSL" on Cloudflare accounts. While this is a possible fix, it effectively prevents users from using Cloudflare proxy mode on any part of their domains. What if they want to run their main website through Cloudflare, but nothing else? Or what if they want to run their support subdomain through Cloudflare but not the main site? Bad luck, you can't do that without then starting to upload your own custom certificates.
it's also worth noting that this used to work perfectly fine in the past. We've been able to issue wildcard certificates for years, even for some of our own domains that uses Cloudflare DNS.
But somehow now it breaks.
But what about non-proxied domains?
We'd expect that customers who use Cloudflare DNS, but don't proxy their sites through Cloudflare would work perfectly fine. After all, they have no reason to really permanently hijack TXT records.
But somehow they decide to do that anyway, but only sometimes! We have some domains that are outright broken, and publish their phantom DNS TXT records permanently, hijacking issuance, while we have other domains that seemingly work fine... sometimes.
## Our domain uses Cloudflare nameservers
# dig ns example.nl +short
alexa.ns.cloudflare.com.
dylan.ns.cloudflare.com.
## Querying for TXT, returns 2 records (Cloudflare managed)
# dig txt _acme-challenge.example.nl +short @alexa.ns.cloudflare.com
"TVYEySVHJN-7efzGmLjGIqaNKWXk1fPVSoFTmuKoUpw"
"CnQaAQybkd6bu-Ci75wRuPkn6dmU1chGR2FwswELHZM"
## Querying for CNAME reveals the actual CNAME we configured within the Cloudflare dashboard
# dig cname _acme-challenge.example.nl +short @alexa.ns.cloudflare.com
49505a49-6c2d-42fe-b2ec-d5c6976176ee.auth.acme-example.nl.
## We're not proxying the site through Cloudflare
# dig a example.nl +short @alexa.ns.cloudflare.com
51.222.105.xx
## The expected records of the _acme-challenge CNAME
# dig txt 49505a49-6c2d-42fe-b2ec-d5c6976176ee.auth.acme-example.nl +short
"-pnf2FO4Q110vVeXkxLrURl5B9a_LsNelp0wI1bdRuY"
"lVwFcy3dfdI4PcWV-3KMjY9GtIOchjZVoml0yd2Ol9I"
The CNAME only returns two TXT records (with different values). And the IP returned when querying for A
on the root domain, is a non-Cloudflare IP. Cloudflare should not return their own TXT records when a CNAME is configured, especially not when the site isn't proxied through Cloudflare.
What should happen is this:
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_acme-challenge.example.nl. IN TXT
;; ANSWER SECTION:
_acme-challenge.example.nl. 298 IN CNAME 49505a49-6c2d-42fe-b2ec-d5c6976176ee.auth.acme-example.nl.
49505a49-6c2d-42fe-b2ec-d5c6976176ee.auth.acme-example.nl. 1 IN TXT "-pnf2FO4Q110vVeXkxLrURl5B9a_LsNelp0wI1bdRuY"
49505a49-6c2d-42fe-b2ec-d5c6976176ee.auth.acme-example.nl. 1 IN TXT "lVwFcy3dfdI4PcWV-3KMjY9GtIOchjZVoml0yd2Ol9I"
And in the Cloudflare DNS dashboard, we see the following:

But what can we really do about it?
We saw this break back in June this year. We managed to get hold of someone internally at Cloudflare, and shortly after we reported the issue back then, it got resolved a few days later, sadly without any communication what was done/changed, or even giving the acknowledgement that they had done something.
But hey, it was fixed, we moved on with our lives.
Fast forward to October, this has broken again. We initially reached out to the person who we spoke to back in June, sadly without any communication either.
We went as far as emailing Dane Knecht, the CTO of Cloudflare. Dane's response? He forwarded it to some people internally, and eventually, we're now back to the original contact. But without response.
We're incredibly thankful for Dane's response, he could have ignored us, but he didn't, so we have to give him the credit where credit is due.
Now, the issue is, we seemingly won't get the issue resolved in time, or maybe ever. Only time will tell.
The only thing we can really do is to start working on an implementation that uses HTTP challenges, and implement the additional complexity required to deal with this, because Cloudflare, The company that powers roughly 1/5 of the internet, can't follow an RFC.
We'll equally have to communicate with all our customers who use Cloudflare, and tell them what to change within the Cloudflare interface to ensure that things will continue to work even when certificates expire, or in the case we can't even issue a certificate in the first place.
It is possible that this issue stems from a bug in Cloudflare's certificate issuance or renewal process, where the temporary TXT records they inject aren't properly removed after completion. We have also spoken with several other IT operations professionals who've encountered the same problem. Both in cases with Cloudflare proxying enabled or disabled.
Conclusion
We'd like to see more focus on RFC compliance, especially from companies that control such a large part of the internet. As a small company, we're forced to implement awkward workarounds because Cloudflare introduces changes that break standards.
Cloudflare's DNS UI also allow configuring TXT and CNAME for the same DNS record, which the RFC1912 section 2.4 prohibits. It does however block adding A or AAAA records if a CNAME exists. Which is why we believe the choice of breaking the RFC is intentional.
Overall breaking the core functionality of DNS this way, means that customers who use Cloudflare DNS are somewhat being forced to only run their things through Cloudflare, alternatively the only thing you can really do is to issue certificates through HTTP-01 challenges. As a result, Cloudflare hinders the possibility to issue wildcard SSL certificates for infrastructure running outside Cloudflare under the same domain.
While people can (for now) simply go buy a wildcard certificate. Certificates will eventually be reduced to a lifetime of 47 days, meaning automation through things such as the ACME protocol are crucial for SSL certificates.
We clearly see the regression started happening somewhere in the past 30 days:

We log every unsuccessful renewal attempt in Sentry, mainly to get a warning when it happens, and every single one of the above entries are due to Cloudflare TXT hijacking, for proxied and non-proxied domains.
About the Author
Lucas Rolff
Senior bug finderLucas is the founder of PerfGrid, with over 15 years of experience in web hosting, performance optimization, and server infrastructure. He also loves to find bugs in software and systems.