Hi All
I have battled an issue over the last week, although it was existing for longer and I cannot pinpoint when it started.
Background: WHM servers all feed to a cluster of 3 physical DNSOnly servers. Each DNSOnly server is stand-alone, each WHM server feeds to each of the stand-alone servers. There is no cross-server pollination, all independent. All on latest releases. I had run updates mid-October. WHM server CL 7.8 v90.0.16 with the DNSOnly servers on CentOS 7.8 same version 90.0.16. All configured to use the new(ish) WHM PowerDNS service.
Situation: A domain that I use on a daily basis was getting customer reports of failing emails. I figured it was client-side, or spammy treatment, and let it go. Silly boy!
When I did finally decide I needed to check, I used the great free tools at dnsstuff.com and did a DNS health check. It showed that the 4 name servers were listed, but that there was no SOA, no MX, and no SPF records. Huh?
Checking on the actual CPanel / WHM server using Edit DNS (the old format) the zone looked fine with all records in correct configuration. Run Sync DNS records expecting that it will propagate and resolve itself.
Roll forward to last Friday and yet another customer contacts me and as I knew this guy really well, I got him to forward his bounce message to an alternate email for me.
Bounce message states that the domain MX server cannot be found. Checking with dnsstuff.com again, the same gaps existed in the DNS Health check. No SOA, no MX, no SPF.
Testing:
First up, I headed to our registrar to check the TLD Glue records and confirmed that there was no issue there.
Next, I again reviewed the domain DNS settings via the DNS Edit (Old) and saved and reviewed again just in case. Looked fine. Reviewed the TLD for the SOA name servers (host domain) and checked the DNS configuration there as well. All looked ok.
I then started looking at the DNSOnly servers, from outside using another linux box with dig and confirmed that all the name servers had the same opinion. The zone did not have any data to provide out for SOA, MX, or SPF.
On one of the servers I used the PowerDNS check utility, ignoring the warning which I discuss in PowerDNS Pased TXT Record Error, the zone reported no errors with 19 records, consistent with the zone file that was being sent from the WHM server to the cluster.
[root]# pdnsutil check-zone clientdomain.com
Nov 01 19:28:27 [bindbackend] Done parsing domains, 3 rejected, 307 new, 0 removed
[Warning] Parsed and original record content are not equal: default._domainkey.clientdomain.com IN TXT '"v=DKIM1; k=rsa; p=MIIBIjANBgkqhk <snipped.....> DAQAB\;"')
Checked 19 records of 'clientdomain.com', 0 errors, 1 warnings.
Throughout the course of the weekend, a simple (and I hope correctly formatted) dig command showed:
[root]# dig @localhost clientdomain.com MX
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-16.P2.el7_8.6 <<>> @localhost clientdomain.com MX
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 2220
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1680
;; QUESTION SECTION:
;clientdomain.com. IN MX
;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Nov 01 19:29:56 AEDT 2020
;; MSG SIZE rcvd: 43
Perplexed, I drilled into the warning message to see if that might impact, but it is simply a warning.
I then reviewed the SSL certificates on the servers, just in case some obscure issue with our 'bought'n'paid' certs had been replaced with something self-signed.
etc, etc.
I then noted that some of the TXT records were double-quoted, while others were not, and in amongst editing them to add double-quotes (just in case it made a difference) I noted the 'Please try the new DNS Zone Manager' prompt and I wondered what-if?
Resolution 90%:
Edit the domain using the new DNS Zone Manager. Sync the zone. Test at the Name Server and Kapow
[root]# dig @localhost clientdomain.com MX
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-16.P2.el7_8.6 <<>> @localhost clientdomain.com MX
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25629
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1680
;; QUESTION SECTION:
;clientdomain.com. IN MX
;; ANSWER SECTION:
clientdomain.com. 14400 IN MX 0 mail.clientdomain.com.
;; ADDITIONAL SECTION:
mail.clientdomain.com. 14400 IN A xxx.xxx.xxx.xxx
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Nov 01 19:49:28 AEDT 2020
;; MSG SIZE rcvd: 80
Last 10% to Resolve:
I am still not getting an SPF record being reported at dnsstuff.com, so I checked with kitterman SPF test with this result
The TXT records found for your domain are:
"v=spf1 +mx +a -all"
No valid SPF record found.
I had added double-quotes to the record. Removing those in the DNS Zone Manager and resyncing the records gave me a clean check of the SPF at both.
But... reverting to the old Edit DNS Zone I see that the SPF record is still wrapped in double-quotes, and one of the DKIM records as well.
Summary:
There is some issue with the 'upgrade' path for DNS in recent CPanel / WHM patches that is potentially creating orphan-ish records that appear ok in the (old) Edit DNS Zone but will be ok if processed (i.e. edit and save one record) using the new DNS Zone Manager.
I suspect that it relates to the double-quote issue, see PowerDNS TXT Parsed and original record content are not equal.
I note for anyone reviewing this that I am truly old-school as a CPanel / WHM user / admin for around 18 or so years ( has it been that long!?!) and I am probably still using techniques that have long faded into disrepair, but hey, change is hard.
I have battled an issue over the last week, although it was existing for longer and I cannot pinpoint when it started.
Background: WHM servers all feed to a cluster of 3 physical DNSOnly servers. Each DNSOnly server is stand-alone, each WHM server feeds to each of the stand-alone servers. There is no cross-server pollination, all independent. All on latest releases. I had run updates mid-October. WHM server CL 7.8 v90.0.16 with the DNSOnly servers on CentOS 7.8 same version 90.0.16. All configured to use the new(ish) WHM PowerDNS service.
Situation: A domain that I use on a daily basis was getting customer reports of failing emails. I figured it was client-side, or spammy treatment, and let it go. Silly boy!
When I did finally decide I needed to check, I used the great free tools at dnsstuff.com and did a DNS health check. It showed that the 4 name servers were listed, but that there was no SOA, no MX, and no SPF records. Huh?
Checking on the actual CPanel / WHM server using Edit DNS (the old format) the zone looked fine with all records in correct configuration. Run Sync DNS records expecting that it will propagate and resolve itself.
Roll forward to last Friday and yet another customer contacts me and as I knew this guy really well, I got him to forward his bounce message to an alternate email for me.
Bounce message states that the domain MX server cannot be found. Checking with dnsstuff.com again, the same gaps existed in the DNS Health check. No SOA, no MX, no SPF.
Testing:
First up, I headed to our registrar to check the TLD Glue records and confirmed that there was no issue there.
Next, I again reviewed the domain DNS settings via the DNS Edit (Old) and saved and reviewed again just in case. Looked fine. Reviewed the TLD for the SOA name servers (host domain) and checked the DNS configuration there as well. All looked ok.
I then started looking at the DNSOnly servers, from outside using another linux box with dig and confirmed that all the name servers had the same opinion. The zone did not have any data to provide out for SOA, MX, or SPF.
On one of the servers I used the PowerDNS check utility, ignoring the warning which I discuss in PowerDNS Pased TXT Record Error, the zone reported no errors with 19 records, consistent with the zone file that was being sent from the WHM server to the cluster.
[root]# pdnsutil check-zone clientdomain.com
Nov 01 19:28:27 [bindbackend] Done parsing domains, 3 rejected, 307 new, 0 removed
[Warning] Parsed and original record content are not equal: default._domainkey.clientdomain.com IN TXT '"v=DKIM1; k=rsa; p=MIIBIjANBgkqhk <snipped.....> DAQAB\;"')
Checked 19 records of 'clientdomain.com', 0 errors, 1 warnings.
Throughout the course of the weekend, a simple (and I hope correctly formatted) dig command showed:
[root]# dig @localhost clientdomain.com MX
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-16.P2.el7_8.6 <<>> @localhost clientdomain.com MX
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 2220
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1680
;; QUESTION SECTION:
;clientdomain.com. IN MX
;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Nov 01 19:29:56 AEDT 2020
;; MSG SIZE rcvd: 43
Perplexed, I drilled into the warning message to see if that might impact, but it is simply a warning.
I then reviewed the SSL certificates on the servers, just in case some obscure issue with our 'bought'n'paid' certs had been replaced with something self-signed.
etc, etc.
I then noted that some of the TXT records were double-quoted, while others were not, and in amongst editing them to add double-quotes (just in case it made a difference) I noted the 'Please try the new DNS Zone Manager' prompt and I wondered what-if?
Resolution 90%:
Edit the domain using the new DNS Zone Manager. Sync the zone. Test at the Name Server and Kapow
[root]# dig @localhost clientdomain.com MX
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-16.P2.el7_8.6 <<>> @localhost clientdomain.com MX
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25629
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1680
;; QUESTION SECTION:
;clientdomain.com. IN MX
;; ANSWER SECTION:
clientdomain.com. 14400 IN MX 0 mail.clientdomain.com.
;; ADDITIONAL SECTION:
mail.clientdomain.com. 14400 IN A xxx.xxx.xxx.xxx
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Nov 01 19:49:28 AEDT 2020
;; MSG SIZE rcvd: 80
Last 10% to Resolve:
I am still not getting an SPF record being reported at dnsstuff.com, so I checked with kitterman SPF test with this result
The TXT records found for your domain are:
"v=spf1 +mx +a -all"
No valid SPF record found.
I had added double-quotes to the record. Removing those in the DNS Zone Manager and resyncing the records gave me a clean check of the SPF at both.
But... reverting to the old Edit DNS Zone I see that the SPF record is still wrapped in double-quotes, and one of the DKIM records as well.
Summary:
There is some issue with the 'upgrade' path for DNS in recent CPanel / WHM patches that is potentially creating orphan-ish records that appear ok in the (old) Edit DNS Zone but will be ok if processed (i.e. edit and save one record) using the new DNS Zone Manager.
I suspect that it relates to the double-quote issue, see PowerDNS TXT Parsed and original record content are not equal.
I note for anyone reviewing this that I am truly old-school as a CPanel / WHM user / admin for around 18 or so years ( has it been that long!?!) and I am probably still using techniques that have long faded into disrepair, but hey, change is hard.
Last edited:
