DNS Cluster shows Bind not PowerDNS

[thowden]

Hi All

I have just noticed an issue with the "Remote Server Type" in my DNS Cluster which is showing as "Bind" on all my servers even though it was originally showing "PowerDNS" and the remote servers are still using PowerDNS, and I have checked that they are.

This is only recent as I was updating my cluster API tokens in December and PowerDNS was reported. I only spotted the change this morning as I was configuring a new hosting server and adding it to use the DNS cluster. I then checked all the existing servers and they are all showing "Bind". I expect it is related to a recent WHM/CPanel update.

Config: 4 x DNS Only servers all on the same WHM/CPanel CentOS versions. CentOS v8.5.2111 DNSONLY xen hvm v100.0.9 - these are all Stand-Alone accepting Write-Only updates from the hosting servers.

Hosting Servers: 5 x Various CentOS and AlmaLinux as the base install, CloudLinux / WHM / Cpanel versions vary over different ages.


Screenshot example from one of the hosting servers, but representative of all the hosting servers. Should be showing and were showing PowerDNS previously.

No error reported, no apparent impact on the DNS service, just disconcerting at 0600 Sunday morning when its not expected.

Bug? Setting change? Ideas please.

 

[cPRex]

Hey there! I know we had an issue with Ubuntu detecting this properly, so maybe the other OS is having an issue now. Let me look into this and I'll get back to you.

 

[cPRex]

It's interesting to me that the "Status" column is also empty. Could you run this command on one of the DNSOnly systems?

whmapi1 installed_versions | egrep 'bind|powerdns'

If you could let me know the output of that we can go from there.

 

[thowden]

Hi

[root@ ~]# whmapi1 installed_versions | egrep 'bind|powerdns'
bind: 9.11.4-16.P2
powerdns: 4.4.1-2.cp11100

 

[cPanelPeter]

Interesting. It shows both DNS servers installed at the same time. This shouldn't be possible. Please consider opening a ticket so that we can take a further look at your DNS Only server.

 

[quietFinn]

Interesting. It shows both DNS servers installed at the same time. This shouldn't be possible. ...

In a DNSONLY server:

[root@ns1 ~]# whmapi1 installed_versions | egrep 'bind|powerdns'
  bind: 9.11.4-26.P2
  powerdns: 4.4.1-2.cp11100

Remote Server Type shows powerdns as expected.

I checked that in several servers and in all both bind and powerdns are installed.

 

[cPanelPeter]

Very odd. Can you please run this on both a WHM and a DNS Only server?

grep local_nameserver_type /var/cpanel/cpanel.config

 

[quietFinn]

Very odd. Can you please run this on both a WHM and a DNS Only server?

grep local_nameserver_type /var/cpanel/cpanel.config

WHM:
[root@finn2 ~]# grep local_nameserver_type /var/cpanel/cpanel.config
local_nameserver_type=powerdns

DNSONLY:
[root@ns1 ~]# grep local_nameserver_type /var/cpanel/cpanel.config
local_nameserver_type=powerdns

 

[cPanelPeter]

Please open a ticket. I'm very interested in reviewing this. There was a past case we had for version 92 that had an issue similar to this. We might need to file a new case if this issue has returned.

 

[quietFinn]

I am not the OP of this thread, haven't had any issues with DNS clustering, but I got interested and noticed that both bind and powerdns are installed.

 

[cPanelPeter]

OK, thank you. It seems we may have changed the way we detect which DNS Server is running/installed in the UI. It used to be with the installed_versions whmapi1 call but that may no longer be the case. I just tested it on my own server and it also shows both installed (which still shouldn't be possible), but if your UI is correct and mine is correct, then that means it's no longer relevant.

If the original poster @thowden could please open a ticket, it would be a great help to determine if we have a new or different defect that needs to be addressed. It might just be a misconfiguration on the server too, so I really am interested in finding out what the issue is.

 

[thowden]

Hi

Okay so I have tried to register a ticket, but the stupid support system crashes after I give the server access details.



So as much as I would like to complete that process, it is just wasting my time.

 

[thowden]

Hi

I was trying to use the option to provide more than one server. With one server only it works. Ticket # 94417070

 

[cPanelPeter]

Hello,

Thank you for the update and ticket number. I see my colleague Jeff responded to you and provided instructions on how to add the additional servers. The message you get (above) happens if you have more than 1 tab or browser window open and are logged on more than once.

The ns1 server Jeff had access to, shows that DNS Clustering is enabled, but there are currently no servers defined. (no reverse trust). That's likely part of the issue.
I'm monitoring the ticket and will assist Jeff or any of our analysts as needed.

 

[thowden]

Hi

As at January 5th and for the past 12 months or more, the 4 DNS servers have operated ok, showing PowerDNS.

The DNS Only servers were newly deployed in April/May 2021 replacing my older servers. Specifically using PowerDNS and not Bind. So I know that PowerDNS was configured on these servers and Bind was not (at least not visibly noted as installed and not expected).

Further, I know Jan 5th as it is in my documentation when I updated some of the API keys connecting to the 4 DNS servers. i.e. The cluster information displayed on the hosting servers showed PowerDNS as the Remote DNS for all. If Bind had been displayed then I would have started this thread on January 5th.

As at last Sunday the hosting servers show Bind as the remote DNS. That status change in the webhosts DNS cluster page has changed between Jan 5th and Feb 13th.

If both Bind and PowerDNS are installed on the DNS Only hosts, then that has occurred since Jan 5th and prior to today. The only change that I am aware of was the update to v100.0.9 applied at some point in time recently, but I do not track minor updates (I am starting to do so now).

In relation to reverse trust, I have never had that. The goal was to have the DNS servers do nothing other than receive the zones from the web hosting servers and serve zones to the interwebs. Each DNS Only server acts independently with Write Only configured from the webhost servers to the DNS Only servers. There is no synchronisation or relationship between the DNS Only servers. This method ensures that all zones are processed to all DNS servers and that any server can fail without impacting any other server (at least that is the expectation) and all zones will still be available.

If there is a requirement for Reverse Trust, it would appear to be redundant, as the configuration has worked without it for 5+ years.

 

[cPRex]

Thanks for the additional details. It's likely best to continue with the ticket since it seems like we'll need to examine the specific configurations on at least one WHM and DNSOnly system in order to come up with more information about this behavior.

 

[thowden]

Hi cPRex

Yes, 1 x host and 2 x DNS Only servers added to the ticket. The information added to this thread was as much a reply to cPanelPeter and for other interested viewers.

 

[thowden]

Hi cPanelPeter

(no reverse trust). That's likely part of the issue.

I had to go and check this. Reverse Trust option is not selectable (greyed-out) when Write-Only is selected as the option.

 

[thowden]

Hi All

The resolution to this was provided via a support ticket. I am documenting the solution here for any follow up.

The issue is that the CentOS 8 repo's for yum are failing. The /var/log/dnf.log showed

CentOS Linux 8 - AppStream                       20 kB/s | 246  B     00:00   
Error: Failed to download metadata for repo 'appstream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
(XID s5s363) “/usr/bin/yum” reported error code “1” when it ended:

To avoid this error I have disabled 2 repo's. The second was made apparent after disabling the appstream and presented an error on baseos repo.

#yum-config-manager --disable appstream
#yum-config-manager --disable baseos

I then ran

#yum update

and restarted the server.

The command previously suggested is still showing both name servers, but that might be a false positive:

[root@ns1 ~]# whmapi1 installed_versions | egrep 'bind|powerdns'
bind: 9.11.26-6
  powerdns: 4.4.1-2.cp11100

But checking a webhosting server for DNS Cluster config shows PowerDNS as desired which confirms the fix.

That said and done, it is obviously an issue with the underlying CentOS 8 and I will now bring forward my upgrade plan and provision new AlmaLinux base for DNS Only which will hopefully avoid further issues.

Thanks to all who assisted.