Published: February 25, 2025 by PerfGrid
Pulse: Week 9, 2025
Welcome to the second edition of our Pulse series, where we share insights into our infrastructure changes, service improvements, and other behind-the-scenes activities at PerfGrid.
Replacement of ns3.dnscluster.dk
During last week, on the 19th of February, we faced an outage of the nameserver ns3.dnscluster.dk during the night. The nameserver is one of our redundant 4 nameservers powering Grid Hosting. We took it out of production during the night, and got back to it in the morning.
Interestingly, some of the software we use called vector.dev, couldn't start due to 4/ILL - ILL is a signal that's triggered if you try to execute invalid CPU instructions.
The machine in this case, is a virtual server that was recently migrated between hypervisors at the provider we use, so likely something didn't go as planned with the migration.
While we tried to fix the issue, the system in fact crashed and became unrecoverable. We've since set up a new DNS server in Amsterdam on our own Proxmox cluster that we have running anyway.
We obviously still spread nameservers across 4 different networks, just to be sure.
Luckily replacing our nameservers for Grid Hosting solutions is super easy, since it's all based on PowerDNS, LMDB and Lightning Stream, so it's simply a matter of deploying the OS, execute an ansible job, and we're up and running. The DNS data automatically gets synchronized without us having to do literally anything. All we then have to do afterwards, is obviously to perform some tests to ensure DNS queries respond as expected, DNSSEC works (usually does considering that's also relying on LMDB), and then change a few DNS entries to point the nameserver hostname to the new IPs.
Postmark replacement with Amazon SES
We've effectively replaced Postmark with Amazon SES for our email sending on billing.perfgrid.com (invoices etc). Prior to this change, we'd always send via Postmark with Amazon SES as a failover. We've phased out Postmark all together.
We initially used Postmark due to exceptional delivery, and the team behind Postmark was really amazing. They since got acquired, and ever since, prices has increased, and we've seen slower and slower delivery over time.
We use Amazon SES for a bunch of other things already, so it was a natural switch, reputation is still great, delivery has been rock solid anyway. Pricing is much better, not that it really matters for the smaller volume we have for the particular use-case, but it does mean we have one less supplier, and thus less accounting to deal with.
Domain pricing changes
Various TLDs have been on the rise over the past years, once in a while we go over the pricing changes, and adjust our pricing accordingly for new orders only. It has been quite some time since we did this last, so we went over it last week, in fact some TLDs we had a loss of about 20% for every registration. Luckily, it's not TLDs we really sell much of these days - but it has been adjusted for the future.
Existing customers/domains obviously won't be affected by these changes. Your pricing remains the same.
Ticket change improvements
As you may know, we use Zendesk for our ticketing system, with an integration into our billing system WHMCS. As ticket count grows for customers, especially with agencies we work closely with, that has slowed down the loading times of the client area quite a bit for those customers when logging in.
We've made a few changes, with how far we query Zendesk's API for tickets, so it should now be much faster; it obviously means you may not see the full history of all tickets you've ever had with us through https://billing.perfgrid.com/; however, this can still always be accessed through the Zendesk portal at https://support.perfgrid.com/
We're still looking into ways to improve this further over time, but for now we at least have a solution that means you don't have to sit and wait for the panel to load all the time.
