Published: March 04, 2025 by PerfGrid
Pulse: Week 10, 2025
Welcome to the third edition of our Pulse series, where we share insights into our infrastructure changes, service improvements, and other behind-the-scenes activities at PerfGrid.
Varnish failure affecting Photon Optimizer
On February 25th we experienced some minutes of downtime on our Photon Optimizer service offered for Grid Hosting customers.
Photon Optimizer, that does AVIF and WebP generation on-the-fly, works by having a load balanced optimization cluster in Frankfurt that handles the optimization logic.
We then have 4 locations across the world, specifically Amsterdam, Miami, Los Angeles and Singapore, these locations are effectively a set of caching servers based on HAProxy and Varnish Cache.
During an automated update triggered by Ubuntu's unattended-upgrades process, it resulted in a bug within Varnish Cache on our Amsterdam location, that essentially corrupted some of the cache, this is usually not an issue since there's mechanisms to handle this. However, the way the corruption was done, resulted in segmentation faults happening every now and then, which would cause the process to restart.
However, twice during this automatic restart, Varnish didn't start correctly again, resulting in it triggering some downtime.
Our DNS health monitoring automatically detects this, and reroutes people to another region, in this case Miami, which would be the second closest location.
We removed Amsterdam from routing, performed a reset of the caching storage layer, which in turn resolves the problem.
We've also taken further steps to prevent this from happening in the future, such as making unattended-upgrades unavailable on the particular servers, so it will go into our normal maintenance schedule for keeping packages up to date in a more controlled fashion.
DNS zone export available in Grid Hosting
We've added the possibility to export DNS zones in AXFR format (sometimes known as bind format as well), this allows customers to export their zone configuration (excluding DNSSEC) in case people are moving the domains away from our DNS management, such as moving it to Cloudflare or other DNS providers.
Since we were working on the DNS page anyway, we've also made a few UI changes, specifically to how DNSSEC is displayed so it looks nicer, and hopefully is a bit more descriptive how to use it, in case you're not too familiar with DNSSEC.
Rspamd fuzzy premium dataset
It's no secret that we use Rspamd for our inbound and outbound filtering for email at PerfGrid, in fact, we've talked about How PerfGrid shield your inbox: Inside our Rspamd-Powered Spam Filtering Solution article from October last year.
One of the more recent changes we've done, is to add the Rspamd Premium Fuzzy storage backend to our filtering servers, so we can detect new waves of spam even quicker.
This is both done to improve the overall quality of the filtering, but more importantly to support Rspamd in their efforts to continue development and maintenance of the Rspamd software and systems.
