A large chunk of the Internet just broke yesterday, here is how Quad State Internet responded

Posted October 1, 2021 by Preston Louis Ursini

As reported by ZDnet, the problem was widespread

Yesterday, the Quad State Internet Network Operations Center was alerted by an automated system that there was an anomaly in accessing certain services.

What we quickly realized is that Let's Encrypt, a popular free service that provides SSL certificates to web hosts and other Internet Service Providers, just had their root certificate expire. In anticipation of this, a new root certificate was created, however, many popular software systems were still utilizing the old root certificate.

Let's Encrypt has issued official statements on the matter

Because of this issue, many systems became unavailable for many people on the Internet. This was the type of scenario that technicians at the Quad State Internet NOC train for. While other ISPs have their technicians leave at 5 PM, or don't provide 24/7 support, Quad State Internet monitors its network and always has a trained technician available.

The nature of this issue meant that there was little to no documentation on this specific problem, where a patched system could no longer validate certificates. The fix required the old certificate be disabled in many CA trust stores, even on updated systems such as macOS.

After responding, Quad State Internet technicians coordinated and assisted other technicians at other companies by providing guidance and specific technical information to others.

This is where the difference between an experienced technician comes into play. A Google search of the problem was not going to yield any results, it takes a core understanding of these systems to understand the error presented, and to understand how the underlying software is validating these certificates.

At the end of the day, this meant that having the new Let's Encrypt certificate on client systems was not enough, on some operating systems, the certificate had to be disabled as well to prevent the Operating System from validating against the expired certificate rather than the new one. While most patched systems took care of this automatically, some did not; which meant skill, not repetition was required.

OpenSSL, a popular encryption library, shows letencrypt.org as having an invalid SSL certificate, even on a fully patched operating system.

This is where the value of having experienced, trained, on site technicians by your side. While the operating systems and libraries were patched, all software publishers did not anticipate this issue. Thanks to testing, alerting, and having the proper staff available, Quad State Internet was able to remediate the issue with little to no impact to customers.

Find more posts