The Old is…still old but at least it’s back

The second part of my Burningbird server re-awakening from the dead is my effort to find any and all past writings and import them into this weblog, one at a time.

I’ve had so many variations of weblogs: some at domains I’ve controlled, others at domains I haven’t. I was able to export the posts from many of the domains, but I haven’t loaded them back into this place. The task just seemed too daunting.

Then I realized something: I’m retired. I can do stuff like this now.

As I note in my About page, you can see many of my writings thanks to the wonderful people at the Internet Archive, and their incredibly important Wayback Machine. Still, I want the posts in one single spot, even though so many of them are so dated.

As I import the page, I set the publication date to the original publication date, which is why you won’t be seeing them here on the front page. I may, from time to time, link an older story in a new posting, for grins and giggles.

I’ve also turned comments back on, though the comment form is a bit buried with this theme. Comments close five days after I post, so make your point quickly. Your first comment will be held in moderation, but after I approve it, you should have freedom to post at will. Do let me know if you run into issues.

It’s been fun to go through the old posts. I can’t believe some of the tech way back when. And we won’t even get into the politics.


Moving servers

It was time for me to upgrade my version of Ubuntu, from 18.04 to 20.04. I upgraded my software, thought I had a clean site, and tried to upgrade in place. After all, upgrading from 16.04 to 18.04 was a simple one line command.

Moving from 18.04 to 20.04 was not so simple, and the upgrade failed. Time to do a manual build of a new server and port my site to it. Which also ended up being more complicated than I thought it would be.

Moving to 22.04

First, if I was going to go through all the work, I was going with the latest Ubuntu LTS: Jammy Jellyfish, otherwise known as 22.04. I spun up a new Linode instance of 22.04 and set to work.

The LAMP installation went very well. I ended up with not quite the latest Apache, since the absolute latest wasn’t supported on 22.04 yet. However, I added the famous Ondřej Surý repository and I was good to go:

sudo add-apt-repository ppa:ondrej/apache2 -y

MySQL is 8.0.29 and PHP is 8.1.

All that certbot stuff

I had manually built a new server when I went from 14.04 to 16.04, but times have changed. That move was pre-HTTPS, pre-HTTP/2, pre-HSTS (HTTP Strict Transport Security), well, basically, pre-everything. I had the support in my existing server, so I know my pages and installation are clean. But the sheer amount of work to set it up again was a bit daunting.

Thankfully, since I had made these moves in the past, my site was already clean. All that I needed to worry about was installing certbot to manage my Let’s Encrypt digital certificates.

You’d think moving a server wouldn’t be that unusual, but neither Let’s Encrypt nor certbot cover what to do when your certificates are on one server and you need to set them up on another. Searching online gave me two options:

– copy everything and change symbolic links for the certificates

– just install new certificates on your new server, and delete the old

And that’s when things got sticky.

Where am I and who is managing my IP address?

When I made the move to 16.04, I was manually setting up my network configuration using ifupdown and editing the /etc/network/interfaces file. But when I went to 18.04, netplan was the new kid on the block for network configuration.

The problem is, I had one foot in both camps. So when I tried to access a test page on the new server, it failed. I certainly couldn’t run the certbot web validation for installing a new digital certificate if I couldn’t even serve a simple page on port 80.

In addition, Linode has the ability to manage network configuration for you automatically, so if you change servers and IP addresses, you don’t have to do a thing. But when I tried to turn it on, even SSH no longer worked. I had to restore the site from a backup.

It took a bit of painful digging around, but I finally upgraded my network configuration to netplan, and only netplan. I could now use SSH again, and install a new digital certificate for my test domain. But then, things got tricky again.

I hate the old propagation thing

When I created the new Linode server, I installed it in the Atlanta data center rather than the Dallas center I was using with the old. After all, Atlanta is now only a couple of hours away.

But doing so meant when I switched, I had to update my name registrar to set my DNS entries to the new server IP addresses. This is a pain, in itself, but it’s also a bit of a fragile time when trying to determine if my site will work on the new server. After all, you don’t want to permanently change your IP address only to find out your site doesn’t work, and then have to change it back. And digital certificates kind of mean you have to have all or nothing.

Thankfully, Linode had a peachy keen workaround: swap IP addresses. If two servers are in one data center, you can swap the IP address between them.

Of course, doing so meant I had to migrate my existing site to the new data center and change the DNS entries, but still, it would be worth it to be able to switch back and forth between servers when making major modifications. And the migration should be a painless button click from the Linode cloud control manager.

So, I migrated my old Linode VPN to Atlanta, and then tried to swap the IP addresses. Crash and Burn.

IPv4 and IPv6

What I didn’t know about the Linode IP swap facility is that it only swapped the IPv4 address, not the IPv6 address. So when I did the following

ip -a

My IPv4 address reflected the new server, but my IPv6 address reflected the old, and everything was just broken. Again.

The only recourse at this point was to bite the bullet, make the move to the new server, do the DNS propagation, and then deal with the digital certificates. I put up a warning page that the site might be off for a time, had a coffee. and just made the move.

After the move, I thought about doing the Let’s Encrypt digital certificate copying like some folks recommended, but it seemed messy—sort of like the network configuration issue I had just cleaned up.

I used certbot to do a new installation, and the move was flawless. Flawless. This is really the only clean way to move your site to a new server when you’re using digital certificates:

– Make sure you site can support port 80, at least temporarily

– use certbot to generate new digital certificates for your site(s)

– delete the old server and certificates

Five Years. I’m good for five years.

So here you are and here I am: on the new server with all new software on the data center closest to me, with clean, uncrufty network configuration and sparkly digital certificates.

Best of all?

Jammy Jellyfish has standard support until April, 2027. I’m good for five years; ten if I want extended support. And who knows where I’ll be in ten years.

Probably back here, doing it all over again.




Just Shelley

The story of the check valve

There’s a story behind this older post. Bit long, but might be fun if you have nothing else going on.

Municipal size check valve with engineer standing in front for perspective
Our place in O’Fallon was in the unincorporated county area, which means it’s a hit or miss what kind of service you get. Originally, the homes in the subdivision had septic tanks, but our plot couldn’t be developed until there was a sewer line, because of the steep hill in back.
When the O’Fallon gravity line was added, our house was barely in range, and they took the lateral sewer line to the manhole, and terminated it in the manhole.

Now, terminating a lateral sewer line in a manhole is a bit of a no-no, but since we were at the start of the line, they thought it would be OK.
The problem is, Lake St. Louis, next door, grew too fast and there were issues with its sewer. So they drilled a _force_ main from Lake St. Louis and terminated it at the manhole for our gravity main.

This is engineering insanity, and actually illegal in most of the country. But such is the quality of sewer management in St. Charles county, Missouri.

What would happen is every time it rained over a certain amount, typically about 1 1/2 inches to 2 inches in a 24 hour period, the water seeping into the force main would overwhelm the man hole and would force the flow back to the next outlet.

Our house.

So when it rained, we couldn’t use our toilets, and if it rained enough, raw sewage would back up into our shower and bath.

I was polite about complaints at first, but then got tired. So, every time this would happen, I would call O’Fallon and tell them we had a sewage backup. They’re required by law to come out to check it out, and it also gets recorded as a sewage backup for the environmental powers that be.

I wasn’t a squeaky wheel…I was a full out bitch. There wasn’t anyone in the sanitation department of two different systems that I didn’t contact by email, or in person.

Water District 2 (force main) blew me off, but O’Fallon took it seriously. The assistant supervisor explored options everywhere for what they could use to protect our lateral sewer line.

In the end, they had the company that developed the check valves you see here, develop a custom made one just for the end of our lateral line in the manhole.

When they installed it, the force main was shut down temporarily and two crews came out: one for the manhole to install the device, and one further down to send fresh air into the manhole.

It installed in about 10 minutes. That was it. And it worked beautifully.

From then on, no matter how much rain, no backup. And we could continue using our toilets, since even when the valve closes, it could still expel liquids out.

When we sold the house, the day before closing, we stayed in a hotel because everything was packed up. That night was one hell of a storm—over 5 inches of rain fell. When we got to the house, the sanitation crew was out washing down our neighbor’s yard because the sewage had overflowed into their yard. It was a mess.

The neighbor across the street came over and said for the first time, they had backup into their house and into their washing machine in the basement. He knew about our previous problems, and he and a couple of other neighbors thought our place would be destroyed by the sewage, and right before closing.

Nope. Not a drop. Clean as a whistle.

These things look simple. No whirring lights, no gadgetry, just plain rubber. But they are brilliant.

And if your wheels don’t squeak nothing good will happen.