Coffee Space


Listen:

Website Down

Preview Image

I recently thought to myself as I sometimes do: “It’s about time I write another article for my website”. Imagine my surprise when I go to the website URL, and it’s down. No big deal I tell myself, the VPS is probably gone again. I try to SSH the website, nothing. I whois the domain for the IP address and ping it - nothing. I login to RackNerd and check the VPS status panel - the VPS is not there. I check history of the VPS - nothing. I start to think “maybe after all these years, I finally got some kind of take-down request”. I check the tickets - nothing.

Finally, I write a ticket to support to find out what is going on. It turns out that they terminated the VPS. Apparently, they have been trying to contact me to pay the annual fee. I check my emails, nothing. I check the notifications in the panel, nothing.

It turns out that RackNerd have been sending out emails, but they have been bouncing off of GMail the whole time. After several months (which is, to be fair, longer than they should have), they terminated the VPS and deleted the virtual disk. Damn.

Fortunately for me, ~2014 I was lucky enough to have been mentored on a project by Drew (whose own site is sadly down), whom taught me the basics of vim and git (and Linux more generally). As a result, the site has a full and perfect history mirrored in several locations - happy days. After setting up a new VPS instance, installing nginx, installing cert-bot, installing a few packages (espeak, ffmepg, pandoc-filters, etc), the site was up and running without a few hours. This could of course have been a more optimal process, but this site is held together with bash scripts and spit(e).

Learning

The ‘social’ (dead social) element that drives the comments section did not have such a backup, but luckily I do have an archive I can dig out and recover it almost entirely. I should really make it more robust.

I used to use an old promo-code from Low End Box where RackNerd offer:

  • 1x vCPU Core
  • 10GB SSD Storage
  • 768MB RAM
  • 1000GB Monthly Bandwidth
  • 1Gbps Public Network Port
  • 1 Dedicated IPv4 Address
  • KVM / SolusVM
  • All Linux flavors supported
  • Pricing: $10.28/year
  • [ORDER HERE]

I have been bumping into that 768MB of RAM regularly. The start-up process of the server has to be done carefully so that it doesn’t try to build too much at the same time. The git server would regularly crash because it would run out of RAM.

The support team did helpfully suggest that I look at their Black Friday offerings, and I found this:

  • 1 vCPU Core
  • 20 GB Pure SSD Storage
  • 1 GB RAM
  • 1500 GB Monthly Transfer
  • 1 Gbps Network Port
  • Full Root Admin Access
  • 1 Dedicated IPv4 Address
  • KVM / SolusVM Control Panel
  • FREE Clientexec License
  • Available in: Multiple Locations
  • ONLY $10.99/YEAR!

Double the SSD space, 256MB (1/3) of RAM extra, 50% extra monthly transfer, for an extra 70 cents a year. Not bad!

These resources of course are easily used up…

0001 $ df -h
0002 Filesystem      Size  Used Avail Use% Mounted on
0003 tmpfs            97M  972K   96M   1% /run
0004 /dev/vda2        19G   16G  2.1G  89% /
0005 tmpfs           481M     0  481M   0% /dev/shm
0006 tmpfs           5.0M     0  5.0M   0% /run/lock
0007 tmpfs            97M   12K   97M   1% /run/user/1000
0008 $ free -h
0009                total        used        free      shared  buff/cache   available
0010 Mem:           961Mi       339Mi       129Mi        11Mi       664Mi       622Mi
0011 Swap:          1.0Gi        36Mi       987Mi

Dead Social

Getting this running again was a pain, I do not exaggerate. It has been years since I last touched java, and it looks like some of the ‘clever’ Java reflection I did is no longer supported.

The core issue boiled down to the String implementation switching from using char[] internally to byte[] internally, causing the following cast to fail:

0012 char[] d = (char[])(stringField.get("#"));

This is likely because they switched from supporting some double-byte String encoding to variable-length Unicode. Luckily, a design choice I made back then was to support ASCII-7 only, meaning that my String implementation would not need to support anything crazy, meaning that the following hack works:

0013 public static char[] byte2char(byte[] a){
0014   char[] b = new char[a.length];
0015   for(int i = 0; i < a.length; i++){
0016     b[i] = (char)a[i];
0017   }
0018   return b;
0019 }

And when bringing in string data, we pay a runtime overhead for converting it:

0020 char[] d = byte2char((byte[])(stringField.get("#")));

I’m pretty sure that all of the performance improvement is now gone from the custom String implementation, which tries its hardest to avoid any copying until the very end. But it was far easier to deliver this patch.

In the future, the database should be a flat binary file (rather than thousands of little files) and these performance improvements can likely be dealt with via a better cache strategy.

Finally

The main point is that we are back in business. Some more things need to be done to improve the robustness, but that is a discussion for another day.