Coffee Space


Listen:

Server Migration

Preview Image

Preview Image

Introduction

C1 rack cluster

C1 rack cluster

For many years now, I have been happily using my C1 instance over on Scaleway, based in France. It's a baremetal four core ARM server that I don't share with anybody, essentially a Raspberry Pi. I've have been a very happy paying customer for quite some years now.

Screenshot from LowEndTalk

Screenshot from LowEndTalk

Unfortunately, quite recently it has been discontinued, and today is the last day (as of writing) that the server will continue to run. I believe they no longer want to support the old hardware or support customers, despite me never really complaining about the service ever and being quite happy (even still). I am therefore migrating to a new server...

Old Server

Single C1 board

Single C1 board

Firstly, let's consider the existing resources I have been paying some 5 euros a month for...

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           2.0G         96M        229M         31M        1.7G        1.5G
Swap:            0B          0B          0B

So one good thing is that we actually don't use much RAM at all. Most of our existing RAM goes to waste, so we can afford to downscale.

We are using 14GB of our 50GB SSD drive:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev           1008M     0 1008M   0% /dev
tmpfs           203M   30M  173M  15% /run
/dev/nbd0        46G   14G   31G  31% /
tmpfs          1011M     0 1011M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs          1011M     0 1011M   0% /sys/fs/cgroup
tmpfs           203M     0  203M   0% /run/user/0
tmpfs           203M     0  203M   0% /run/user/1000

And the internet is not so bad either:

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=1.79 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=1.48 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=1.72 ms

Not too bad!

New Server

Now what are we getting with the new $15 per month server? Not so much...

$ free -h
             total       used       free     shared    buffers     cached
Mem:          128M        47M        80M       7.9M         0B        16M
-/+ buffers/cache:        30M        97M
Swap:           0B         0B         0B

Hmm, much less RAM at just 128MB - we will certainly be challenged here.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/simfs       10G  582M  9.5G   6% /
devtmpfs         64M     0   64M   0% /dev
tmpfs            64M     0   64M   0% /dev/shm
tmpfs            64M  5.2M   59M   8% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            64M     0   64M   0% /sys/fs/cgroup
tmpfs            64M     0   64M   0% /tmp
none             64M     0   64M   0% /run/shm

We also don't get much disk either! Only 10GB, we will need to think thin!

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=114 time=9.84 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=114 time=9.78 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=114 time=9.74 ms

And as you can see, the networking is not quite nearly as good either. What's worse is that it's a super old version of Debian 8, so we'll have to do some crazyness just to get it to even work.

Setting Up

We follow this guide to install apt sources by adding this line to /etc/apt/sources.list:

deb http://archive.debian.org/debian/ jessie-backports main contrib non-free
deb-src http://archive.debian.org/debian/ jessie-backports main contrib non-free

And then we need to add an exception because it's super old:

$ echo 'Acquire::Check-Valid-Until no;' > /etc/apt/apt.conf.d/99no-check-valid-until

First things first, time to update the machine:

$ apt-get update; apt-get upgrade

Even though it is new, there are 103MB of updates to be installed.

Next I want to create a user for running stuff under non-root:

$ adduser user
$ usermod -aG sudo user

Remove Apacahe:

$ service apache2 stop
$ apt-get purge apache2 apache2-utils apache2.2-bin

Next we want to kick off the installing process for some useful packages:

$ apt-get install \
  git             \
  nginx           \
  screen          \
  openjdk-7-*     \
  pandoc          \
  espeak          \
  ufw             \
  default-jre     \
  default-jdk     \
  ffmpeg          \
  ant

Now to setup the firewall:

$ ufw allow ssh
$ ufw allow http
$ ufw allow https
$ ufw enable

From the old server, I now want to start copying over files:

$ scp /root/* root@198.12.85.147:~/
$ scp -r /var/www/* root@198.12.85.147:/var/www
$ scp -r /home/user/* root@198.12.85.147:/home/user
$ scp -r /etc/nginx/* root@198.12.85.147:/etc/nginx
$ scp -r /srv/* root@198.12.85.147:/srv

And then we want to actually own the files (from the user account):

$ sudo chown 667 -R /var/www
$ sudo chown 667 -R /home/user
$ sudo chown 667 -R /srv

Update the screwed up permissions in the user directory (as root):

$ find /home/user -type d -print0 | xargs -0 chmod 0775
$ find /home/user -type f -print0 | xargs -0 chmod 0664

Next I need to generate an SSH key and copy the output to the relevant places:

$ ssh-keygen # Press enter multiple times
$ cat ~/.ssh/id_rsa.pub # Copy this output

Now I spin up the main programs running the website...

Then update the DNS records to point back at the correct server...

Now setting up certbot (to handle SSL) (NOTE: As certbot is too old, we are forced to use acme.sh from here):

$ wget -O -  https://get.acme.sh | sh -s email=###
$ acme.sh --issue               \
          -d coffeespace.org.uk \
          -w /var/www/website/www/
$ acme.sh --install-cert                        \
          -d coffeespace.org.uk                 \
          --cert-file /etc/nginx/cert           \
          --key-file /etc/nginx/key             \
          --fullchain-file /etc/nginx/fullchain \
          --reloadcmd "systemctl reload nginx.service"

And now we need to point nginx to this...

    listen [::]:443 ssl ipv6only=on;
    listen 443 ssl;
    ssl_certificate /etc/nginx/fullchain;
    ssl_certificate_key /etc/nginx/key;

And check the configuration with nginx -t - hopefully it returns okay!

And done! Kinda.

Some Changes

So 128MB turns out to be a very small amount of RAM, especially for a server that does so much! I've hard to rip out a few of the previous features...

IRCd is now gone for now, but it was severely under utilized anyway. It turns out that nobody really intends to speak to me over on there, which is fine. Even if they did, I simply wouldn't have the time to join them.

The next thing is that cgit is now also gone - it was already extremely RAM and CPU intensive anyway, and it simply won't fly on this server. I can't even store the repository mirror files on the new 10GB disk, so it was never really an option.

Lastly, I shut down the old Scaleway C1 instance. Very sad to see that server go after 5 years of almost perfect service, but when needs must. Hopefully the new server is just as good in this regard!

In general, the migration took about a day as I needed to figure out how to get the code working on an older server OS. It was quite a pain, but ultimately I was able to figure out how to migrate the server into a resource constrained option for the most part.