Coffee Space


Listen:

Server Migration

Preview Image

Introduction

C1 rack cluster

For many years now, I have been happily using my C1 instance over on Scaleway, based in France. It’s a baremetal four core ARM server that I don’t share with anybody, essentially a Raspberry Pi. I’ve have been a very happy paying customer for quite some years now.

Screenshot from LowEndTalk

Unfortunately, quite recently it has been discontinued, and today is the last day (as of writing) that the server will continue to run. I believe they no longer want to support the old hardware or support customers, despite me never really complaining about the service ever and being quite happy (even still). I am therefore migrating to a new server…

Old Server

Single C1 board

Firstly, let’s consider the existing resources I have been paying some 5 euros a month for…

0001 $ free -h
0002               total        used        free      shared  buff/cache   available
0003 Mem:           2.0G         96M        229M         31M        1.7G        1.5G
0004 Swap:            0B          0B          0B

So one good thing is that we actually don’t use much RAM at all. Most of our existing RAM goes to waste, so we can afford to downscale.

We are using 14GB of our 50GB SSD drive:

0005 $ df -h
0006 Filesystem      Size  Used Avail Use% Mounted on
0007 udev           1008M     0 1008M   0% /dev
0008 tmpfs           203M   30M  173M  15% /run
0009 /dev/nbd0        46G   14G   31G  31% /
0010 tmpfs          1011M     0 1011M   0% /dev/shm
0011 tmpfs           5.0M     0  5.0M   0% /run/lock
0012 tmpfs          1011M     0 1011M   0% /sys/fs/cgroup
0013 tmpfs           203M     0  203M   0% /run/user/0
0014 tmpfs           203M     0  203M   0% /run/user/1000

And the internet is not so bad either:

0015 $ ping 8.8.8.8
0016 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
0017 64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=1.79 ms
0018 64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=1.48 ms
0019 64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=1.72 ms

Not too bad!

New Server

Now what are we getting with the new $15 per month server? Not so much…

0020 $ free -h
0021              total       used       free     shared    buffers     cached
0022 Mem:          128M        47M        80M       7.9M         0B        16M
0023 -/+ buffers/cache:        30M        97M
0024 Swap:           0B         0B         0B

Hmm, much less RAM at just 128MB - we will certainly be challenged here.

0025 $ df -h
0026 Filesystem      Size  Used Avail Use% Mounted on
0027 /dev/simfs       10G  582M  9.5G   6% /
0028 devtmpfs         64M     0   64M   0% /dev
0029 tmpfs            64M     0   64M   0% /dev/shm
0030 tmpfs            64M  5.2M   59M   8% /run
0031 tmpfs           5.0M     0  5.0M   0% /run/lock
0032 tmpfs            64M     0   64M   0% /sys/fs/cgroup
0033 tmpfs            64M     0   64M   0% /tmp
0034 none             64M     0   64M   0% /run/shm

We also don’t get much disk either! Only 10GB, we will need to think thin!

0035 $ ping 8.8.8.8
0036 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
0037 64 bytes from 8.8.8.8: icmp_seq=1 ttl=114 time=9.84 ms
0038 64 bytes from 8.8.8.8: icmp_seq=2 ttl=114 time=9.78 ms
0039 64 bytes from 8.8.8.8: icmp_seq=3 ttl=114 time=9.74 ms

And as you can see, the networking is not quite nearly as good either. What’s worse is that it’s a super old version of Debian 8, so we’ll have to do some crazyness just to get it to even work.

Setting Up

We follow this guide to install apt sources by adding this line to /etc/apt/sources.list:

0040 deb http://archive.debian.org/debian/ jessie-backports main contrib non-free
0041 deb-src http://archive.debian.org/debian/ jessie-backports main contrib non-free

And then we need to add an exception because it’s super old:

0042 $ echo 'Acquire::Check-Valid-Until no;' > /etc/apt/apt.conf.d/99no-check-valid-until

First things first, time to update the machine:

0043 $ apt-get update; apt-get upgrade

Even though it is new, there are 103MB of updates to be installed.

Next I want to create a user for running stuff under non-root:

0044 $ adduser user
0045 $ usermod -aG sudo user

Remove Apacahe:

0046 $ service apache2 stop
0047 $ apt-get purge apache2 apache2-utils apache2.2-bin

Next we want to kick off the installing process for some useful packages:

0048 $ apt-get install \
0049   git             \
0050   nginx           \
0051   screen          \
0052   openjdk-7-*     \
0053   pandoc          \
0054   espeak          \
0055   ufw             \
0056   default-jre     \
0057   default-jdk     \
0058   ffmpeg          \
0059   ant

Now to setup the firewall:

0060 $ ufw allow ssh
0061 $ ufw allow http
0062 $ ufw allow https
0063 $ ufw enable

From the old server, I now want to start copying over files:

0064 $ scp /root/* root@198.12.85.147:~/
0065 $ scp -r /var/www/* root@198.12.85.147:/var/www
0066 $ scp -r /home/user/* root@198.12.85.147:/home/user
0067 $ scp -r /etc/nginx/* root@198.12.85.147:/etc/nginx
0068 $ scp -r /srv/* root@198.12.85.147:/srv

And then we want to actually own the files (from the user account):

0069 $ sudo chown 667 -R /var/www
0070 $ sudo chown 667 -R /home/user
0071 $ sudo chown 667 -R /srv

Update the screwed up permissions in the user directory (as root):

0072 $ find /home/user -type d -print0 | xargs -0 chmod 0775
0073 $ find /home/user -type f -print0 | xargs -0 chmod 0664

Next I need to generate an SSH key and copy the output to the relevant places:

0074 $ ssh-keygen # Press enter multiple times
0075 $ cat ~/.ssh/id_rsa.pub # Copy this output

Now I spin up the main programs running the website…

Then update the DNS records to point back at the correct server…

Now setting up certbot (to handle SSL) (NOTE: As certbot is too old, we are forced to use acme.sh from here):

0076 $ wget -O -  https://get.acme.sh | sh -s email=###
0077 $ acme.sh --issue               \
0078           -d coffeespace.org.uk \
0079           -w /var/www/website/www/
0080 $ acme.sh --install-cert                        \
0081           -d coffeespace.org.uk                 \
0082           --cert-file /etc/nginx/cert           \
0083           --key-file /etc/nginx/key             \
0084           --fullchain-file /etc/nginx/fullchain \
0085           --reloadcmd "systemctl reload nginx.service"

And now we need to point nginx to this…

0086     listen [::]:443 ssl ipv6only=on;
0087     listen 443 ssl;
0088     ssl_certificate /etc/nginx/fullchain;
0089     ssl_certificate_key /etc/nginx/key;

And check the configuration with nginx -t - hopefully it returns okay!

And done! Kinda.

Some Changes

So 128MB turns out to be a very small amount of RAM, especially for a server that does so much! I’ve hard to rip out a few of the previous features…

IRCd is now gone for now, but it was severely under utilized anyway. It turns out that nobody really intends to speak to me over on there, which is fine. Even if they did, I simply wouldn’t have the time to join them.

The next thing is that cgit is now also gone - it was already extremely RAM and CPU intensive anyway, and it simply won’t fly on this server. I can’t even store the repository mirror files on the new 10GB disk, so it was never really an option.

Lastly, I shut down the old Scaleway C1 instance. Very sad to see that server go after 5 years of almost perfect service, but when needs must. Hopefully the new server is just as good in this regard!

In general, the migration took about a day as I needed to figure out how to get the code working on an older server OS. It was quite a pain, but ultimately I was able to figure out how to migrate the server into a resource constrained option for the most part.