Maintenance
It is expected that initially, this could cost 200 man hours to prepare and later 10 hours per month to maintain. Over 10 years this could cost 1400 man hours.
Coffee Space 
The first thing that got me thinking about the importance of archiving data is the famous case of the missing Dr Who episodes, of which the BBC routinely deleted archived programs and deciced it wasn’t worth archiving old Dr Who episodes anymore [1]. 97 epsiodes of the 253 episodes from the programs first siz year are gone.
The reason these things dissapeared was not due to lack of popularity, lack of archiving capability (VHS) or lack of want to archive. I believe the real reason these were lost is because nobody (at least of significance) ever took it upon themselves to do so. There was possibly some assumption that “it’s okay, somebody else will surely have done it”, where that simply wasn’t true.
For me, there are things I really don’t want to lose, such as certain Youtube channels, films, books and various other media. There are some things that I have come across in this life that would simply we a shame to lose, things that I think the future generations of the human race ought to have available to them.
People have a certain expectation of archived material, listed below:
Of course, not all of this is possible, as will be discussed in the “Issues” section.
The following are the predicted issues with the project:
With these issues taken into consideration, we will attempt to tackle them.
Firstly, the restriction that will be made that only content that is no easily reproducable will be stored. A billion digits of Pi for example would not be considered a good use of archiving resources, whereas the works of Shakespear would be considered worthy of archive.
By far the easiest format to compress, human written text compresses very well due to it’s built in redundancy. We should try to encapsulate as much information in this form as possible to increase our ability to compress data.
The information that should be included in this category are:
txt files. These can be upgraded to include further formatting that helps the text be more readable.pdf files. Where the PDFs have images, they shall be extracted and stored in the image database.htm and html files. These will be extracted as best as possible, but clearly presentation will be lost.A proposed format for this data is markdown (md), as discussed in a previous article - an appropriate format for storing data in a near pure format. Some documents can be stored in the standard Unicode format as to include special characters, which may be paramount to the data. If possible, only 7 bit Ascii will be stored.
Images will come in many formats and will be storted in various versions of jpg, to a quality that allows the original meaning to be known. A 4k image of a circle is both easily reproducable and pointless to store in large resolution.
Video is an interesting problem, as it is naturally very large. It needs to be stored in both a future proof format and something that compressed well. It may be worth limitting the video resolution, video colour depth and video sound quality in a controlled mannor. It’s bad enough that the quality must be dropped, it makes sense to at least do this in a controlled mannor.
For now, with no real need to serve data, a 10Mb connection would suffice. This would easily serve 100 connections at a time given that each connection is restricted to approximately 100kb.
This connection would likely cost £10 per month, or £1200 over 10 years. This is a sizeable cost, but may be reduced if the internet connection also served other purpsoses. This of course also assumes that the cost of a world wide connection does not reduce over time.
Each hard drive would approximately need 10W [2], assuming 10 hard drives we would need 100W to serve the hard drives alone. The device to power these hard drives consumes 100W of power [3].
As for the computer, assuming the worst power consumption for any varient of the Raspberry Pi, we could consume as much as 5W [4].
Each hard drive will need a data connector to convert the system to USB [5], with power consumption being roughly equal to that of the hard drive already calculated.
It is expected that initially, this could cost 200 man hours to prepare and later 10 hours per month to maintain. Over 10 years this could cost 1400 man hours.
For computation, it has been decided that the Raspberry Pi would be a good fit as it offers many features, including but not limited to:
The cost of this device is approximately £30, making it a very affordable option. A USB hub with 10 ports will also need to be purchased for data communication.
SATA drives have been chosen for this task as the current cheapest form of memory. Currently, these are also well supported and available. It is anticipated that this may not be the case in the future.
10 drives with 4TB each would in theory give you 40TB. Assuming a third redundancy, you may be looking at just over 10TB of safe storage data. Another decision would be to implement a safer file systen, such as ZFS (Oracle), which is supposedly more difficult to corrupt.
From past experince, hard drives have an operating temperature of about 50 degrees. Anything hotter than this may have a bad effect on the performance and life span of the disks. It’s therefore important to make sure there is enough space for cooling. This requires some kind of rack.
For the purpose of cost, it makes sense to build a custom storage rack for the hard drives, computer and various power supplies. The drives can be loaded sideways, meaning the drives aren’t stacked. This should significantly reduce the heat building up between drives.
The cost should not exceed £50.
A simple web server could be used to serve, through the use of nginx. This can be configured to run PHP and connected to an sqlite database. Some roles of the webserver and database combination would be:
In the file storage, the way to get the energy consumption down is to reduce the amount of devices that are on. This could be automated using umount/mount, some relays and some settings in the database.
For 10TB for 10 years, it looks like one could easily sink £5k into the project and a lot of time. Whilst a good cause, at the time of writing it’s outside of my budget. Given one or two more years, it’s likely that a lot of this technology may be more advanced and cheaper.
[1] https://en.wikipedia.org/wiki/Doctor_Who_missing_episodes
[2] http://www.tomshardware.co.uk/forum/267776-32-hard-drive-power-consumption
[3] https://www.amazon.co.uk/Compact-12-Port-Charger-ORICO-DUB-12P/dp/B01IVF3WK0/ref=sr_1_5?ie=UTF8&qid=1483826650&sr=8-5&keywords=100W+usb+hub
[4] https://en.wikipedia.org/wiki/Raspberry_Pi
[5] https://www.amazon.co.uk/Serial-Adapter-Cable-Laptop-Drive/dp/B0182C8Z40/ref=pd_vtph_107_lp_t_3?_encoding=UTF8&psc=1&refRID=HJYX2VNAWAG0AFKEQ8YS