Coffee Space

TL;DR

This was an idea for an open source alternative video chat software that ultimately failed. After having spent weeks experiencing awful video calls, I thought to myself “how hard can this be?”. I learned it’s quite tough.

Restrictions & Implementation

I’ve set myself the following restrictions:

• Don’t worry about audio. I think this is generally a solved or nearly solved problem, at least in my experience.
• Any packet can be dropped without killing the video. This means that dropping several frames shouldn’t be an issue.
• Each frame should fit in a packet. The frame should be less than 4kB.
• The video feed should be 640x480 colour. This is the output of many webcams and is generally an acceptable video quality. We only want the data for face-to-face chats.
• A successful video compression would be less than 30kb/s. This is apparently the video bit rate of Google’s new fancy codec, we should try to beat it.

We also set ourselves the following implementation details:

• The code should be written in Java. It’s where my experience is and I’m confident it should allow me to do what I want.
• Server only links clients. The server isn’t to be used for transmitting the video itself, so a lot of networking can be avoided.

Approach

I used the Java library from sarxos as it worked just out of the box. The library is pretty complete and works reliably across multiple platforms. This unfortunately required the addition of SLF4J for logging and Bridj to provide Java bindings. I used this to implement a camera wrapper that abstracted away the details, such that I could just get a BufferedImage whenever I wanted one.

Next I implemented a basic GUI - this turned out to be the best way of debugging the compression of the images. It was relatively simple to have a basic UI display scaled images to an AWT window.

Initially I implemented JavaFastPFOR for compressing the output of the image data. Whilst it can compress relatively ordered integers pretty well, it was awful at compressing unordered integers. I tried to really limit the amount of data, compressing from RGB (3 bytes) to RBB (1 byte in 3:3:2) - still to no avail and much image quality loss. Unfortunately sending the delta of the image data didn’t work either.

I then looked at simply using the ImageIO compression utilities offered in standard Java. This was slower, but massively reduced the size of the data - turns out JPEGs are pretty good at compressing image data. Whilst the size per image was approximately ~2kB (~16kb), this is well over our budget of ~30kb/s. We’ve expended our budget in just two images.

Conclusion

It didn’t work. Turns out that winging your way through it won’t beat current best methods. Some ideas to try in the future:

• Use smaller images for compression and then use nice upscaling. Obviously smaller input data equals smaller bandwidth use. Maybe we can still get reasonable image quality using 320x240.
• Better development of delta frames. I think this idea had legs, it just needed to be implemented better. Perhaps the delta frame could be a JPEG compressed into a smaller image.
• Colour key frame and monochrome delta frames. The colours in the image shouldn’t be changing all that much.

Hopefully when I get some free time in the future I can chase this up again.