Egil Hasting, Synergy SKY's CTO, gives insights into why we built our solution on top of FreeBSD
Synergy SKY offers multiple software solutions, however, the one I am going to talk about here is an analytic solution for video-related CDRs (Call Detail Records).
The solution (Synergy ANALYZE) is harvesting CDR data from various sources, then trying to correlate, group, and match data together. This gives us the fundament for various analytics and reporting services on our platform.
The platform is utilizing XMPP as a carrier bus, together with web-frontend in angularjs. This all requires a lot of various components to be installed (XMPP server, database, web server, misc runtimes, and various homebrewed bins), configured, and managed on each installation. Also since we are using XMPP, the general architecture is distributed by nature (a final deployment consists of two servers). At an early point in the development we figured out we needed an easy way of deploying our software, this lead us to some choices.
One is to avoid the whole customer premise install and make a pure cloud solution, however, this kind of architecture has too many caveats in terms of data storage, compute requirements, and regulation (also given the new GDRP in EU). The second option would be an on-premise installation of the software, which is what we probably needed to do… But how, when the stack is complex?
We landed on that we wanted to push out appliance hardware, eventually, a virtual appliance using VMWare or similar. This gives us the ability to create a controlled environment with the full stack included, and in the expected working state.
In a very very very simplified way, the diagram looks like this:
Now we needed to decide on what would be the OS of the hardware or virtual hardware (The one called "SAS Appliance at Customer location" in the diagram)? Linux of course.. we had experience with CentOS from previous products and building virtual appliances. However the more we talked about it, the more we realized that we should be looking around for options.
One major issue we experienced was that Systemd managed to crash the whole dbus-systemd connection if our software took too much memory, leaving the system in an unstable state, with the reboot as the only option (I don’t think we can blame CentOS directly for that, however pulling in such premature technology into a proclaimed stable platform, makes no sense at all!). We had some other minor issues as well which are probably weeded out in the latest versions of CentOS (and not worth mentioning).
We had already a lot of experience with CentOS Linux, which is a very good thing when deciding on what to do. And it is usually better to know the problems and work around them than starting on scratch with a new distro/OS!
What we looked into in regards to building the platform was:
- Container technology (docker was tested extensively, LXC was tested some)
- Package and repo manager (yum/dnf)
- Automated install (never tested with CentOS, but we knew it was possible)
- Service management system (Systemd)
- How to deal with the basic configuration of the system (network, service start, etc.)
But still, with the knowledge of CentOS, we wanted to broaden our view.
Next up was FreeBSD - which a colleague had been tinkering with, for pure fun - installing FreeBSD 9, compiling anything from the ground up on his home computer.
He was very positive about the concept of FreeBSD being a full OS, with a proper base system, a well-designed package system for building your own packages, and the demarcation between OS and Userland package system, leaving the system easy to maintenance. Also the jail technology and was interesting since we were investigating using docker as a container on the server-side, and looked at this as an option to that.
However, the thought of building anything from scratch was not appealing. However, I agreed on testing out getting our early version of the software with the required stack up and running quote: "to get some impression, before moving on" - with quests for another/the perfect Linux distro.
The first impression of FreeBSD 10.3 was way beyond expectations. The installation was done within minutes, the online handbook was exceptionally well written. To my surprise, there were precompiled packages, and everything just worked neatly out of the box. I was amazed by the fact that after a few hours with FreeBSD, I had the whole stack running.
With this experience and the appealing case of FreeBSD Jail and ZFS (which is matured and tested solutions we both needed), we stopped looking around for alternatives instantly, and instead focused on testing and learning more about FreeBSDThe table from above became looking like this.
- Container technology (docker was tested extensively) ⇒ FreeBSD Jails
- Package and repo manager (yum/dnf) ⇒ Pkgng / pkg / ports
- Automated install ⇒ bsdinstall (we even build a pxe installer image that would get you up and running with a dev. System within minutes)
- Service management system (Systemd) ⇒FreeBSD rc, and later on we also adapted runit for some services.
- How to deal with the basic configuration of the system (network, service start, etc.) ⇒ FreeBSD rc and rc.conf
We now started building our application around the FreeBSD appliance utilizing Pkgng (pkg) as our binary package format for distributing. We have of course had some minor challenges here and there but never with stability and performance. The clear separation between OS and userland software (port/pkg repo) is logical and perfectly executed. This also automatically makes the whole system cleaner in sense of "where do I find the configuration", and also - where should I place my data.
FreeBSD has given us the "it just works" experience, without sacrificing full control. It feels and behaves mature and stable with great performance!
Using FreeBSD has been great, I have also changed all my workstations to FreeBSD to learn the platform better, and the few times I miss Linux (Windows or OS X) is when I want Spotify and/or Netflix who both require compiled binaries that do not exist for FreeBSD.
I am very glad we took our heads out of the sand and looked around because FreeBSD turned out to be a real gem! It's now been 2 years into our journey with FreeBSD and we haven´t looked back.
Other honorable mentions which make FreeBSD great:
- Freebsd-update, keeps your base OS updated, and is separated from the pkg system.
- Pkg repository is up-to-date, which is not always the case for ubuntu/debian and/or CentOS.
- Small footprint out of the box.
- Very easy to modify, rebuild and distribute custom modifications to packages using the ports system.
- Tightly integrated with ZFS (not just a userland driver).
- Easy license to deal with commercially.
- Network stack (there are in total 3 different firewalls that are included in FreeBSD, we landed on pf)
If you are a developer and you think it would be challenging and fun to work in our team check out www.synergysky.com/careers or contact Egil directly.