If you have moved to the AWS Cloud, or are in the process of doing so, chances are you don’t have a great deal of time to worry about monitoring it. Now, ideally there would be an off-the-shelf solution to take this pain away. This was the aim for the AWS CloudWatch service, but we decided to go a step further. Today we are pleased to announce the release of the Dataloop.IO AWS integration.
At the November DOXLON, Facebook gave us an in-depth look into the way we have managed containers over the past 8 years, and why a complete overhaul is coming, with an entirely new control group system. Epam Systems showed us how to use the ELK stack together with Beats Plugin data shippers for system metrics, along with monitoring multiple Docker containers. Neil Chowdhury at Strft focused on how they are using Splunk and data models for collaboration, and the need for semantic logging.
The topic of the October 2016 DOXLON Meetup was container orchestration. The goal: shed light on and compare existing technologies: Kubernetes, Mesos and Docker Swarm. The Meetup started with lighting talks about Kubernetes and Mesos in production at two large companies, Pearson and Schibsted. This was followed by a panel discussion, moderated by our CTO Steven Acreman.
Not a lot of people know this, but our name, Dataloop.IO, derives from the term OODA Loop (It was either that or OODA Daddy!). OODA Loop is a term I learnt about in the Big Data space when we were starting out, and has really driven our thinking around how we’ve designed Dataloop.IO to help online services become more agile and competitive through monitoring.
Running an online service isn't easy. Every day you make complex decisions about how to solve problems and often there is no right or wrong answer, there are just different ways with different results. On the infrastructure side you have to weigh up where everything will be hosted. Is that on a cloud service like AWS, or in your own data centres, or any number of other options, perhaps even a mix.
Open source monitoring can be quite confusing for those who haven't spent a lot of time reading about the options. At Dataloop we track the most popular standards and then offer wire compatible endpoints. This helps new customers migrate onto Dataloop with very little effort, it means we don't invent yet another proprietary standard ourselves and it stops vendor lock-in.
At the beginning of August, we embarked on two huge projects: an in-depth AWS integration, and the first stage of our analytics features. Some customers have been beta testing these for a couple of weeks. We'll do some specific blog posts next week when they go GA.
My Top10 Open Source Time Series databases blog has been incredibly popular with over 10,000 views and growing. It sat on the front page of Reddit /r/programming for a day or two and we got a bunch of traffic from Hacker News and DBWeekly. The raw data in the spreadsheet that accompanies the blog has been constantly updated ever since by a team of volunteers which now includes some of the database authors.
Today is the culmination of over a years worth of hard work (and significantly more than that for its creator Heinz). We're releasing DalmatinerDB v0.2 which is a free, highly scalable time series database built on top of Riak Core that features an expressive data model and query language.