We are big proponents of the Elastic Stack here at CrossBrowserTesting – we use it for gathering metrics about system performance, tracing down issues in our stack, and identifying targeted improvements to our systems.
Elastic, the company that produces the Elastic Stack’s core tooling, offers a lot of flexibility and power. We are able to take whatever data we want to examine and make use of it in the Elastic Stack, which opens up a lot of doors in a system the size of ours.
Elasticsearch is Elastic’s search and analytics engine, and I’d be lying if I said I wasn’t a big advocate of it. It’s fast, scalable, and efficient – even on modest cloud instances, we can have thousands of messages stored, indexed, and aggregated without the system even breaking a sweat. It has joined the ranks of HAProxy as being one of my go-to examples of incredible software design.
Elasticsearch provides a JSON-based API and gives us a lot of novel ways to look at our data. Some examples include filtered indexes, aggregation, and roll-up indexes that we can use to summarize data for long-term storage and use.
Kibana is Elastic’s front-end for Elasticsearch. Its interface can be daunting at first, but there’s plenty of utility laying right under the surface. The log viewing interface provided by Kibana makes it easy to see anomalous states in our logs; the ability to make dashboards out of any set of visualizations, allowing for anything from performance metrics to error breakdowns; there are APM and platform metrics modules that can be attached, offering insight into where performance issues are coming from with pre-configured dashboards and easy-to-set up integrations.
Logstash is Elastic’s heavy-duty logging tool, used for parsing or cleaning up incoming data for ingestion by Elasticsearch. The ramp up for it is pretty straightforward and it is extremely powerful. The use of Grok and Oniguruma expressions make it easy to follow what you’re actually parsing in the text – a real step above using standard regular expressions.
The introduction of Beats by Elastic was one of the biggest changes to their ecosystem. With Beats, there are simple-to-use, powerful, pre-packaged tools for turning existing data into actionable data inside Elasticsearch.
Elastic Stack Beats and what they do:
- Filebeat: Reads the content of files and makes them available in Elasticsearch
- Metricbeat: Sends process and system metrics to Elasticsearch
- Packetbeat: Sends network metrics to Elasticsearch
- Winlogbeat: Sends Windows event logs
- Auditbeat: Sends audit and integrity data
- Heartbeat: Sends uptime and ping data
- Functionbeat: Sends data for serverless architectures
These tools fit the most common needs of most businesses. Of course, I wouldn’t be writing about extending the Elastic Stack to meet our needs if our needs were completely within those boundaries, would I?
How to build your own Beats
The Beat platform that Elastic releases for the public is built on libbeat, a Go library that handles things like configuration parsing, sending data, and command line arguments. All of the current versions of Beats are built on this library.
Fortunately, libbeat is readily available!
You can find Elastic’s guide to using libbeat here. If you know Go, you can get started on writing custom data shippers right away with this guide.
The second thing we built at CrossBrowserTesting using libbeat was a custom shipper for Android mobile device logs. We have hundreds of real Android devices and trying to keep track of logging across that many devices poses a unique challenge that the majority of businesses will never have.
With Androidbeat, our in-house data shipper for Android mobile devices, we had several requirements:
- It needed to use the standard Android Debug Bridge (adb) functionality built in to Android
- It needed to handle temporary unavailability of a device gracefully
- It needed to be able to add in metadata about the devices attached, such as our unique identifiers, the severity of the logs, and the time the message was sent.
We probably could have hacked something together using Filebeat and the existing adb tool, but it would be fragile and require a number of moving parts. Not something we want to build in a complex system.
Fortunately, each problem was straightforward to solve:
- We were able to find an adb library for Go that made connecting to our Android devices easy
- By having a goroutine stop when the connection is broken and having it start again when the device is reconnected, we’re able to gracefully handle disconnects, and by tracking the time of the last logs, we can prevent ourselves from getting duplicates
- Thanks to the configuration parser inside libbeat, it’s easy to create a YAML file that contains things like what device ID something has, what type of device it is, even a specific log level for that family of devices or even just a specific device, and this data can then be inserted into the data that gets sent to Elasticsearch
It took about a day to build Androidbeat, but there is one major caveat that, should you feel like traveling down this rabbit hole, you likely want to be aware of: your first project using libbeat will take considerably longer. I realize this sounds like just not accounting for spin-up time, but the libbeat platform has a few things that make getting started take a little longer, such as a new build system (using mage), a set of setup scripts that, while they make it much easier to get going with the full functionality, can also add a layer of confusion, and a build configuration system that is, to put it mildly, somewhat arcane.
Overall, the Elastic Stack is extremely powerful, and, even better, it’s open source (under the Apache 2.0 license) and more or less freely available, the only caveat being that certain features are locked behind the licensing process. Like any powerful tool, there’s a learning curve, but you can make it do whatever you need it to do.