At CrossBrowserTesting, we wanted to give our users the ability to schedule their suite runs and screenshot tests to run regularly.
An example user story for this feature would be: Every morning Natalie wants to be reminded to check the status of one of the websites she administers. These tests help her make sure her website looks correct and functions properly. While she drinks her coffee at 8AM, our service will run their tests automatically and notify Natalie once they’re done.
Rather than making our users create a cron job or set an alarm on their phone to remember to do it, we gave them a UI to set up their schedules quickly and conveniently. In this article, we will describe our usage of a node package called Agenda.
Choosing Agenda
We chose Agenda because it fits in our stack as a node module that utilizes MongoDB for storage. The Agenda documentation was very nice and helped us get up and running quickly, and we found there was a set of different options for a tool with the features we needed.
Agenda’s flexibility to have multiple workers, accepting cron format, limiting concurrency, and process frequency all made us feel confident in integrating it into our stack.
Some default behavior we felt it was missing was a reasonable default for failure behavior for jobs and being able to pause a job. Luckily these behaviors were still possible to implement.
A comparison of tools similar to Agenda.
Handling Downtime
We are using an API to do the work of a job, so in the case our service was temporarily down, we need to be prepared for our jobs to fail and act accordingly. To account for this, we automatically reinsert the job into Agenda to run a minute later.
We keep track of the number of times we’ve done this and give up after five tries. We also save the failure message to give to the user.
|
1 |
catch(err){ |
|
1 2 3 |
//Something went wrong Try again soon :) // retry 10 seconds later if(!job.attrs.data.failCount){ |
|
1 |
job.attrs.data.failCount= 1; |
|
1 2 |
} else{ |
|
1 |
job.attrs.data.failCount++; |
|
1 |
} |
|
1 2 3 |
job.attrs.data.failReason = err.response.data.message; var slack_data; if(job.attrs.data.failCount<5){ |
|
1 |
job.attrs.nextRunAt = |
|
1 2 3 |
moment().add(60000,'milliseconds').toDate(); } else{ |
|
1 2 3 |
//failed delete job.attrs.failCount; job.attrs.data.failed = true; |
|
1 |
} |
|
1 2 |
await done(); await job.save(); |
|
1 |
} |
Agenda relies on MongoDB for its datastore. Our MongoDB handles a large amount of our data’s info requests and sometimes experiences outages. When this happens, our Agenda Instance is incapable of gracefully closing.
Agenda locks jobs that it is working on, and when Mongo goes down, it has no way of unlocking those jobs. So It’s important for on restart we find any jobs stuck in ‘Locked-Limbo’ and unlock them.
Because the execution time of our jobs is so short, it’s typically safe to assume the job was in the queue but not actually started. We use Forever and Monit to daemonize and monitor our Agenda Instance. We receive Slack notifications whenever Agenda goes down.
Closing Thoughts on Agenda
After our experience, we can say Crossbrowsertesting recommends Agenda.
At first, we were wary of using Agenda considering it’s seemingly dead status on GitHub concerned us. Ultimately, now that Agenda is a part of our stack and in production, we feel confident in our choice. Agenda is a handy and versatile tool for scheduling jobs.