Overview:

This article provides information on how to schedule a huge number of tasks using Node and node-cron. The author discusses the limits of scheduling jobs with Cron and Node and how to overcome them. They explain why they chose to use Cron over a database-based schedule, as it is already standardized and flexible. They also mention the benefits of using a front-end expression builder to allow users to create Cron expressions. The article also addresses what to do if something goes wrong and how to ensure that Cron is exportable to a new schedule system. Finally, the author provides benchmarks to test the basic performance of Cron and determine the number of scheduled Cron jobs that can exist. By following this guide, developers can learn how to efficiently schedule tasks with Node and node-cron.

The best way to schedule a huge number of tasks with Node and node-cron.

Part 1: Finding the limits of scheduling jobs with Cron and Node.

Hopefully, this will be informational and not turn out to be the ramblings of a mad developer, it will either serve as a great case study or as a warning to others.

I’ll try and include any code for the developers out there wanting to skim read, and follow along to the conclusion.

Context

I have a project that has the requirements of having jobs being scheduled and executed.

The jobs are simple (let’s say sending an email), but there will be lots of them (say about 100-1000+ per user)

The schedule of a job will be repeated each year or just a single year and can be something like

  • Execute job every x minutes/hours
  • Execute a job every 2nd Friday during September, November, and December
  • Execute job every x minutes during July

Why use Cron over a database based schedule?

I did a quick check around for how you should store schedules in a database, and boy did I learn a lot. I learned that there’s no real standard and your best bet is to see how Microsoft does it in there SQL server and copy that.

All the stuff I found looked like I would have a lot of work to do and a lot to maintain. Keeping capacity and budget in mind, it seemed like something that could swallow a lot of time up, then take time to maintain.

I would prefer to find an easier way to do it than trying to create the database structures and its supporting stored procedure.

Cron is already standardised and flexible!

Cron has an already established format for representing a schedule that can hold a complex schedule. however node-cron isn’t as flexible as full cron. a few of the notations (such as L for last) are not supported, meaning you need to work out alternatives or know that it isn’t supported.

10:15 AM on the last Friday of every month?

Cron: “0 15 10 ? * 6L” or node-Cron “0 15 10 25-31 * 6”

Every minute starting at 2 PM and ending at 2:05 PM, every day? yep can do that too with *“0 0-5 14 * * ”

Every November 11th at 11:11 AM? that too “0 11 11 11 11”

So on and so forth, as you can see pretty much anything can be expressed as a Cron expression. So it meets the flexibility I need in a standardized way.

Cron expression builder

Because Cron has been around for some time now ( May 1975 ) people have already built tools around it, including a front end expression builder. (React)

https://github.com/one-more/react-cron-builder#readme

https://github.com/sojinantony01/react-cron-generator

This means within a short amount of time you can have a front end that allows users to create Cron expressions that your backend reads and run from. quicker time to value win!

What if something does go wrong!?

When you’re about to make a potentially terrible design decision, think of an exit plan!

Say we run with this idea and further down the line when we have millions of scheduled CRON jobs (leaving out the obvious load balancing issues), a new requirement comes in and now Cron is no longer suitable. what now?

Would Cron be exportable to whatever new schedule system? possibly? certainly more likely than if I hacked my own schedule system up myself!

I would imagine something that would run through each job, parse the Cron expression into the new format

Cron performance

So the basic performance of Cron, does that meet our requirements? Lets get some benchmarks!

  • How many scheduled Cron jobs can exist, and whats the performance draw of them existing. currently ignoring the load of what executes inside the cron job)..
  • How long does it take to load “x” cron jobs? considering cron jobs are loaded on server start (maybe there’s a persistence solution?), then each time the server starts it will take x amount of time. if the time is x hours, then… bad. To test this, let’s create a quick project > cd into it > npm init

Install node-cron with the line below

npm install node-cron

Create a index.js file and stick the below into it

const cron = require ("node-cron");
cron.schedule("*/5 * * * * *", async () => {
  console.log("cron task executed")
});

Update our package.json to include a start script, the package.json should look something like.

{
  "name": "crontests",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "node index.js"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "node-cron": "^2.0.3"
  }
}

Give it a go with the below in the console.

npm run start

every 5 seconds you should see “cron task executed” pop into your console.

Because I want to benchmark this in some sort of formal manner so I can at least try to extrapolate out the limits, I have added the following to the scripts

"start-1024": "node --max-old-space-size=1024 index.js",
"start-256": "node --max-old-space-size=256 index.js",
"start-16": "node --max-old-space-size=16 index.js"

All this does is start it with limited memory, that limit being whatever value (megabytes) that are assigned to –max-old-space-size. I’m going with a gig, it’s a round number, it gives the server a chance, and is small enough for the horizontal scale

Lets get our first benchmark

Changing our index.js to the below will give us some ability to test performance (roughly)

const cron = require ("node-cron");

console.time('loadTime')
const CronJobsToCreate = 10000

const logCronCreation = (i) => {
  console.log("message from:", i)
}

for (let i = 0; i < CronJobsToCreate; i++) {
  console.log("creating job:", i);
  cron.schedule("0 15 10 * 12 5", (i) => logCronCreation);
}
console.timeEnd('loadTime')

So

  • npm run start-1024: 4820.223ms (1gb)
  • npm run start-256: 5453.423ms (256mb)
  • npm run start-16: n/a (16mb) (dies at 6480 cron jobs)

So 16mb could only hold 6480 cron jobs (roughly, it varied time to time). Some quick maths means that if this is linear then 256mb limit would be 103,680‬ and 1 gb would be < 500,000… Give it a go, let’s raise the number of cron jobs created. (adjust CronJobsToCreate variable)

const CronJobsToCreate = 120000
npm run start-256

So a quick go with setting the limit to 120000 on a 256mb limit resulted in 103, 968 jobs created before it died. which unsurprisingly shows a linear scale.

so 1gb should be 414,720

const CronJobsToCreate = 500000
npm run start-1024

Created job 424,058 then ran out of memory.

Problem 1 with running a huge amount of Cron jobs: it takes memory to hold the Cron job… duh!

For most this wouldn’t matter, I mean, who would look at 400,000+ cron jobs taking a single GB and say “That doesn’t the performance I need”… I don’t know what I expected in all honesty.

I guess my main problem with this is that for my scenario, these jobs could be set up to fire once a year, which means there could be jobs holding up memory for a year fire once and then hold up memory for a year again…. doesn’t seem very efficient.

On top of this, its memory just to hold the job, we would also need memory to execute them.

So based on the above, and taking into account that its a function that console logs, so not a complex setup, to get a million schedules in memory I would need about 2.5gb to hold them in memory. and this is how it directly scale from then onwards.

How long would it take to load jobs into memory?

This is obviously affected by the performance of the server

  • 10,000 loadTime: 6.452 seconds
  • 100,000 loadTime: 59.610 seconds
  • 200,000 loadTime: 123.952 seconds
  • 1,000,000 loadTime: 631.843 seconds

So basically for a million jobs, we would be looking at roughly 10 minute load time.

Conclusion for using Cron

I would think for the 99% of projects that require some sort of scheduling that node-cron probably fits the bill, and would work wonders.

for use to schedule a million jobs, probably not ideal, you would need 2.5gb of memory just to hold them in memory, start-up of the server would be 10 minutes or so, where ever you store the jobs to be collected and loaded would need to retrieve a million records too which could be problematic.

So my last thought on this part: is there a way we can support node-cron with other tech?

The best way to schedule a huge number of tasks with node-cron and a storage system.

What if we support Cron with another piece of tech, we can still keep the Cron expressions, but use a database system to store the scheduled items.

The benefits I would hope to attain from this are:

  • less memory to “hold” the tasks, as the database will essentially be the storage.
  • Quicker start-up times as the server doesn’t need to fetch the tasks and load them into memory.
  • If there’s downtime, then there is a lower chance of tasks being missed as the database keeps track of what tasks are due to run.
  • For tasks that are a one-time thing, then we will not be using resources holding them once they have been executed.
  • possibility to expand on the flexibility of node-cron
  • Moving tasks from the memory of a server to a database should mean better options when we are looking at a horizontal scale (if we don’t immediately end up in AWS, Azure, etc)

The potential downsides?

  • potentially a high database load as we pull x records from a table, and then potentially insert x times back into the table if the jobs are to be repeated which is highly likely
  • more time invested upfront, The previous concept pretty much works now. by going forward we are adding a database and all its infrastructure as well as the code that interfaces with it.

The core of this is storing a record in the database which using Cron expressions and cron-parser we also keep the “next run time”. We then collect tasks to be executed from the database based on the “next run time”, run them, parse the Cron expression to find the next date, and store it back to the database.