Removing finished jobs

By default, BullMQ will keep all completed and failed jobs. Most likely you will want to limit to sane amounts in order to not fill up your Redis™ database with older jobs.

The workers support two options that can be used to specify the behaviour of completed and failed jobs:

interface WorkerMetadata {
  opts?: {
    removeOnComplete?: {
      count?: number;
      age?: number;
    },
    removeOnFail?: {
      count?: number;
      age?: number;
    }
    // .. more options
  }
  // .. more options
}

The "count" field specifies the maximum number of jobs to keep, whereas the "age" specifies the maximum age in seconds of the jobs to keep. You can use any of these options standalone or combined.

Note that the removal of jobs is lazy. So older jobs will only be removed if new jobs are completed or failed.

Lets update our endpoint so that we only keep the latest 1000 completed jobs, and no failed jobs older than one day but never keep more than 5000:

curl --location 'http://localhost:8080/workers' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer 1234' \
--data '{
    "queue": "my-test-queue",
    "opts": {
      "concurrency": 300,
      "removeOnComplete": {
        "count": 1000
      },
      "removeOnFail": {
        "count": 5000,
        "age": 86400
      }
    },
    "endpoint": {
        "url": "http://mydomain.dev",
        "method": "post"
    }
}'