Background jobs with Pointguard 🏀

I already shared something about my reserve days. I’m working on a Next.js app and we’re using it as a complete full-stack framework. Being a product developer for a while teaches me what's missing in my tech stack of choice, and allowing me to engineer solutions to have solutions that fill the voids.

Our routing is using Next.js, with RSC and Server Actions. We use Prisma for data access and migrations. We use Auth.js (aka Next Auth beta) for authentication. But one thing was still missing that we needed to solve. Background job processing.

Next.js excels in the request/response realm. But some things don’t work well in that paradigm. I generally believe that if something can be pushed to the background, it’s probably better to do so.

In the background, you can have retries, you can offload the work to a cheaper server. You can scale easily. But unlike in Rails and similar frameworks, background jobs is not a solved problem in Next.js. Or, to be exact, it does not have a standard solution.

I already used BullMQ at Wix. I decided not to use it because we don’t use Redis and I didn’t want to introduce a big infra piece for that project. Using PostgreSQL as our job store sounded like a good idea.

I tried my way into using pg-boss. It had good potential. However, it caused massive pain: Next.js compilation was unrelated to my jobs. It didn’t feel like a single app with multiple entry points. The paradigm was too different.

I thought about integrating a product like trigger.dev: it’s an open source project running on Postgres that looks very good. However, this felt like a massive infra addition too with a complex application in order to solve a small-ish problem.

I couldn’t find a small project that will satisfy my needs:

• Open Source and easily self-hostable
• Simple to use and operate
• Built around PostgreSQL primitives
• HTTP as a way of executing tasks

Enter Pointguard.

Pointguard is a simple background job management service. You can think about it as an alternative to Faktory or Trigger. It is built on Rust so it has low resource consumption and ships as a single small binary (or a very small docker container).

As an anecdote, it was so great to see that over time, I got pretty fluent with my Rust development flow. I had no issues with the borrow checker, which is a first for a project that contains async. It's a fairly small one, but still: I had a really great time working on it and see that I'm getting better.

Pointguard works by having a catch-all route in your application which handles the jobs. This allows you to keep building stuff using your favorite framework (Next.js, Remix, Golang or Rust) and in any deployment provider of choice (Vercel, self-hosted): we just need to make sure Pointguard can make a request to your host.

For instance, you can write code like this in your Next.js app in order to define a job:

import { defineJob } from "@pointguard/nextjs";

export const MyJob = defineJob({
  name: "my-job",
  handler: (name: string) => console.log(`Hello, ${name}!`),
});

Then you can create the catch-all route using the createHandler helper:

import { createHandler } from "@pointguard/nextjs";
import { MyJob } from "./my-job";

export const POST = createHandler({ jobs: [MyJob] });

Then, you can enqueue tasks and it'll just work:

export default function Home() {
  return (
    <form
      action={async (data: FormData) => {
        "use server";
        // we can enqueue the job like so!
        await MyJob.enqueue(data.get("name"));
      }}
    >
      <input type="text" name="name" placeholder="your name" />
      <button type="submit">Enqueue</button>
    </form>
  );
}

It feels very native to the modern JavaScript ecosystem.

Out of the box, Pointguard supports the following features:

1. A full OpenAPI schema generated from source so it's always up to date
2. A very basic Admin UI (that will get better, pinky promise)
   1. Currently enqueued tasks
   2. Finished tasks and their status
3. Single binary deployment that contains the migrations (pointguard serve --migrate)
4. Can work on different database schemas to prevent collisions,
5. Delayed tasks: don't run the job immediately
6. Deduplicating tasks using a unique name: passing a name to a task will cause it to be unique in the queue. Meaning that it will only run once and you can use it as a way of throttling requests.
7. Retries: tasks can be retried up to a given amount of times

Many things are yet to be implemented:

1. Different engines, and specifically SQLite support. It might not be the best for a huge production system (no locks, etc), but could be great for small projects, for tests and local env as it can use a simple file system or be in-memory.
2. Cron jobs: scheduled recurring tasks
3. A more thorough admin dashboard with real-time events and summary of the system
4. Progress reporting for long running tasks
5. Complex workflows (jobs that invoke other jobs, maybe call jobs in parallel and then reduce over the results)

I think this project has a great potential to be a core piece in my upcoming projects. The more we solve that can help us focus on the product we want to ship, the better.