Implementing webhook delivery retry to improve reliability

When sending webhooks on the internet, it's only a matter of time before something goes wrong and someone doesn't receive the message they wanted. To make your webhooks reliable, you need to detect these errors and retry sending those messages. But this can be a little tricky to get right. With a few tips, you can ensure your webhooks are flowing smoothly.

This post will look at standard retry techniques and how to use them without issues like infinite looping, useless attempts or overloading the server. We will use TypeScript as an example, but the methods will apply to other languages. For brevity, we will leave out webhook seurity.

Basic typescript webhook sending using fetch

First, let's look at a basic example of how we might be sending webhooks with some reliability issues. This example uses the native fetch function to send a JSON webhook. There is some basic error handling that logs if the message wasn't successfully delivered.

function sendWebhook(destinationUrl: string, event: any): void {
  fetch(destinationUrl, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify(event),
  }).catch((err) => console.warn("unexpected error deliverying webhook"));
}

And you might send call

sendWebhook("http://example.com/webhooks", {
  type: "new-order",
  items: ["Sandwich", "Coffee"],
});

This basic solution will work sometimes but has some issues.

Firstly, If there is an issue sending the message, there is no attempt to retry sending the message. Secondly, the error logging doesn't always work. fetch can return HTTP errors as a successful promise with a result having a status code other than 200. You need to check the response.ok property.

Nieve immediate webhook retry

To add retry to the delivery we might first try re-calling

- }).catch((err) => console.warn('unexpected error deliverying webhook'))
+ })
+ .then((response) =>
+    response.ok
+      ? Promise.resolve(response)
+      : Promise.reject(
+          new Error(`Http error! received status: ${response.status}`)
+        )
+ )
+ .catch((err) => {
+    // immediately retry sending
+    console.debug('error delivering webhook, retrying',err)
+    sendWebhook(destinationUrl, event)
+})

Except, by immediately retrying like this, there are some nasty side effects.

• The retry could cause an infinite loop if there is always an error. It could be that the server we are sending to is down and never comes back, or the URL could be wrong. It should have a maximum number of attempts.
• The server might be crashing because it is under too much load, and our retry is too fast, making it worse. It should have an increasing delay between each attempt.

We should keep these situations in mind, and our retry strategy should help them.

Retry with max attempts and increasing delay

To fix the retry causing an infinite loop, we will have a maximum number of times to try, 3.

To improve the chances of the server recovering when under too much load, we will add an increasing delay between attempts. We will start with a 3-second delay, then 6, then 12. The exponential backoff is vital because if many requests fail and they are all retrying with a constant 3-second delay, they would build up and eventually overwhelm the server. By increasing the delay each time, the build-up would be much slower and not overwhelm the server.

const MAX_ATTEMPTS = 3;
const INITIAL_DELAY_MS = 3000;

function sendWebhook(destinationUrl: string, event: any): void {
  sendWithRetry(destinationUrl, event, MAX_ATTEMPTS, INITIAL_DELAY_MS);
}

function sendWithRetry(
  destinationUrl: string,
  event: any,
  maxAttempts: number,
  delayMillis: number
): void {
  attemptDelivery(destinationUrl, event).catch((err) => {
    if (maxAttempts === 0) {
      console.error("error delivering webhook. giving up after retries.");
    } else {
      console.debug("error delivering webhook, retrying after delay");
      setTimeout(
        () =>
          sendWithRetry(
            destinationUrl,
            event,
            maxAttempts - 1,
            delayMillis * 2
          ),
        delayMillis
      );
    }
  });
}

function attemptDelivery(
  destinationUrl: string,
  event: any
): Promise<Response> {
  return fetch(destinationUrl, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify(event),
  }).then((response) =>
    response.ok
      ? Promise.resolve(response)
      : Promise.reject(
          new Error(`Http error! received status: ${response.status}`)
        )
  );
}

Next steps

By implementing a retry like this, we have considerably increased the reliability of our webhook delivery. But there are some further improvements we could make. For example

• Alert the recipient if their server is failing
• Store the webhook delivery attempts in a database so if our server restarts, it can continue.
• Scale out the delivery by putting messages on a queue and using many servers to send.
• Ability to re-send all webhooks in the last week

If you are interested in improving the reliability of your webhooks and don't want to build all of that, take a look at Webhook Wizard.