Getting 429 from the check-in endpoint in our Kubernetes cluster

Ninety jobs firing at the top of the minute would exceed the budget instantly. Things you can do today: 1. Stagger your schedules. Rather than everything at minute zero, spread jobs across the minute or across several minutes. In Kubernetes this usually means varying the cron expression rather than the code. This is the single most effective fix. 2. Add a small random delay before the ping in jobs you cannot reschedule. Even a few seconds of jitter spreads the load enough. 3. Retry the ping on failure. Curl can do this with a retry flag, so a rejected request is re-attempted a moment later rather than lost. Since a failed ping means the heartbeat may be marked overdue, this is worth adding regardless. 4. If you have several egress addresses available, distributing jobs across them raises the effective ceiling, though this is more of a workaround than a fix. I have also passed this to our development team as feedback, because keying this limit by token rather than by IP would be the correct behaviour for exactly your scenario. In the meantime, staggering should resolve it completely at your scale. For reference the other public endpoint, the alert trigger, is limited to 30 per minute per IP, so the same consideration applies if you fire alerts from the same cluster.