How do you catch silent data corruption bugs early?

Visual bugs or broken buttons are easy to spot right away, but silent failures are terrifying because everything looks completely normal on the dashboard. Our scariest silent bug happened when a background worker quietly stopped processing webhook retries due to a silent timeout exception leaving hundreds of user orders stuck in limbo for days without any logs throwing an alert. We ended up setting up an external health ping just to make sure those queues are actually moving every hour. What's your go to way for catching background failures before users notice?