Founders Should Be On Call

When we first launched the Luxor mining pool and it started to get real scale, the database went down almost every night.

It wasn’t one bug. We were pushing a lot of data through a system that wasn’t built for it, and a handful of inefficient Postgres queries would start fighting over the same tables. One lock would cause another, and by the end the whole thing would seize up. Then we’d get paged, find the query, fix it, and go back to sleep. The next night a different table would lock, for a slightly different reason.

It was exhausting. Part of it was just inexperience. We didn’t yet know what a database under that kind of load looks like, so we learned it the slow way: bug by bug, lock by lock, one query at a time, until the system stopped falling over.

A report would have told me how many minutes each outage lasted and how many miners it affected. Every word would have been true, and none of it would have taught me what those nights did.

Most founders stop being on call as soon as they can afford to. It makes sense on paper. Your time is expensive, the engineers know the system better, and you have a company to run. So the pager goes to the team, and the founder learns about problems in the morning, summarized.

I think that’s a mistake, and not for the reason people usually give. It isn’t about solidarity. It’s about information.

A dashboard shows you what someone decided to measure. An outage shows you what you didn’t think to measure. When something breaks at night, you see the system the way it actually is: which parts are fragile, which alerts are noise, and which customers notice first. Our nightly locks didn’t show up in any metric we were watching. They only showed up when we were the ones holding the problem.

You also learn what your product feels like when it fails, and that’s most of what a customer remembers. People forget the hundred days everything worked. They remember the night it didn’t, and how long it took you to tell them.

There’s a second effect. When the founder is on call, the system gets fixed differently. Every page that wakes you up becomes a reason to make sure it never wakes anyone up again. We didn’t fix our database problem with one big rewrite. We fixed it because we were tired of being woken up, so every night we made it a little harder for the same thing to happen twice. Founders who never get paged tend to underinvest in exactly this kind of boring, incremental work, and then wonder why their best engineers burn out.

This doesn’t mean a founder should be on call forever. You’ll become the weakest person in the rotation pretty quickly, and the team shouldn’t have to wait for you. The point isn’t to be the hero. It’s to stay close enough to the real system that you can’t fool yourself about it.

If you’re starting a company now, take a shift. Read every page that fires, even the ones you can’t fix. You’ll learn more about your product in a month of bad nights than in a year of dashboards.

A founder who has never been paged doesn’t really know their product. They know a summary of it.