Feature Rollout

Since incorporating claude into our daily workflows, our engineering output has multiplied. We can rollout a large architectural change in a week instead of a month. We can implement new features in days, or even a day. We can fix bugs in the background while we focus on the features and architectural changes we are trying to deliver.

Speed of execution is not something we give much thought to anymore. Safety is. We can only rollout new features, changes, and bugfixes if we feel safe to do so. And safety in engineering comes from numbers: metrics. We've invested heavily in making sure we have metrics and observablity logs everywhere. Those metrics are collected and sent to our internal grafana instance where we build dashboards to help us understand the health and state of the platform. We then create alerts on the metrics and send them to slack. With the correct combination of dashboards and alerts, it's easy to say when platform performance is down, and we see the errors before the customer does.

We've also extended our metrics usage to help us rollout new features that might have an impact on platform performance and architectural changes that we know will have an eventual net-positive effect on performance. All architectural changes and performance-affecting features are rolled out behind a backend feature flag using a feature rollout ladder and environment rollout ladder. We measure first, and then follow the ladder. The full ladder isn't always necessary, but there are some key rules: always measure first, always start disabled, use shadow if you need to compare calculations, and follow the ladders.

The full feature rollout ladder is:

• disabled: nothing changes, code is inert on merge
• shadow: do the new calculations, and compare them to the existing one to make sure the code holds up against production data. user sees no difference in results
• enabled: new code is being used
• single write: used when we are changing who or where we write the data, this is the step that would stop writing the old path and only write the new path. the read can still fallback to the old data when needed
• unified: no more fallback, only the new code is used

Each step in the feature rollout ladder is designed to soak (24 hour minimum), measure, and evaluate.

Environment rollout ladder:

• our edge (test) saas and edge private instance first
• the less-heavy saas environment
• the more heavily used saas environment
• the private instances

Shadow is the most important step, because it gives us safety in the results we show to the users when modifying calculations, but it's not always needed - if we're not changing a calculation or modifying how we store data, we don't need to run shadow. Shadow is also not free, which is one of the big reasons to always measure first. Shadow, by its nature, runs an extra calculation on what is usually a hot path: the scan, the scoring pipeline, the user read, and so it is expected that when running in shadow, we take a small performance hit.

The key there is measuring that hit. If p99 pre-code is 150ms, and shadow brings that up to 250ms, that's an acceptable hit. But if shadow brings that performance hit up to 600ms, that's a harder hit to accept. In that case, there are two things that need to happen: improve the performance of the new code, because a hit that large would certainly carry over into enabled and slow down the hot paths, and try the shadow rollout over the weekend. The key there is that when all this is measured with metrics, the engineer doesn't actually need to watch anything, they just need to merge the rollout flag friday morning to let it soak.