Designing a LiveOps alerting system to reduce production incidents cover
← Case study · Tactile Games2 min read

Designing a LiveOps alerting system to reduce production incidents

Role
Product Designer
Duration
3 months
Team
Product Manager, CTO, 2 BE Engineers, 1 FE Engineer
Skills
Product Design, Product Strategy, Prototyping
22%fewer incidents
Problem
Things broke in production, and the people who shipped them were the last to know.
What I did
I put failures where creators already work, and gave every one of them an owner.
Outcome
22% fewer production incidents. Failures now surface in minutes, not hours.

Live mobile games rely on continuous updates after release.

At Tactile, updates are managed through an internal LiveOps Dashboard, where teams configure events, experiments, player targeting, and feature rollouts without shipping new app versions.

Creator schedules a reward
!Critical: reward not deliveredRewards service timed out · View issue →
Holiday bonus500 coins to all playersScheduleStatus: live
Backend delivers it
Delivered
Player gets 500 coins
Ideal flow — A creator schedules a reward, the backend delivers it, and the player gets it.

The problem

Failures were detected by backend systems, but not surfaced in the LiveOps Dashboard.

Goal

Surface backend failures to LiveOps creators early so they can act before players are affected.

Discovery

Critical signals lived outside the LiveOps Dashboard

Engineering tools
Monitoring!
Logs!
Slack!
Support tickets!
Player metrics!
LiveOps Dashboard
Hello, DianaNo alerts
Experiments
Game items
Illustration — Every engineering tool flagged the failure. The Dashboard, where creators actually work, showed nothing.

Solution

To close the observability gap, I redesigned the LiveOps Dashboard to surface the right signals, to the right people, at the right time. The solution combines UI surfacing with ownership and subscription logic, so failures are visible, actionable, and clearly owned.

Illustration — A failure reaches its owner in the Dashboard: banner on the homepage, badge in the sidebar.
Show failures where creators workA critical issues banner on the homepage and a red badge on each game in the sidebar.See how ↓Give enough context to actIssue and update lists that say what broke, where, and how it was detected.See how ↓Make every alert someone’s jobEvery resource has an owner who is always alerted. Everyone else opts in.See how ↓

Surface failures where creators already work

I made critical states visible directly in the left-hand sidebar, where they could not be missed.

Sidebar with persistent attention badges for issues and updates
— Sidebar with persistent attention badges for issues and updates
— Avatar-level signal
— Game-level signals

Give enough context to act

Clicking into the Critical Issues or Latest Updates views reveals structured lists with clear context.

— Give enough context to act

Make every alert someone’s responsibility

Ownership and watchers

Alerting only works if responsibility is clear.

I introduced ownership as a first-class concept across LiveOps resources.

— Transferring ownership of a resource
— Adding a watcher to a resource

Notification settings

To balance signal and noise, I designed a subscription-based notification model.

— Make every alert someone’s responsibility

Key tradeoffs

DecisionFromTo

Impact

Faster detectionCritical failures detected in minutes instead of hours
Less escalationTeams relied less on Slack and external monitoring
22% fewer incidentsOne in five production issues caught before full rollout
Trusted workflowAlerting system became part of daily LiveOps work

Source: incident data tracked in Datadog.

What I learned

Showing every update from every game to everyone was a headache. People should only see what they care about.

Disagreeing with the team about keeping critical issues on the homepage felt uncomfortable. It taught me to trust my instincts.

People are busy. Alerts have to find them, not wait for someone to open the Dashboard.