November 12, 2021

triage

whitesands.jpg Minolta X-570, Ektachrome 100 - Whitesands National Park - July 10 2021

What is this, a rant?

Say you have, I don’t know, a 3 person team where one of which is ideally a leader of some sort. In a System Reliability context, this team will perhaps handle a wide variety of tasks for both the now and the future. This is to say, tasks encompassing development, prevention, maintenance, but also developer support, and incident response. In an ideal world, the former grouping resolves the latter. This never happens. Many dependencies are out of your control (i.e. cloud providers) and there are only so many unknowns that can be known unknowns. Indeed, what you can control, is the level of risk accepted and the process for filtering these risks into tasks handled by your sad, small team.

If filtering fails to control for an acceptable level of risk, which is to say the person with the widest subject matter knowledge, technical ability, and bureaucratic leverage (a leader, maybe) fails to provide necessary judgment upon the tasks handed to their Individual Contributors (IC’s), assumptions will be made that could violate a risk profile agreed to. Assumptions are no issue when the ambiguity of a task lends itself to discovery in an environment for discovery. This defines low risk. If tasks reach a level of assumption that surpass acceptable risk in a production environment, and lead to lapses in procedure, the reliability team has failed on a fundamental level. Service Level Agreement’s will be violated and contracts will be dropped.

A hierarchical team structure should control for these things by default, but if you find that your IC’s are delegating tasks without process, without context, and without filtering, a risky game of assumptions is being played. This is not to say that small, loose teams are unfit for reliability work, but that mismatched and confused team cultures have to be addressed.

Computers are mathematical, rigid machines, but what they produce is a social service, for other humans, and I think this has to be accounted for on both ends of the line.

do you like links to things you were not planning on thinking about today?

if you read one thing from here, let it be this


powering through