Be good-argument-driven, not data-driven
When are data-based arguments appropriate? ATTACH
Sometimes data-based arguments are perfectly acceptable. Let’s have a diagram!
Are all factors that drive your metric well-understood?
Some things you can measure and ~completely understand their causes. For example: server costs.
Can you run an experiment?
that cannot easily be understood, especially when human factors are involved. Take new user growth. You can easily measure how many new users signed up in a period. But can you understand what is behind this number? Was it a new feature you released? Was it a change you made to your pricing? Was it a change a competitor made to its pricing? Did some tech influencer drag you on Twitter? Was it the overhaul you did to the landing page? Was it the SEO project that completed? Can you know? You can’t fully attribute user sign-ups in the same way that you can fully attribute server costs.
If you can collect your data like a scientific experiment, i.e. you can randomly assign users to a control group and a treatment group, and measure a difference between the two groups.
Are you prepared to do some very very fancy statistics?
It is possible to make good arguments based on “field data” collected about events that are occurring without any sort of randomized control, but this is enormously difficult. How do you distinguish correlation from causation? How do you grapple with omitted variable bias? There are techniques: this is “causal inference”, the province of econometrics and data science.
Is data-drivenness a psyop from Google?
Call to action
Data has its place. Metrics are a useful tool for making a certain class of persuasive arguments in certain domains. But they are only a tool for making good arguments. Data is not an end in itself. A weak argument founded on poorly-interpreted data is not better than a well-reasoned argument founded on observation and theory. Stop going all googly-eyed (tee hee) at statistics. Metrics are tempting.