Comments on: Three Masquerades of Metrics

By: Ron C. de Weijze

Ron C. de Weijze — Wed, 11 Nov 2009 17:13:02 +0000

I agree that mimetics do degenerate the contents of our interactions and the meaning of our lives. Girard uses it as a measure of civilization: how long can we procrastinate aggression before we offer the scapegoat? However, I believe we all have our precepts and whether we know so or not, all we do is test the truth of them, measured by comparing them to other’s assumptions. Of course, research has shown that many choose those who are most like us to ‘measure’ our feelings (Pettigrew, ‘67), so there is hardly any validity in that and gives rise to nepotism. But many also use systematic skepticism or falsificationism, seeking independent confirmation to find out the truth of their own views and there is nothing wrong with that.

By: kate niederhoffer

kate niederhoffer — Mon, 09 Nov 2009 15:42:41 +0000

Thanks for the pithy comment, Enikao. Per wording and framing, exactly– the more we can rely on objective data, bypassing self-reports (when perceptions are not the goal), the better we can probe ‘the truth.’

Third party effects require ample validity testing, but most importantly an awareness that correlations shouldn’t always be taken at face value.

By: [Enikao]

[Enikao] — Mon, 09 Nov 2009 12:51:52 +0000

As said Mark Twain : “Facts are stubborn things, but statistics are more pliable”.
It’s not about numbers, it’s about what they mean, and the way we collect figures implies a part of teleological choice. The wording of the question or the sentence can highlight differently the numbers, or suggest other hidden data.

More confusing : trying to see correlation where the correlation relies on a third party effect. The sales of sunglasses and ice creams are correlated, but it is false to assess that ice creams are shiny or that sunglasses makes people thirsty…

By: Steve Poppe

Steve Poppe — Fri, 06 Nov 2009 14:04:14 +0000

Nice stuff. I’m a brand planner and what I do is create selling strategies and brand ideas out of consumer data, insights and attitudes. Social media is really helpful, but too much is too much. In my work it’s all about the boil down and deciding what not to say (or act upon). The planner’s brain is the ultimate arbiter. I love your “pronoun” observation.

By: Guest

Guest — Thu, 05 Nov 2009 23:47:50 +0000

Hi Kate,
Its scary, the feeling of insecurity that goes with challenging metrics, and measurements. We all want to hold onto the side of the boat…
All I know, is that the future of measurement is a process of being in the stream,
listening to patterns, going with the flow, and recognizing that control and command,
and hierarchy in decision-making is giving way…What you guys are doing with
the Collaboratory..is saying ok…lets go..learning together, holding hands when
we need to…I am climbing onto my inner tube and will keep reporting in..

By: Chris Hall

Chris Hall — Thu, 05 Nov 2009 16:36:32 +0000

Great analogies. Right on, Kate.

By: Wittkewitz

Wittkewitz — Wed, 04 Nov 2009 15:27:19 +0000

Metrics are somekind archeological way of doing business. It refers to times were strategic planning was the pivotal strategic approach. These olden golden days were people thought one could organize and control a company by balanced scorecards, six sigma and the like. This was embedded into the IT structure by the data warehouse. Both are ways to drive a car by looking into the rear mirror – btw this is the reason for so many crashes that one could think of sueing for damages all the consultants that up to now earn a living by still doing presentations about this excellent way of organizational suicide.

By: Metrics Metrics Metrics — hallicious

Metrics Metrics Metrics — hallicious — Wed, 04 Nov 2009 10:10:40 +0000

[...] Last week I was all over Chris Brogan’s blog, and this week it’s the Dachis Group’s apparently. Today I’m all over Kate Niederhoffer’s post called: The Three Masquerades of Metrics. [...]

By: Kate Niederhoffer

Kate Niederhoffer — Wed, 04 Nov 2009 04:32:09 +0000

You actually touch on another theme I considered including– metric sensitivity. For two important reasons you must be able to impact the metrics selected. As you point out, it’s a compelling form of agency, making employees, for example, feel empowered by their communication and collaboration. From a business POV, a metric that doesn’t fluctuate in response to various programs/ initiatives is not appropriately calibrated. Imagine you were on a treadmill and increasing the level had no effect on your speed; imagine if running faster didn’t increase the rate of calories burned…

By: Chris Hall

Chris Hall — Wed, 04 Nov 2009 04:05:02 +0000

Kate,

I agree that putting the snapshots together into a moving picture is something that we should all care about more intensely. Tying the moving picture back to interactions at or around distinct snapshots along the time line is something else we should try to get our heads around.

I should be able to see the trends AND see how effective/ineffective I am at affecting the trends.

-chris

By: Kate Niederhoffer

Kate Niederhoffer — Wed, 04 Nov 2009 03:33:26 +0000

Right. I would call the former measurement strategy and the latter interpretation. Both are critical. Thanks again for your thoughts.

By: John Lane

John Lane — Tue, 03 Nov 2009 23:22:41 +0000

Oh yeah… and thanks for the quick response!

By: John Lane

John Lane — Tue, 03 Nov 2009 23:21:58 +0000

Again, well said. I reckon there are two types of insight in this regard: the insight to know what to measure (so both machine and human are gathering the right information / striving in the right direction) and the insight that comes from the analysis of the data.

I didn’t mean to imply the tools were not far more advanced and useful than in the not-too-distant past. Rather, that many people put in the situation of “measuring” simply don’t know better than to take basic data at face value (and often tout it as a success), or how to use the tools available to see what those basic numbers (and the deeper ones they don’t even know to measure) really mean.

By: Kate Niederhoffer

Kate Niederhoffer — Tue, 03 Nov 2009 22:43:41 +0000

Thanks for reading it, John. My response to you is: Sometimes… unless you’re counting the wrong things or grouping data in unwarranted ways. Generally yes, raw variables become stronger in “alliances” (as factors), but I think too often the basics are taken from the surface and not a layer deeper. Transitioning from counts to patterns is indeed part human, but it’s also part machine. There are a lot of new tools available for social data (e.g. networks, conversations)– things like social network analysis and text analysis that surface completely different counts and patterns. Believe me, the resulting data is subject to some of the same obstacles mentioned above, in that not everything reported matters, but they do succeed in going a layer deeper to uncover more informal ties and invisible signals. Still, human insight is *vital* to make sense and refine subsequent measurements.

By: Kate Niederhoffer

Kate Niederhoffer — Tue, 03 Nov 2009 22:25:38 +0000

Hello Corey, Yes! Although the lexicon from “design thinking” has become more popular than that of the scientific method, what we’re really talking about is the necessary and oft-missing first step in experimental design. Why are you measuring? What are you trying to achieve? In absence of this approach, people are likely to retro-fit assumptions using the existing buckets, which — as in your thermometer example– may be unrelated, even arbitrary. So exactly, just because you can ‘count’ something, doesn’t mean you should. Furthermore, just because you can count something, doesn’t mean you should push it into an existing mold which makes sense for other data (i.e. traditional media, GRPs, etc.)

By: John Lane

John Lane — Tue, 03 Nov 2009 21:57:33 +0000

This is a thought-provoking, compelling post. Thanks for writing it, Kate.

My takeaway (and addition to Corey’s comment) is this: the counts (basics) are what will build the patterns (complex) that lead to insight. In effect, those basics are what allow us to create hypothesis, and then we have to dig in to find patterns that support or prove the hypothesis wrong.

It’s that point—moving from counts to patterns—where folks tend to get stuck. And, in my opinion, that happens not only because of the shine-effect (as alluded to in the post), but because that’s as far as the analytics tools really take us. It’s up to someone (hopefully us smart marketing/business folks) to dig in and find the insights in the data the tools have helped us collect. And that should also help us better refine what we count the next time.

Thanks again.

@johnvlane

By: Corey

Corey — Tue, 03 Nov 2009 18:23:42 +0000

Hi Kate, masked metrics aside (and no disagreement here that the measures tend to be coarse or superficial much of the time) don’t we technically need a hypothesis to test in order to determine what metrics are relevant to the objective at hand?

It seems to be that metrics are most often offered up in buckets defined by the technologies’ tracking capabilities more than the data needs to confirm/refute an experiment (in social media or otherwise).

I’m no scientist but I seem to recall from high school science (further back than I care to admit) that if you’re testing for surface tension in a liquid you need a tool to measure that. Just happening to have a thermometer in the fluid at the time doesn’t mean temperature matters to the experiment.

Just because we can track certain metrics easily in social media doesn’t make them the metrics that are worth tracking. My 2¢ anyhow.