There’s a flaw in the design of how we measure ‘things social’ today.
Like kids with full bags of Halloween candy, we’re impulsively reacting to what can be measured without thinking about what should be measured and how. Our obstacle is our gut. We rely on it at the cost of validity… enduring value for a business.
As far as I can tell, there are three major opportunities that could improve the situation, unlocking the value of conversations and other social interactions. But first, we have to overcome some very basic human tendencies:
1. The ease of counts
We’re human, so ease and simplicity reign. This leads us to prioritize basic counts, things like the volume of tweets and re-tweets. Counts are often good shortcuts, but they masquerade as much as they reveal. We so easily feel pride when our own Tweets rake in the love without realizing the incestuous network centrality: we’re all telling each other the same news. The beckoning opportunity is to analyze the structural patterns now at our fingertips with massive and varied social graphs. Simply put, patterns are better indicators than counts.
2. The shine of the surface
We’re attracted to shiny, superficial objects. For example, when measuring blogs, we think about ‘brand mentions’ or how many comments there are. Our focal point is what’s vivid. The opportunity is the depth: the style with which people discuss brands, the memes that emerge through unique semantic associations. Simple tricks, focusing analysis on slightly more invisible cues like pronoun use tell us how intimate someone is with a brand. Looking at correlations and clusters of words based on frequency and uniqueness reveal the guts of a text: the themes that are really being discussed.
3. The convenience of snapshots
We’re obsessed with snapshots. We fixate on how things are, not how they’re changing or where they’re going. Rarely do we take the time to appreciate time; time series, trends, in-the-moment measurement. Aggregates are typically not as revealing as precise peaks and valleys; asking someone to report what they’ve accomplished at the end of the day will never be as nuanced (or accurate) as asking them in the moment, throughout the day, not to mention collecting the data without asking. Opportunity knocks with real-time, in-situ measures. Measuring momentum and identifying cycles will better highlight the social dynamic. Interactions aren’t static, so their measurements shouldn’t be either.
To move beyond what we’re doing today and embrace what can be done, we need to agree there are wholly different types of data available and measurement techniques involved. We need to abandon some traditional standards and stop forcing social data into shapes and sizes that work for other media measurement. Tomorrow is about patterns, depth, and dynamic metrics.
Speak up if you’re having trouble quantifying the value of social interactions. We’re here to provide solutions to all these gaps so that more meaningful measurement can help our clients capture the value of social business. Tell us about your challenges below and subscribe to this feed to keep the conversation going.
If you’re at Enterprise 2.0 today, stop by my panel on Social Analytics, moderated by my colleague, Jevon MacDonald. Margaret Francis of Scout Labs, Timo Elliott of SAP, and Tim Young of Socialcast will be talking through themes similar to the above (Twitter #e2conf-31).
Hi Kate, masked metrics aside (and no disagreement here that the measures tend to be coarse or superficial much of the time) don’t we technically need a hypothesis to test in order to determine what metrics are relevant to the objective at hand?
It seems to be that metrics are most often offered up in buckets defined by the technologies’ tracking capabilities more than the data needs to confirm/refute an experiment (in social media or otherwise).
I’m no scientist but I seem to recall from high school science (further back than I care to admit) that if you’re testing for surface tension in a liquid you need a tool to measure that. Just happening to have a thermometer in the fluid at the time doesn’t mean temperature matters to the experiment.
Just because we can track certain metrics easily in social media doesn’t make them the metrics that are worth tracking. My 2¢ anyhow.
Hello Corey, Yes! Although the lexicon from “design thinking” has become more popular than that of the scientific method, what we’re really talking about is the necessary and oft-missing first step in experimental design. Why are you measuring? What are you trying to achieve? In absence of this approach, people are likely to retro-fit assumptions using the existing buckets, which — as in your thermometer example– may be unrelated, even arbitrary. So exactly, just because you can ‘count’ something, doesn’t mean you should. Furthermore, just because you can count something, doesn’t mean you should push it into an existing mold which makes sense for other data (i.e. traditional media, GRPs, etc.)
This is a thought-provoking, compelling post. Thanks for writing it, Kate.
My takeaway (and addition to Corey’s comment) is this: the counts (basics) are what will build the patterns (complex) that lead to insight. In effect, those basics are what allow us to create hypothesis, and then we have to dig in to find patterns that support or prove the hypothesis wrong.
It’s that point—moving from counts to patterns—where folks tend to get stuck. And, in my opinion, that happens not only because of the shine-effect (as alluded to in the post), but because that’s as far as the analytics tools really take us. It’s up to someone (hopefully us smart marketing/business folks) to dig in and find the insights in the data the tools have helped us collect. And that should also help us better refine what we count the next time.
Thanks again.
@johnvlane
Thanks for reading it, John. My response to you is: Sometimes… unless you’re counting the wrong things or grouping data in unwarranted ways. Generally yes, raw variables become stronger in “alliances” (as factors), but I think too often the basics are taken from the surface and not a layer deeper. Transitioning from counts to patterns is indeed part human, but it’s also part machine. There are a lot of new tools available for social data (e.g. networks, conversations)– things like social network analysis and text analysis that surface completely different counts and patterns. Believe me, the resulting data is subject to some of the same obstacles mentioned above, in that not everything reported matters, but they do succeed in going a layer deeper to uncover more informal ties and invisible signals. Still, human insight is *vital* to make sense and refine subsequent measurements.
Again, well said. I reckon there are two types of insight in this regard: the insight to know what to measure (so both machine and human are gathering the right information / striving in the right direction) and the insight that comes from the analysis of the data.
I didn’t mean to imply the tools were not far more advanced and useful than in the not-too-distant past. Rather, that many people put in the situation of “measuring” simply don’t know better than to take basic data at face value (and often tout it as a success), or how to use the tools available to see what those basic numbers (and the deeper ones they don’t even know to measure) really mean.
Right. I would call the former measurement strategy and the latter interpretation. Both are critical. Thanks again for your thoughts.
Oh yeah… and thanks for the quick response!
Kate,
I agree that putting the snapshots together into a moving picture is something that we should all care about more intensely. Tying the moving picture back to interactions at or around distinct snapshots along the time line is something else we should try to get our heads around.
I should be able to see the trends AND see how effective/ineffective I am at affecting the trends.
-chris
You actually touch on another theme I considered including– metric sensitivity. For two important reasons you must be able to impact the metrics selected. As you point out, it’s a compelling form of agency, making employees, for example, feel empowered by their communication and collaboration. From a business POV, a metric that doesn’t fluctuate in response to various programs/ initiatives is not appropriately calibrated. Imagine you were on a treadmill and increasing the level had no effect on your speed; imagine if running faster didn’t increase the rate of calories burned…
Great analogies. Right on, Kate.
Metrics are somekind archeological way of doing business. It refers to times were strategic planning was the pivotal strategic approach. These olden golden days were people thought one could organize and control a company by balanced scorecards, six sigma and the like. This was embedded into the IT structure by the data warehouse. Both are ways to drive a car by looking into the rear mirror – btw this is the reason for so many crashes that one could think of sueing for damages all the consultants that up to now earn a living by still doing presentations about this excellent way of organizational suicide.
Hi Kate,
Its scary, the feeling of insecurity that goes with challenging metrics, and measurements. We all want to hold onto the side of the boat…
All I know, is that the future of measurement is a process of being in the stream,
listening to patterns, going with the flow, and recognizing that control and command,
and hierarchy in decision-making is giving way…What you guys are doing with
the Collaboratory..is saying ok…lets go..learning together, holding hands when
we need to…I am climbing onto my inner tube and will keep reporting in..
Nice stuff. I’m a brand planner and what I do is create selling strategies and brand ideas out of consumer data, insights and attitudes. Social media is really helpful, but too much is too much. In my work it’s all about the boil down and deciding what not to say (or act upon). The planner’s brain is the ultimate arbiter. I love your “pronoun” observation.
As said Mark Twain : “Facts are stubborn things, but statistics are more pliable”.
It’s not about numbers, it’s about what they mean, and the way we collect figures implies a part of teleological choice. The wording of the question or the sentence can highlight differently the numbers, or suggest other hidden data.
More confusing : trying to see correlation where the correlation relies on a third party effect. The sales of sunglasses and ice creams are correlated, but it is false to assess that ice creams are shiny or that sunglasses makes people thirsty…
Thanks for the pithy comment, Enikao. Per wording and framing, exactly– the more we can rely on objective data, bypassing self-reports (when perceptions are not the goal), the better we can probe ‘the truth.’
Third party effects require ample validity testing, but most importantly an awareness that correlations shouldn’t always be taken at face value.
I agree that mimetics do degenerate the contents of our interactions and the meaning of our lives. Girard uses it as a measure of civilization: how long can we procrastinate aggression before we offer the scapegoat? However, I believe we all have our precepts and whether we know so or not, all we do is test the truth of them, measured by comparing them to other’s assumptions. Of course, research has shown that many choose those who are most like us to ‘measure’ our feelings (Pettigrew, ’67), so there is hardly any validity in that and gives rise to nepotism. But many also use systematic skepticism or falsificationism, seeking independent confirmation to find out the truth of their own views and there is nothing wrong with that.