Small Data

I was helping David Andrade prepare for his Tableau Conference presentation today.  We were talking about the various data sources that we integrate with Tableau.

And we came across the idea of Small Data.  The hype is focusing on Big Data, but in many aspects, smaller data is way more interesting and important.  David and I have developed a digital data dashboard and receiving display ad click summaries from our client's ad agency comprises only 15 lines of spreadsheet rows and no more than 10 columns, but each cell is critically important to the dashboard and our resulting strategic recommendataions.

Compare that to our large clients' web log data - we could certainly miss a few rows of that data feed and not really impact the strategic analysis.

So, is small data big?

You Can Take a Horse to Water

I mostly wear a heart rate monitor when I work out.  I do it so I don't sandbag myself.  It is a way old Garmin unit, and once it is full, I download it into my Garmin desktop program (pre-app days) and I look it and then forget it.

I work with a guy who has one of those Under Armour bands.  I see them on other folks as well.  HBR has a quick blog post about this trend.

We're creating our own little Big Data networks on our heart rates, sleeping patterns, blood chemistry, and all sorts of great stuff, but will the data from these 485,000,000 auto-analytics devices by 2018 (in the HBR article) actually be used?

Like all BI projects, your reports need to support a decision.  For me, it is stop lolly-gagging, but that is a moment in time.  I don't use the data again.  I don't have a good BI application to compare and contrast my workouts and to make that useable.  Do they exist?  I'm sure Olympic and professional athletes use them - but for this middle-aged, balding desk jockey?  And even if I paid for the best BI auto-analytics tool and I can increase my cycling efficiency by 5% - so what?

Just because we can collect data and create reports - it doesn't mean we should.  Make sure to align reports with a true decision making need or you'll run the risk of wasting money on something no one will use.

Now, if you will excuse me, I need to update my food diary app...

NSA Big Data is Really a BI Problem

Currently, we are hearing a multitude of thoughts regarding the recently exposed NSA Big Data program.  The current issue of BusinessWeek has a rational essay on it.

From a BI point of view, the essay doesn't say much that we already don't know - the struggle is in collecting data, organizing it, and relating it to other "indexes."  These are basic Big Data/data warehouse concepts.

What I found most thought provoking is this sentence - that the NSA, "if it needs to, it can actively cross the between your statistical self and your real, physical self."  In its purest sense, that's the point of BI - creating a meaningful, actionable, reliable, data-driven proxy that crosses to reality and can be used to influence decision making.

The NSA BI program is built for just a handful of uses (thank goodness) at the beginning of the decision making journey and our BI projects are built for regular, decision-making.  One measure of BI success is decisions per quantity of data collected.  On this basis, our projects have a higher success rate, but let's check this out:

My most popular BI environment has under 200 global users of 24 dashboards - and the data size is about 50 MB.  No matter how you do the success rate ratio it'll be much better than the 18,000 NSA annual uses (what I remember reading someplace) of what has to be a gazillion gigabytes.  Sounds like a negative BI ROI - obviously the NSA would disagree, but then again, their success (no terrorist attack) completely changes the equation.

But it does lead to a BI question for you - are your BI projects getting used enough?  How do you monitor it?  When would you decide to stop a program because it is not being used?

Transparency

I enjoy reading Kevin Coupe's Morning News Beat - he provides insights into the retail industry, particularly grocery. He also has a column in Forbes discussing transparency in the food industry in relations to food labeling GMO products.

From a food point of view, I agree with him, but he makes me think about transparency in BI.  Too many report users have no idea where the reports come from or how definitions are made.  Questions exist about application of the metrics and incorporating report insights in decision making.

The standard answer is documentation - here's a data dictionary, here's the technical requirements, here's a huge pdf that answers all your questions (if you can find them).  Honestly, the true standard answer is nothing other than, "Oh, that's Fred's report - go ask him."

For my projects, I have a governance committee - a group of end-users who help figure out what's next for the reports to keep them actionable, relevant, and trustworthy.  But that transparency only goes so far.  Some of my clients even have user group meetings, which is fantastic - if people attend.

Kevin says:

"Don’t do it, and I have to ask why...  But keep fighting the calls for labeling, and you create uncertainty and mistrust."

He has a great point.  In today's world, in which content is ubiquitous, the absence of BI transparency in a single report is curious.  In previous work, Kevin has mentioned that food labels exist with QR codes in which you can scan it and see videos and explanation of the farms where the food came from.

Why can't we do the same thing for a report?  How can we do it with our existing tools? Why does it have to be so hard?  Is BI fighting the calls for report transparency?

DIY

I spent the past weekend with a friend who owns a farm in southeastern Washington State.  I helped him replace the main electric line into his old barn.  After toiling over a keyboard, it felt great to get out and be phyiscally productive; the satisfaction of physical work is different for me than just moving electronic bits around.

I have done my fair share of wiring, but I've never played with the main lines before.  I'm certainly not going to become a lineman, but I can say I did it.

As I drove home, I reflected on my accomplishment and thought of it relative to my client's desires to want to create their own BI reports.  I have always understood the desire to be self-sufficient and the need to manage costs, but I think I have underappreciated the desire to create a useful report.  Of course, I can't personally direct all my clients as my friend directed me, but I can certainly provide an environment in which they can be successful on their own:

• Data - have all the metrics pre-aggregated and ready for any potential use
• Definitions - ensure everything can be explained easily with either a click or a document
• Training - take time to create meaningful scenarios when I show them how the system works
• Examples - have ample existing reports for them to reverse-engineer
• Support - celebrate their successes!

Carl

555 Dashboards

I've been working on a quick hit dashboard product for sometime now and we finally have the marketing material ready for it - here's the 2 page handout that'll be handed out at the DMA show soon.

I've completed one version of this 555 dashboards (its name is inspired by Herman Cain's 999 plan) in which we:

• Work with 5 stakeholders to create a project plan
• Incorporate 5 datafeeds that will provide the basis for the:
• 5 interactive dashboards to help you make better business decisions

Check it out and let me know if you'd like to talk further about it.

Carl

Tool Harmony

I've been working with several clients lately to help them decide which tool they should select for their BI project.  Historically, it has typically been a single choice, but lately, the discussions have focused on a suite of tools.

I've always been a "right tool for the job" guy - I use SQL, SAS, MapInfo, Tableau, Excel, PowerPoint, or whatever it takes to get the job done.  Often, I will rely on multiple tools for a single task, but I have never sat down and thoughtfully considered how I can get tools to live in harmony.

Here's my first go at it - my top 10 tool harmony ideas:

1. Cost
• If I have to buy an additional license, then that's not harmonious.
2. New, New Thing
• Unless you want to support it yourself for the long haul, avoid any new tool
3. Naming
• As you jump from tool to tool, can you follow the data path?  Is the same variable called the same thing in all tools?
4. Value Add Trail
• The whole point of using multiple tools is to do something in one tool that is superior to another.
• If a tool feeds another tool, then clarify the value add trail - ensure that you remember where one tool starts and ends.
• Use the other tool name in a variable name:
• MapInfo_DMA_Response_Rate
• SAS_Response_Model_Score
5. Calculate as Much as Possible at the Source
• There's no sense in making a custom calculation that does the same thing in all tools - that's not efficient and runs the risk of having them not coordinated.
6. If using a database, use a view
• Sourcing from a view is an easy way to keep multiple tools coordinated.  You can insert custom calculations, rename variables, and coordinate multi-tool filters (where clause).
• It also provides a strong baseline that anyone can follow.
7. Standardize Colors
• Make sure blue and red have the same visual metaphor across all tools.
8. Create a creative template
• Ensure fonts, design colors, logos, etc. are uniformly applied.
9. QA
• Check the data inputs - are the row numbers and a total sum of a handful of columns foot between each tool?
• Check the calculations - is response rate consistently calculated?
10. Don't over do it
• Really, you can do it all in one tool.  Check Google - someone has done it before.

Alright - what else am I missing?

Carl