Is Big Data Giving You Grief? Part 4: Depression

Posted August 4, 2014 by Walt_Maguire

Programmer

“”The problem is too big.  How can we possibly address it?””

Continuing the five part series which explores how organizations coping with big data often go through a process that closely resembles grief, this segment addresses the point at which the organization finally grasps the reality of big data and realizes the magnitude of the opportunity and challenge… …and gets depressed about the reality of it.

Having seen this more than once, I’’ve observed a few ways this shows in an organization.  Here are the most common reactions.

It’’s too big

This reaction makes sense. After all, as much as we in the industry say that “”big data”” is more than big and describe it with a laundry list of varying attributes, we all agree that it’s big. It represents addressing data at a scale never before attempted by most organizations. It represents analytic abilities perhaps never done before—and a capability pivot towards being an analytics-driven company. And it may represent opportunities that are so big they appear to be nebulous: “”If I capture ten thousand times as much data about my product, how does that translate into value?  Does that mean I’’ll sell ten thousand times as many widgets?  How do I quantify the payoff?””

It may be challenging just to get a handle on the costs of a big data program for reasons mentioned in earlier parts of this series, much less the potential payoff.  This can make for a very challenging return-on-investment calculation.

We’’re not ready

I believe I may have heard this particular form of worry more than anything else.  The infrastructure isn’’t ready, the people aren’’t ready to build big data applications, the business isn’’t ready to consume the new data, and so on.  And, in fact, the company may not be prepared to size the big data effort because the team may not have the know-how for the ROI calculation (see above).  Also, the executive leadership may be unprepared to make a strategic wager on the program because of the uncertainty around the risks and benefits.

This can seem like a true show-stopper.  It’’s not easy to change an organization.  Skills and technologies may not appear to be aligned with big data needs.  The various lines of business may not realize the ways they can improve or revolutionize their business.  The leadership team may be unaccustomed to making big bets on unproven technologies, or may believe that big data is a fad and will pass.

We’’re too late

I hear this a lot too.  Everywhere a business turns today there’’s a story about how someone has transformed their business, created new markets, broken old barriers, etc. It’’s easy to believe that all the opportunity is gone—that there’s no more benefit to tackling big data because it’s already been done.  It’’s also easy to believe that it would be impossible to “”catch up”” with others because of all the time and effort required.

While this can be an intimidating belief, it can also be hard to characterize accurately.  After all, do you think your competitors will announce that the big data project they recently publicized in the media is a year late and $10M USD over budget?  Instead, they’’ll play it up as if it’s a runaway success.  Vendors help this along too—who wouldn’’t want to tout that their product helped a company?

So the saying goes—”The darkest hour is just before the dawn.””  Sage words written long before computers that apply to this situation.  But this is actually a positive place to be, because once a team has moved through anger, denial, bargaining , and into depression, it’’s ready to come to terms with the situation and make an action plan to move forward.  I’’ll discuss that next week in the final part of this series: acceptance.

Next week the series concludes with…… acceptance.  “”We can do this.””