Archive for the ‘Big Data’ Category

The Broken World of Marketing Data Sciences


Do you know why the world of marketing has a response rate of 2-3% and is happy with it? Even lesser? It is because the world of marketing data and marketing data sciences is broken. And marketers don’t know a way around it.

Here’s what the data looks like typically-

  • Market Research & Panel data – For decades, marketers have been happy dealing with panel data that fulfills what they believe in. Like the huge TG for their product and innovation. So much so, that it takes years of business disasters in a row for a Nokia to understand how things have changed. But Panel Data is at best a coarse approximation. The sampling is flawed 9 out of 10 times. Flawed. I am not even talking about biases. Or, fraudulent data entry.
  • Focus Groups – Less said the better. The FGDs are not representative (by design), the insights are more often than not an accident, and there is a serious dearth of good marketers capable of running half-decent and probing focus groups. What you can usually depend on here, though, is something which is unanimously panned or applauded. Like the Kala Bandar (Black Monkey) of Delhi 6 (the movie).
  • Test & Control Groups – Several practitioners have lost their voices screaming about the need for quality T&CG setup. The only real deal here, to some extent, was the world of database marketing. Unfortunately, DBMs believed a little too much in their potential, and focused less on their improvement areas, a problem that often accompanies success.
  • POS/Warehouse Data – Logged in/ loyalty customers is what we are capable of tracking, they’d tell you. And can they track you across your properties? Their properties? Households? Don’t they have the data already? And what about the other data elements? How good are they? What about that graph model thing?
  • Big Data – Lets mine social media, twitter sentiments, facebook posts, and let us use key opinion leaders and influencers, and viral content. And we will have machine learning for real time recommendations and offers. And… I know. I believe in the possibilities. But do you really understand it? Where does it even begin?

What this has, sequentially, led to, is –

  • Lack of trust in the data – an obvious side effect, won’t you think?
  • Rise of Witch Doctors –People with creative halos, learned expressions, snob behavior, and/or a “heightened sense” of what the consumers want. The primary reason why you’d trust them – because they tell you so. And, because you have no alternate version.
  • A Kodak Moment – All traditional data providers are getting destroyed by Google, Facebook, Inmobi and the likes. These companies have brought “attribution” and measurement to the center of the discussion.

There has been, therefore, a dearth of high quality and effective use of marketing data. Take for instance, Amazon. It’s a data-science influenced company. Yet, what you get to see on the landing page tells you how broken the listings/ recommendation algorithms are. And they’ve been playing with it for a while. Flipkart fares much worse. The others are not even talking about it or worth talking about. Offline retailers have little good data. Brand managers work with incorrect data most of the time (or trade data). Yet, conversations about analytics and big data makes for good headlines.

Nevertheless, there are few philosophical departures, none too easy, to keep pressing forward.

  • Invest in data – Most companies need a multichannel data play, common denominators across platforms, and investments in “data collection”. It’s long, tough and they don’t have the right focus or resources to do it.
  • Improve data quality – Data needs to become a constant companion in every discussion. Across HR, Finance, Legal, Admin, Marketing, Sales, Strategy. Any investment in that direction would, today, mean creating hundreds of cross-walks, resolving inconsistencies, requiring layers of data cleansing, and finally, standardizing the definitions. It would be, for most companies, a journey of at least 12 months if they are smart, ambitious and ruthless. Ruthlessness is the key.
  • Data in the front office – One recurring funny moment in every consultants’ and analytics professionals’ life is walking into a cross-functional meeting, with at least 40% of the attendees reacting at some point– “I’ve never seen this data before” OR “this data is wrong, because I checked last week and it was something else”. That’s because data and analytics sits in the back-office, and reacts to distributed demands and definitions. If it were in the front office, like revenue and profits, it would be reported correctly 8 out of 10 times.
  • CEO’s Mandate – All professionals, in time, will have basic understanding of data, like they (should) have of English (as a proxy for language of the business) and Mathematics. Till then, however, it needs to be driven like a CEO’s mandate. Will that happen?
  • Data Monetization – All of this pain becomes easier if start monetizing the data itself. Data and Analytics are a cost center. They can be a P&L. Especially, if companies stop being scared of how priceless their data is. It is not. The only thing that makes your data priceless is how it is used. Governance around the usage of data might help companies more than sitting on it and hatching it.

You know one thing that would really help? Knowing when you need help and being candid about it! And – I’ll not get into the psychological reasons (I know better than data), infrastructural reason (our technology and analytics team is not ready for it) or business reasons (it is not a priority at this point) why marketers don’t ask for help.

<First published on linkedin>


Analytics in Healthcare: 5 Initiatives

(Image source:

Quite a few of the projects I have done in the last three years have been around the practical use of analytics in healthcare. One could thank the Obama administration and the healthcare reforms for it. But these projects were interesting. They ranged from market expansion strategies for pharma clients to operational effectiveness for healthcare providers to process and risk assessment for payors. And while going through the drills of business development and engagement delivery, there are three things I noticed –

  • Data assets– Healthcare, more often than not, has the best and the worst data. Best because of the nature of trail left behind for everything. Drug trials have data. Your medical history is captured in some shape or form. Medical transcription started some years back, adding more backbone to data. Claims data is analyzed significantly. Your test reports are getting digitized. At the same time, under the impression that most of it is “art”, its not organized data in some very significant parts. The movement from ICD9 to ICD10 – from analysis standpoint, is in progress. That being said, someone who can create a better, robust and standardized taxonomy stands to gain a lot here. Most of what I saw was trash. And I did see a bit.
  • Application – The application of analytics has widened from the traditional clinical trials and drug discovery cycles to everyday business of healthcare. It is one of the better and more expansive adoptor right now. People may remember the Heritage Health competition with good bounty for analytics problem solvers. Which, by the way, followed the ubercool Netflix challenge.
  • Simultaneous change across all entities. Payors, Providers and Pharma guys – All of them are adopting analytics in a wider way almost at the same time. Within banking, for instance, risk was one of the first, database marketing the second. I don’t think operations use analytics as much still. Within retail, companies are still struggling with POS analytics. A lot of the brand guys across industries still think of the art as the driver of sales, while discounting behavioral economics and analytics.

So, what are my top 5 picks for analytics investments in healthcare –

  • Market Restructuring – Does this mean that there should upstream and downstream investments made by the players, yes. Like, Intermountain becoming an integrated payer and provider. Or, maybe, a Pfizer payer-provider-pharma integrated business. or Kaiser? At the least, the payor provider integration is imminent. And when that happens, how will you maximize your benefits, customer benefits, and social benefits, is where analytics should focus. This is where most strategy firms will find opportunities. If someone can productize the approach, awesome!
  • Sales Force Redesign – As an effect of PPACA and Healthcare Act, there has been some consolidation in the industry, and the subsequent standardization of protocols and medical care. Hedging your risks seems to be one of the priorities that hospitals, individual practitioners have had to focus on. There has been an increase in network and institutional affiliation over the last couple of years. Hence, there is going to be a changing buyer design as well. Instead of the traditional sales rep model, newer sales strategies that will focus on horses for courses – such as account management model for network penetration, risk sharing model for adoption and share of wallet, and price war for competitive categories. Pharma companies will need to rethink their sales models, that’s a given. Analytics can do that. ZS traditionally has done that well. I believe Deloitte and PwC will be other companies to watch out for here.
  • Plan design and administration – with the change in customer portfolio , and the commitment to medicare/Medicaid segments,  payors will need to revisit their plans, their pricing, and their claim analytics business. Organizations with strong risk and pricing analytics, underwriters, and actuaries, should customize their play to make some money on this one.
  • From human to inhuman – A fun moment in an engagement is to convince a young analyst why, in certain cases, creating a disincentive for traveling to the hospital (which is a higher cost destination) and opting for low cost channels (telephonic care, medical vans on route, etc.), is not an inhuman way of looking at healthcare. But I guess, finding these more efficient, non-human interventions can help bring down healthcare costs. This will require working very closely with nurses, administrators, doctors. Non-human should become inhuman.
  • From patient to household – Almost all healthcare analysis is conducted at a patient level. For payers, a part of the analysis is subscriber and/or plan level. However, for modular plan designs, better cross-sell, better service administration, the three groups need to evolve to the household level too.


And while on the subject, where is a good healthcare bureau data? There are some that are evolving rapidly and provide a good starting base (SKA, HIMSS, AHD, etc.), but lots of work is needed.



Data Democratization To Analytics Democratization

One of the more recent buzzword in the world of analytics is data democratization, used in the context of public data, govt. data or just organizational data. In layman terms, it means that everyone in the organization should have access to consistent (and preferably correct and comprehensive) information presented to them in easily consumable formats.

I am a huge fan of the possibilities that data democratization would lead to. However, the real tipping point of the world moving to an analytics led marketplace is analytics democratization, and not just data democratization.

Questions before I go further –

  • Do you think it takes a lot of time for your team to get a particular report/analysis/data slice that you need for, say, an urgent meeting/review?
  • Do you think that there are far too many people  involved in this process of getting the data to you? DW/ BI/ Analytics/ Product/ BU/…?
  • If you are an analyst, do you think you get too many ad-hoc requests in a day? Are some of these what you’d call a re-hash of a previous request? And are you scared of telling the requester that you’ve already given it to them albeit in a different format/ it can be derived from the earlier data dump?
  • If you are a business manager -Do you always get that feeling that you do not have enough information? Do you feel that you need something more? Do you also get frustrated with how long it takes to get the analysis that should have been  a cakewalk? Do you also feel that the analyst has just dumped a whole lot of data without stopping to think about the insights?

If your answer to most of the questions is yes – then your organization is most likely miles away from reaping the benefits of data democratization.

Barrier to adoption

Merely by giving access to data, access and consumption of data cannot be guaranteed. For instance, open source and freely available Linux got the appreciation of a lot of people, but remained the geek’s preferred OS. Masses stuck to their user friendly iOS and Windows systems. Even when Linux became more user friendly, the half-baked OS democratization had already created more barriers for the commoners. And the whole idea of migrating from their existing “warranty led OS” to an open source OS is something that masses (and democracy) were not ready for.

The barrier is not data, but understanding

Most stakeholders who do not advocate data driven thinking, and hence analytics, are the ones who have a limited view of what analytics is. Providing data and reports to them never solves the issue of analytics adoption. A closer look at the real leaders in this industry, e.g. CapOne, P&G, Amazon, Walmart, etc. would tell you that “analytics thinking” is ingrained in the way these organizations function and think. In these organizations, analytics has been democratized.

Data by itself is only half the value

Data is almost like a blunt weapon that can be used to club the enemy, or carved into a knife for the kitchen, or a sword for the battlefield. It can also be converted into a home equipment like needle or farm equipment like spade. Without questioning the merit of democratizing data, the real value of data can only be unleashed by making customizable adoption formats. Most end users would still look for flexible thin applications that help them understand data as per their need, or provide enough reports/dashboards that makes the need to be creative redundant. The latter will lead to the generation and maintenance of hundreds of reports that never get used, or whose existence is never ROI justified.

A report is as the reporter does

bad_dash1The number of dashboards and reports that I have seen in my career is an insignificant number. The significant aspect though is the “narrow’ application of most of these reports. Or, the demand-supply gap. Demand of the user, and supply from the report designer. Also, a lot of these reports are so rigid that a change in them requires raising a service request of sorts that goes through three layers of approvals and few weeks of analyst time. The number of consulting engagements that get delivered on the basis of inconsistent reports that different parts of the organizations use is a different story altogether. I believe that once analytics thinking is democratized, the products/reports/dashboards will start evolving to the needs of the masses, have flexibility to adapt on tap. And that’d be an interesting place!


So, even if you’ve democratized data sufficiently, you need people to understand the tools and skills that enable their conversation with data. I have seen too many “smart” people get scared when you throw too much data at them at a high speed. And whoever tells you that you cannot train the whole world what multicollinearity and mixed models are is likely trying to fool you. They need to tell you that you don’t need to.

Let’s aim for a culture where a quant(?) can communicate in a language that business can understand and the business can communicate in a language the quants can understand. And to that extent, organizations need to invest not just in building Big and Bigger Data capabilities, but also an analytics thinking mindset second.

And that’s an area where I see a significant gap. There are few people capable of educating analytics teams, and fewer capable of educating executives.

Image Credits:
1. Infolytics, here

2. Here

3. Here

Betting Big on Big Data?

Are you in a hurry to catch up with all the Big Data news and how it’s going to affect your organization? Are you worried that you’ve missed the social media bus? Or, is someone telling you to move to the cloud? Hadoop? Terabytes and Petabytes of information that you need to process? Real time systems and dashboards? Enterprise mobility solutions? iPads? Micro-segmentation? Platform enabled solutions? 1-to-1 solutions?

If you’re close to a coffee shop, I’d recommend you walk in, get yourself a cup of coffee and sit by the window and relax. Some of us want you to over-react and buy technology, services, analytics, cloud, or something else because that’s what we always do. And that’s what you always do with your customers. Once you’ve identified a buzz word, you want everyone to catch on to it. We want you to geel that you’re missing out on something groundbreaking.

Even though the fundamentals of what is being consumed hasn’t shifted significantly. What do I mean by that?

  • Are you selling a different product?
  • Are you selling it to a new customer?
  • Have the underlying economics changed? Of creating/delivering the product experience?
  • Has the channel changed? Are all your customers shifting online?


And several such basic questions. What exactly are you expecting this investment to deliver for you? And while you’re still noodling over big data and analytics, think about this –

  • Big Data is a very contextual thing. For my mother, big data would mean that all three of us siblings start talking at once. For the HR department of a small organization, it would be sifting through the paperwork required to get everyone a work-permit in the various countries where our team members might be required to work. For the marketing team of another organization it could be the buzz that each of their campaigns is generating across channels, and whether its effectively being tracked
  • Data has always been big, in relative context terms. What has been a challenge is your ability to process this data. Microsoft excel moved from 65k rows to a million plus and continues integrating it with other Microsoft database tools to add more functionality. Likewise, programming interfaces started developing intuitive UIs for tech-incapacitated analysts carry on with their analysis. The tools will evolve to support the needs of the hour. Your need is to evaluate your game and what lies ahead, and not get caught up in what the critics are saying all the time (not to say that you should never listen to them). Don’t always look in the rear view mirror. And don’t always listen to the back-seat driver. Sometime’s you have to deal with the cockpits.
  • The basic rules of engagement have not changed. Analytics should focus on the business. Business should always start with the basics. One of the best managers I worked with had this habit of never recommending anything analytically complex to start with, but rather focusing on a few questions. Consulting firms take a lot of pride in their hypotheses driven approach to problem solving. The same applies here. The analysis/analytics/modeling etc. is a tool to answer the needs of the business. It is not the answer itself. I think it was Einstein who said that if I have finite time to solve a really difficult question, then I’d spend 95% of the time thinking about the right questions to ask, because asking the right question invariably gets you to the right answer.
  • Don’t let it go the IT way. Remember the large scale technology investments in your data warehouses, organization systems, POS implementations, etc. Remember how you realized every three months that something was not being captured by that system? Or, not accurately enough. Almost every client that we have worked with, and this includes the ace financial services firms, insurance firms, retail giants, etc. using analytics heavily, the quality of data has been suspect. For three reasons – the difference between legacy systems and incremental value added infrastructure for specific needs. B) No clear owner. C) Constant back and forth between business and technology on what is required vs what is possible. Analytics is at the same cross road, and combined with the mistakes made on the data quality front, you will soon find yourself repeating your mistakes in a more real time manner.
  • Differentiate between analytical capabilities, technological capabilities and business capabilities. Technology will help you process big data, but you need analysis capabilities to question the changing dimensions of your business. And if any analysis that is not tied to the business it’s impacting, it might as well stay in the analyst’s laptop.

It’s not to say that you should not invest. It’s time you start running a fact based business, if nothing else. Or, as consultants are blamed for – stop pulling insights out of your backside. It’s time you developed the capabilities to do backward and forward looking analysis backed by strong business cases and communicated through effective visualization and quality dashboards. At the same time, don’t get swayed with this large wave of discussions. Its time you pulled out the rocking chair in the attic, sat back, and thought about how to do more with less, and how to get the basic fence in place. Because, it’s also the time when confusing information will hit you at ever increasing speeds.

Big Data – A Hype?

Almost uncanny, this thought coming from Jim Davis, around the same time as this post from me.

Here’s how I like to look at it: High-performance computing is, simply, an enabler. Most importantly, it enables you to get answers faster than before. But – and this is important – high performance computing is only as good as what you’re computing.

No matter how fast you go with summary statistics, you’re never going to get to the future.

At its most basic level, high-performance computing reduces the time dimension.

The Pit, The Fall and The Reichenbach Retreival

The world is abuzz with words like Big Data, Cloud, Efficiency, Real-Time, Analytics, Power, Complex, etc. Most companies are being implored to think about these words. By organizations that are building “capabilities” around it.

Somehow, I think, we are missing the point.

The problem can never be smaller than the tool that solves it. And that to me, has been the bane of the world of analytics/big data/ <fit the new key word> professionals. They create a world where the tool becomes bigger than the end goal it serves. Right from poorly implemented data warehouse solutions, to point-in-time  dashboards that answer a question relevant 5 years back. Do you need to solve your “Big Data” problem? Or, do you need to solve for your business issues?

  • Nevertheless, over close to a decade of trying to be an analytics professional, I have seen more examples of organizations trying to latch on to a fad rather than focus on what they have. Social media is just an example.

You’re walking towards a pit, and you should do it only if you’re fond of them and the treks and the views they offer. Not to fall.

If you are just about venturing into the web world, a free plugin of google analytics can reveal a lot to you, to get you started. Instead of a million dollar investment in sophisticated tools and dashboards. Secondly, a good looking dashboard does not always reveal something additional. It reveals in a palatable way. If you’re focused on what you want, that is, if you’ve figured out your business problem, you don’t always need that sexy solution.

Reminds me of that debate we had about two attractive girls – the difference between the two was that the first was sexier, but the second more marriageable. The second had more suitors while the first evoked desire a lot more often.

  • A lot of these giant ideas fail. You do fall in those pits. Investments that don’t seem to be worth it. Models which stop being predictive unless fresh blood is pumped into them at regular intervals. Technology that becomes obsolete faster than your ability to eat French fries. You will make mistakes, and I guess that’s not that bad an idea, but the least you need to do is be aware of the costs.

For instance, internet was the in-thing. It still is. But the world is already looking at mobile as the next big thing. Right from iOS to Android enabled devices to extremely interesting content delivered in real-time. For every dotcom that succeeded, there are many that got bust because the fundamental idea itself was not thought through.

Microsoft Excel could handle 65k rows of data with great difficulty at some point. Today, it can handle a million rows with lesser amount of difficulty! And people have started talking about billions of data points. However, a lot of beautiful insights are driven sometimes out one unbiased and strong validated hypotheses. Which often, smartly done, does not take a million rows.

  • Last point – it’s difficult to recover from a large investment gone wrong. And I do not always mean monetary investment. Most firms fail to look at executive time investment on half-baked investments as a loss. The opportunity cost of such falls may be significant.

EndNote: The environment today has given us more power in our own hands than most of us are capable of handling. It overwhelms us. We run hither tither and grope around to latch on to the keywords that supposedly enlightened souls are spewing at venomous speeds. No-one is wrong. Yet, the right question for you needs to be the same as it has always been – Is this what the business needs? No. Not wants!

In the Indian mythology, the tale of Bhasmasur is fairly popular. Bhasmasur was a demon who practiced austerity of several years in extreme conditions to please the gods. Lord Shiva appeared in front of him and agreed to grant him a boon in return for his perseverance and dedication. Bhasmasur requested for a power that allowed him to burn anything down to ashes (bhasma) that he’d put his hands on. As the powers, and hence, the tyranny of Bhasmasur reached its pinnacle, the other Gods implored Lord Vishnu to save them. Vishnu took the form of a beautiful danseuse (Mohini Avatar), and tricked Bhasmasur into copying his (or her) dance steps. However, as Mohini put her hand on her own  head while dancing, so did Bhasmasur on his head. And thus, Bhasmasur burnt himself down.  Bhasmasur forgot the reason why he aspired for that powerful hand. The hand that burnt him down.

%d bloggers like this: