In this day and age of Big Data, where "data is king" and AI tempts the world with its potential, everyone wants to be data-driven.

Data people may often seem like wizards — after all, they are able to pull information out from what appears to be a crystal ball.

The truth, however, is that the path from raw data to actionable insight is anything but pretty.

To understand why, let's look within the BI department to see what really happens when a business decision maker asks a data question.

A typical analytics process is not so straightforward

A data process begins when a business user has a data question. The business user goes to a data analyst and expects him to return with a result.

In the business user's head, this is what she imagines happens:

In reality, however, what happens behind the scene is ... well, a huge mess:

Your mysterious data analyst is anything but a wizard. Just like everyone else in the industry, he is caught up in conflicts with his client (business user) and the people who are meant to support him (data engineers, or even just plain engineers):

The flow chart above is a little tongue-in-cheek, but it contains the core frustrations that people in a BI department. It's worth asking ourselves:

Why are business users, analysts, and engineers so at odds with each other?

The data department's job is to give the business the insights it needs to make better business decisions. In theory, everyone is supposed to row in the same direction. And yet, we find ourselves in a situation where the three roles of a BI team are constantly frustrated with each other.

Let's take a closer look at these frustrations:

Business User vs Data Analyst

The first set of frustrations exist between the data analyst and the business user:

Communication between business users and data analysts leave much to be desired.

Business users rarely convey the full context of their requests, simply because they do not know what is necessary for the analyst to understand them. When they aren't sure of what they want, they put out dozens of requirement changes, leaving the analyst confused and frustrated.

Without proper understanding of the business, data analysts often feel like an English-to-SQL translator, and thus have no motivation to produce high-quality results. They are annoyed and stressed by the constant requirement changes. They feel even worse when they are criticised for getting numbers wrong.

At the same time, business users suffer other pains. At Holistics, we've met many brilliant marketers, salespeople and product managers who are confident in their analytical abilities but feel powerless and ineffective because of their reliance on data analysts.

Data Analyst vs Data Engineer

While feeling "terrorised" by business people, data analysts also unknowingly cause distress for the data engineers:

Under pressure from business users, analysts feel restless when data engineers take too much time to respond to their own requests. Analysts may sometimes require the data engineer to give them a simple data pull, or a complex data transformation. While they may have translated the business question into calculations and transformation sequences, it's still difficult to convey everything to the data engineers.

The data engineer, on the other hand, suffers from similar pains of repeated, rushed and meaningless data extraction requests. Again, without proper understanding of the business context, there is no motivation for them to maintain pipelines that appear irrelevant to them.

Data Analyst vs Data Analyst

At the same time, data analysts in the same team or at different departments have their own share of conflicts:

Teams should have proper documentation and easy visibility of previous work if they want to collaborate effectively. All too often, however, data teams struggle to share their knowledge.

Business requests are turned into one-off transformation queries and reports. Replication of previous work means copying-and-pasting SQL, or passing around code snippets in chat channels. Sometimes an analyst spends days coming up with a convoluted query, only to realise a teammate has already devised a very elegant solution some months ago.

New analysts are likely to stumble in their early days on a data team. They will drown in a sea of tables and fields in their team's production database. A previous analyst may know how to navigate that sea, but their precious knowledge is lost forever with them when they walk out of the organization.

In short, in a traditional data team, everyone is frustrated at something. They're frustrated because the processes and tools in a typical data team are set up for the past, an era where businesses had less demand for data.

There has to be a better way

This cycle of pain is so prevalent and so common that it's accepted as the norm. Most of the analysts and engineers we know across multiple verticals seem to think that "This is the way it has always been done. This is the only way that it should be".

We don't think this is true.

We think it's worth asking: what if it doesn't have to be this way?

• What if business users can get most answers to their business questions without going through a data analyst?
• What if data analysts can pull the data they need and transform it into forms they desire ... without having to bother data engineers?
• And what if it's trivially easy for data analysts to collaborate because their work can be well-documented, reused and maintained going forward?

This sounds pretty good, doesn't it?

It would mean speed. Your business can get the answers it needs faster without everyone waiting for one another.

It would mean efficiency. You won't have to continually hire analysts if business users can serve themselves. And you won't have to hire an army of engineers if analysts are responsible for data transformation.

It would mean transparency and easier collaboration. You won't have to reinvent the wheel if you know someone created solid building blocks for your analytics projects. You won't have to waste so much time running around departments just to understand how to calculate a certain metric.

We've talked to our friends, our clients, prospects, veterans as well as members in the data field ... and the same pattern kept showing up. These people have a sense of how data work should be, but they've found no suitable tool to facilitate that. That's when we decided to take matters in our hands.

Enter Holistics

The software development world has suffered the same problems of speed, efficiency, collaboration and knowledge management. However, they have also developed better methods for solving them.

The same thing cannot be said about the now-emerging data world, where everything is still based on the old way, and where most components of the data value stream are disconnected.

At Holistics, we think data analytics is an integrated component of your technical stack, and it should benefit from all the best practices discovered in software development.

But what does this look like in practice?

Holistics makes your data analysts 10x effective

Efficiency comes with ownership. Analysts are uniquely positioned in the data team: they know what data is needed, and they know how to translate business logic into queries. It makes sense, then, that we've put the ability to pull data and write transformations in their hands.

This means that data analysts will have more control over their projects. It also means that your engineers can focus on higher-value tasks, like improve your data infrastructure or your organisation's BI capabilities.

With Holistics's data modelling layer, data analysts can restructure their queries into models for better reusability. They are also able to give meaning to their data, by writing descriptions and annotations within the model itself.

The whole data team will have visibility of everyone's work, and business users will be able to understand what business entities maps to which data. This leads us to our second point ...

Holistics gives business users answers any way they want, any time they want

Giving your business users the ability to self-serve is a game-changer for their productivity.

With Holistics's exploration UI, your sales, marketing, and product team members ... will be able to generate their own complex reports, visualisations, and dashboards with a familiar drag-and-drop interface. They can freely experiment and get insights themselves, in any format they want, with little reliance on data analysts.

Holistics enables a continual learning culture

Knowledge is power. At Holistics, we believe knowledge holds even greater power when everyone has access to it. This belief is embedded right into our platform, where we've built Data Cataloging features to help you easily document your data knowledge.

Descriptions, labels, constraints, and logic tests are coupled with their data models, which reduces the need for separate wikis and tools. Your analytical code is also visible for everyone to view, which enables proper QA processes and allows for collaboration.

This means that your senior analysts have a centralised place to document and pass down their knowledge.

This means that your junior analysts will have much faster onboarding time and can deliver value sooner.

In short, locally captured insights can be shared globally with ease, and that multiplies the benefit of the knowledge itself.

Conclusion

Your data holds much potential. However, in many situations, that potential is not fully realized because of friction and conflict between stakeholders in your data team.

These stakeholders do not collaborate well because the components each of them owns are disconnected from one another. There is no obvious "flow", and there is no robust process that fosters collaboration between those departments. This frustration is born from the old BI world, and it's high time we grew out of it.

At Holistics, we are not just building a new tool. We are advocating a better way for your data operations.

Want to know more? Talk to us.