Skip to main content

Author: Suraj Chatrath

Data Science: Critical role of a Business Analyst

The last decade has witnessed a sudden spurt in vast amount of data, which enterprises possess, on their customers.

This data is generated from disparate sources like social media, mobile, applications, financial transactions, e-commerce, search and Internet of Things etc. According to an IDC estimate, from 2005 to 2020, the digital universe will grow by a factor of 300, from 130 exabytes to 40,000 exabytes, or 40 trillion gigabytes. From now until 2020, the digital universe will about double every two years. Further, IDC estimates that only a tiny fraction of the digital universe has been explored for analytic value. By 2020, as much as 33% of the digital universe will contain information that might be valuable if analyzed.

Previously, companies were able to analyze relatively smaller set of data through various data mining techniques and tools. However, with the data exploding from multiple sources, data science as a field is quickly emerging. Data science refers to interdisciplinary field about scientific methods, processes, and systems to extract knowledge or insights from data in various forms, structured or unstructured, similar to data mining. The key highlight of the definition is “insights” as it means generating findings which are previously not known from traditional data mining techniques or simple trend and regression analysis. That is why data science as a field is considered to be at an intersection of mathematics, statistics, software and business domain knowledge. One of the simplest examples of usage of data science for any company is its ability to predict customer churn in advance. It can help the company to work on the customer retention instead of focusing only on costlier customer acquisition.

While there are various tools available in the market for data science, but a very critical part for success of any data science initiative is defining the use cases from business standpoint. In most companies, technology teams are adept at understanding the data and running analysis over them. However, field of data science in unique in a way as the teams can sometimes not know what they are looking for in the data as the insights may mostly not be a established fact. Hence the role of business analyst becomes so critical in this field as you would need a person who is fluent not only in the IT domain but also speak in the language of business leaders.


For example, data science business analyst would be expected to convert the business problem statement “Given past performance and current trends, what is the likely outcome of a certain action” to an IT problem statement which means what data needs to be analyzed to arrive at the insights. The data would then be reviewed with the technology team and results would be delivered to the business team in form of insights and data patterns. Business analyst should also be knowledgeable enough to apply various predictive modeling techniques and right model selection for generating insights for the problem at hand.

One would argue, what’s the difference between a business analyst in data science domain compared to a general business analyst. One of the key skills, which differentiate the data science business analyst, is its deep understanding of data as well as industry and functional expertise, which would enable him/her to understand the business context and identify the use case. Data science business analyst is required to have deep business knowledge and understanding of data as depth and breadth of data increases. Further, business analyst would also have to work alongside with business and technical teams and should be comfortable speaking in their language. Mckinsey in its report on “The age of analytics” mentions that while data scientist is a critical skill, companies need “translator” who serves as the link between analytical talent and practical applications to business questions. Mckinsey also highlights that first of the five critical elements for establishing successful data and analytics transformation require use cases which are also defined as source of value. The uses cases should clearly articulate the business need and projected impact.

Another key role of the business analyst for data science assignments is to identify the “optimal” model for the data based use case at hand. The business analyst should possess the knowledge to frame the right hypothesis to test it. Albert Einstein once said, “If I were given one hour to save the world, I would spend 59 minutes defining the problem and one minute solving it.” In simple terms, business analyst puts a framework to the problem solving process. Business Analyst should be able to answer questions like:

  • “What is clustering or regression and when should I use these techniques?”
  • “How do I formulate a hypothesis on my data?”

Most companies are not able to realize the true potential from data science assignments as they rely heavily on data scientists who are very proficient at data preparation, cleaning and modeling or writing software code via tools like python, R but lack the domain knowledge and meaning of data in business context. This is where the business analyst plays a very crucial role of bridging the divide between the business teams and IT department for complex data science assignments.

Minimize Risk With Effective Requirements Gathering

“We can only see a short distance ahead, but we can see plenty there that needs to be done.” – Alan Turing

Requirements gathering is the first step in any software development methodology, be it waterfall, agile, iterative or spiral. The essence of this step is to elicit and comprehensively capture the requirements from an actual customer who uses the product. These requirements could be captured in various forms such as use cases, user stories, wireframes, process models etc.

Most organizations have business analyst or product manager to gather requirements and communicate them to the technical team. Product managers also maintain the constant communication between the teams during the entire product lifecycle. Some companies also appoint third party facilitators.

Requirements gathering is such a critical step that according to IBM System Science Institute’s study, the relative cost of fixing the defect is the lowest if it occurs earlier in the product development cycle1.

Personally speaking, a requirements gathering exercise is a combination of art and science. The art lies in communicating and asking the right questions of the customer and deriving insights from customer’s pain points to lead to the development of detailed requirements. The science lies in capturing the requirements in a very cogent and legible way so that the they are easily understood by the development team.

Some of the steps that follow can be taken to effectively manage the requirements gathering process and aid in addressing issues early on in the product development process. These steps are more relevant from software industry perspective though these can be customized for other industries.

1. Apply design thinking principles to gather requirements

Design thinking is a human-centered approach to innovation that draws from the designer’s toolkit to integrate the needs of people, the possibilities of technology, and the requirements for business success2. Product managers are in a constant struggle to understand, prioritize and meet customer requirements. Design thinking can help them be in constant touch with their customers and design better, meaningful products for their users.

Empathy plays a critical role in the design thinking process where the product manager should step into the shoes of their users and understand their true needs. Applying design-thinking concepts can greatly improve the requirements gathering process resulting in products that rightly address the user’s need.

Rapid prototyping and sharing proof of concepts with the users can avoid surprises after products have been significantly developed.

2. Keep user context in mind

Sometimes product managers are so obsessed with functional requirements that they get the product right but the context wrong. For example, the product is developed on certain platforms, which may not be compatible in customer’s environment. A product compatible with the most recent version of Internet Explorer faces issues in Chrome or Firefox. Customers have to either upgrade/install browsers or have to maintain multiple browsers.

Another example is websites that are built for desktops viewing. When these are accessed on mobile devices, a part of the functionality does not work.

There are also cases where the applications are accessed via virtualization software. This may restrict functionality and and inhibit experience.

These issues could be avoided by understanding the environment in which the product would be used.

3. Collaborate with all stakeholders across the value chain

Addressing the needs of the customer is surely the key for building a great product. However, sometimes it makes sense to look sideways to gather requirements. For example, when one department hands over data to another department, there may be a gaping funcitionality hole. The requirements gathering process should address all the significant interactions between cross-functional teams and see where data migrations and handovers are happening. Even though these stakeholders would not be direct customers their requirements have an indirect impact on how the product would perform in a real world settings. Product managers should also seek feedback from within the team and leverage on their experience in building products.

4. Leverage cloud based tools

It’s so surprising that to date there are many companies that manage their requirements in Word documents and Excel sheets. Though these are wonderful tools there are several limitations with them in terms of version control, audit trail, and collaboration. They also don’t offer very rich visual dashboards. Reusability of various components also is a great impediment.

Cloud-based tools address these issues transparently at a much lesser cost. These tools are all the more important today as more products are shifting to agile methodologies where requirements have to be updated, added and modified in real time.

Cloud-based tools also have rich visual interfaces which help in tracing each requirement and delivery schedule. These tools help in maintaining a traceability matrix and change management process. Companies should consider using these tools for detail documentation and requirement-by-requirement collaboration.

5. Prioritize and perform competitive analysis

It is important that during the requirements gathering process product managers are aware of the customer priorities and share it with the development team. It is also the responsibility of the managers to keep close track of the competition and market knowledge as any changes in either could result in a change of priorities or revision in requirements – to which even the customer can be oblivious.

Requirements gathering is an extremely critical exercise that any error at this stage could lead to an unusable product, resulting in financial loss. Companies can customize and implement the above steps to ensure minimal lapses.