Skip to main content

Author: Srikanth Chintamaneni

Agile Business Intelligence

Feature May8 2012  4973821An iterative methodology for fast, flexible and cost-effective Business Intelligence

Can Agile Business Intelligence finally deliver analytics and insights to the people that need it? Or is it potentially a distraction? Agile philosophy has been around for more than a decade, and it looks like BI is catching up to it, with increasing talk of Agile BI. While Agile development emphasizes technology and business teams working side by side, Agile BI puts forth the notion that business end-users should be free of the technologists altogether – with more self-service tools and ability to design interfaces and analytics to explore information without having to first go through the IT department.

Traditional approaches to BI cannot deliver solutions and reporting fast enough in this era of hyper-competitiveness. Companies spend a lot of time modeling data, and that’s precisely what IT team does very well. They collect requirements and transform those requirements into data models. But the problem is it takes too long. By the time the development is done, the requirements have changed. If the IT team didn’t foresee some of the requirements, and didn’t model them, well then, the organization really cannot analyze the condition. So we definitely need environments that run on the data itself, not from data models. Given the difficulty that many organizations have faced in delivering the BI applications their managers and executives need to understand performance and make critical business decisions, it’s not surprising that an alternative development approach is being embraced. Indeed, there is a broad and growing consensus that Agile BI’s time has come.

 

Some of the key traditional challenges that the Agile BI addresses are as follows

Growing Demand:

Demand for information about business performance has risen dramatically. (The Information Age could just as well be called the BI Age.) BI delivery teams have a large backlog of projects from business users looking for more information to support their decisions. But it’s not just more information users want; it’s more information faster. Agile BI helps IT meet the imperatives for quantity and speed in unlocking the full value of data assets.

Flexibility:

The agile methodology is designed to adjust to changing requirements – and BI requirements change more frequently and profoundly than those for nearly all other types of software projects. In fact, in a 2011 survey of 200 businesses and IT executives conducted by Forrester, 67% of respondents said that BI requirements change at least monthly. A full 20% of respondents said their BI requirements change on a daily basis. Such changes wreak havoc on the traditional waterfall delivery cycle, yet they are inevitable during the lifetime of any BI project.

User Engagement:

The great strength of the agile methodology is that it fosters collaboration between IT and the business. While traditional approaches have struggled to place user needs at the core of the process. Agile BI is all about giving users faster access to functionality and more opportunities to provide feedback. Ultimately, user engagement equates to higher user satisfaction and adoption rates.

Manageable Scope:

Budget overruns and blown schedules can damage IT’s credibility, besides costing the company real money. Because Agile BI focuses on the delivery of smaller sets of functionality in shorter time periods, projects are driven by business defined scope and value. Project timelines and budgets can be tracked in smaller units, and users pay for the value defined. Avoiding scope creep is good news, but it’s better news that these budgets are significantly smaller and the project timelines much shorter.

Lower Costs, Higher Value:

Agile methods in BI have a strong track record in reducing project costs and shortening timelines. Further, because project budgets are aligned to high-priority deliverables and outcomes – that is, high-powered, easy-to-consume applications that users like and that meet real and urgent business needs – overall technology ROI also increases.  Conventional SDLC approaches are poorly suited for BI. Traditional waterfall methodology for SDLC calls for collecting user requirements, documenting them, transforming them into specifications, and then turning specifications over to developers, who then go through the design, build, test, implement cycle. While this approach is often successful for traditional enterprise application implementations, it is almost guaranteed not to work for the majority of BI requirements. The “build it, and they will come” mentality is directly applicable — and recommended — for BI, since only once an end user sees something in front of him, something he can touch and feel and “play with,” will the real requirement materialize.

Clearly a different approach is needed to make BI applications more flexible and able to react much faster to ever-changing business and regulatory requirements. Agile BI is first and foremost a different approach to designing and building BI applications.

The purpose of Agile BI is to: 1) get the development done faster, and 2) react more quickly to changing business requirements. Mostly Agile BI is no different than any agile development methodology that calls for incrementally delivering products versus a big-band approach; for rapid prototypes versus specifications; for reacting versus planning; and for personal interactions with business users versus documentation. The Agile BI methodology differs from other agile approaches in that it requires new and different technologies and architectures for support.

The key to driving an Agile BI project is minimizing project management overhead, reusing existing assets, and automating inefficient manual tasks. Aligning your project budgets to deliverables and outcomes generates more value – that is, high-powered, easy-to-understand, easy-to-consume business intelligence solutions that users like and that meet real business needs.

How to achieve agile development

Agile BI projects should focus on people over process. This does not mean that Agile BI is inherently opposed to thorough and careful development processes, but is guided by a minimalistic business user focused approach, to streamline development cycles. There are three key agile processes that should be adhered to ensure an Agile BI rollout:

Iterative ‘Sprint’ development cycles

  1. Systematize ongoing BI processes
  2. Implement Barely Sufficient Processes

Iterative ‘Sprint’ development cycles

In Agile development methodology, teams work in ‘sprints’ to produce bite-sized deliverables in an iterative manner. Sprints can be one to four weeks long depending on the size and complexity of the project. At the end of each sprint, the business has a working deliverable, such as a new report or dashboard, delivered to them in a production setting.

By contrast, the waterfall development cycle used in traditional BI rollouts, is cemented in a regimented, sequential progression. It is inflexible to changing reporting needs and costly to make changes to. Agile BI, as a process, is about delivering functioning software regularly in short weekly or monthly timeframes – the shorter and more frequent the better (working software is the measure of BI success). Agile BI is about responding to the immediate needs of the BI user, rather than working to establish and deliver ALL potential reporting needs upfront. To establish an approach that facilitates Agile BI, reporting objectives should be readjusted regularly based on available resources and intermediate business goals to help focus attention on what is really important. Working in this manner ensures the relevancy of reporting to business goals and enables a faster, more flexible approach to changing reporting needs.

Systematize ongoing BI Processes

Agile BI development teams must automate any repetitive tasks/processes to allow more time and focus to be spent on developing and delivering end-user features. For example, whilst critical, testing the BI system manually takes up unacceptable resources each and every sprint cycle. Automated testing conducted by the users actually involved in the development of new reports, means that new changes can be quickly tested within the sprint, rather than waiting to be processed by a separate testing team. This way, accountability resides within the team.

Implement Barely Sufficient Processes

Minimizing the amount of ‘ceremony’ associated with BI development reduces the length of development cycles and allows development teams to concentrate on the work that matters. This minimalistic approach does not suggest that careful planning is unnecessary during the development process, but that formal planning and documentation should be aimed at satisfying the practical needs of the project. For example, a concept document for each sprint should focus on business user requirements and nothing more. Additional verbiage simply adds no value. Agile BI is just as much about maximizing the amount of work not done.

Summary

Successful Agile BI deployments enhance organizational flexibility and responsiveness. Increasingly, businesses and their personnel are exploiting the benefits of Agile BI, allowing them to respond with immediacy to business demands. To survive and prosper in a competitive marketplace, businesses from all industries and sectors have to be able to scan their external environment, review their internal processes and make appropriate proactive and reactive changes. Modern companies are striving to spread fact-based decision-making throughout their organizations. Agile BI solutions, with their end-user centric approach, enable organizations’ to anticipate and adapt to shifting market conditions.

References

  • Information Management Blog : Agile BI
  • Yellowfin : Agile Business Intelligence
  • Forrester report on Agile BI Out Of The Box
  • Balanced Insight: Enabling Agile Business Intelligence with Balanced Insight

 Don’t forget to leave your comments below.

Business Analytics with In-Memory Databases

Abstract

The Business intelligence (BI) and Data Warehouse vendors are increasingly turning to in-memory technology in place of traditional disk-based storage to speed up implementations and extend self-service capabilities.

For years, it has been noticed that the process of creating customer data queries and building business intelligence reports has been a prolonged activity. This is because the information needed must be pulled from operational systems and then controlled in separate analytical data warehouse systems that can accept the queries. Now, however, with the advent of true ‘in-memory analytics’, a technology that will allow operational data to be held in a single database that can handle all the day-to-day customer transactions and updates, as well as analytical requests – in virtually real time.

Starting Questions

Successful Business Analytics project implementations start by asking the right questions.  Here are a few that should be on your short list.

·   How do I manage and maintain the performance of my existing reports with the ever increasing data?

·   What is the cost effective alternative to data warehouses that provide the ability to analyze very large data sets, but is much simpler to set up and administer?

·   What can I do today to support near-real time reporting requirements and not relying heavily on IT departments?

·   How can I demonstrate value to my company to extend real-time ad-hoc query capabilities for high volume transaction functionalities such as Financial Services?

·   How do I minimize the administration overhead and yet provide a transparent reporting environment to end user?

The purpose of this article is to put both BI technologies in perspective, in-memory and disk-based, explain the differences between them, and finally explain, in simple terms, why disk-based BI technology is not on its way to extinction. Rather, explain the requisites for considering an in-memory database BI solution.

But before we get to that, let us understand the differences between disk-based and in-memory databases.

Disk-based and In-memory Databases

Database irrespective of disk-based or in-memory, we are talking about where the data resides while it is actively being queried by an application: with disk-based databases, the data is queried while stored on disk and with in-memory databases; the data being queried is first loaded into RAM (Random Access Memory).

Disk-based databases are engineered to efficiently query data residing on the hard drive. At a very basic level, these databases assume that the entire data cannot fit inside the relatively small amount of RAM available and therefore must have very efficient disk reads in order for queries to be returned within a reasonable time frame.  On the other hand, in-memory databases work under the opposite assumption that the data can fit entirely inside the RAM. The engineers of in-memory databases benefit from utilizing the fastest storage system a computer has (RAM), but have much less of it at their disposal.

The fundamental trade-off with disk-based and in-memory technologies is faster reads and limited amounts of data versus slower reads and practically unlimited amounts of data. These are two critical considerations for business intelligence applications, as it is important both to have fast query response times and to have access to as much data as possible.

Fast analysis, better insight and rapid deployment with minimal IT involvement!

What is it?

As the name suggests, the key difference between conventional BI tools and in-memory products is that the former query data on disk while the later query data in random access memory (RAM). When a user runs a query against a typical data warehouse, the query normally goes to a database that reads the information from multiple tables stored on a server’s hard disk. With a server-based inmemory database, all information is initially loaded into memory. Users then query and interact with the data loaded into the machine’s memory.

BI with In-memory databases may sound like caching, a common approach to speeding query performance, but inmemory databases do not suffer from the same limitations. Caches are typically subsets of data, stored on and retrieved from disk (though some may load into RAM). The key difference is that the cached data is usually predefined and very specific, often to an individual query; but with an inmemory database, the data available for analysis is potentially as large as an entire data mart.

In-memory database is designed specifically to take advantage of the immense amount of addressable memory now available with the latest 64-bit operating systems. In-memory technology uses the multi-gigabytes of memory space available in 64-bit servers as for its data store. In-memory analysis is designed to improve the overall performance of a BI system as perceived by users, especially affecting complex queries that take a long time to process in the database or when accessing a very large database where all queries are hampered by the database size. With in-memory database, it allows data to be analyzed at both an aggregate and a detailed level without the time-consuming and costly step of developing ETL processes and data warehouses or building multidimensional OLAP cubes. Since data is kept in-memory, the response time of any calculation is lightning fast, even on extremely large data sets analyzed by multiple concurrent users.

This kind of immediate, interactive analysis is particularly important when people are trying to discover unknown patterns or learning new opportunities.

Who is it for? Know your challenges Finding the right mix
  • When selecting an in-memory solution consider one that operates seamlessly within an end-to-end BI platform where its usage is completely transparent to users and report developers
  • Ideal for setting up departmental BI applications and for meeting the BI needs of small to medium sized businesses as it requires very little up-front effort, and no ETL
  • Populated quickly from any database source, users can seamlessly use in-memory databases and associated meta-data layers as a source for many reports, dashboards, and analysis
  • Look for technology that has been designed to avoid the excessive administrative burdens and can scale to enterprise levels in terms of user number, data security and data governance
  • The leading benefits of Business analytics with in-memory databases are to deliver decision insight with the agility that businesses demand. It is a win for business users, who gain self-service analysis capabilities, and for IT departments, which can spend far less time on query analysis, cube building, aggregate table design, and other time- consuming performance-tuning tasks
  • Regardless of what fancy algorithm is used with an in-memory database, storing the entire dataset in RAM has a serious implication: the amount of data one can query with this technology is limited by the amount of free RAM available, and there will always be much less available RAM than available disk space
  • Limited memory space means that the quality and effectiveness of the BI application will be hindered: the more historical data to which we have access and/or the more fields we can query, the better analysis, insight and, well, intelligence one can get to
  • One could add more and more RAM, but then the required hardware becomes exponentially more expensive. Beyond 64GB, we can no longer use what is categorized as a personal computer but will require a full-blown server which brings us into very expensive computing territory
  • Note that the amount of RAM required is dependent on the number of people simultaneously querying it. Having 5-10 people using the same in-memory BI application could easily double the amount of RAM required for intermediate calculations that need to be performed to generate the query results.
  • A key success factor in most BI solutions is having a large number of users, so we need to tread carefully when considering in-memory technology for real-world BI. Otherwise, the hardware costs may spiral beyond what the organization is willing or able to spend
  • Some of these databases introduce additional optimizations which further improve performance. Most of them also employ compression techniques to represent even more data in the same amount of RAM
  • The future of BI lies in technologies that leverage the respective benefits of both disk-based and in-memory technologies to deliver fast query responses and extensive multi-user access without huge hardware requirements.   These types of technologies are not theoretical anymore and are already utilized by businesses worldwide. Some are designed to distribute different portions of complex queries across multiple cheaper computers (this is a good option for cloud-based BI systems) and some are designed to take advantage of 21st-century hardware (multi-core architectures, upgraded CPU cache sizes, etc.) to extract more juice from off-the-shelf computers

Summary

Business Analytics with in-memory database provides companies with a faster, more flexible, and arguably lower-cost way of accessing and processing information allowing users to get answers to business questions in seconds rather than hours. By virtue of its high performance architecture in-memory has the potential to help midsize organizations become more informed, agile and respond quicker to changing market conditions.

In addition, advances in technology and lower costs of memory and CPU make this type of technology more attractive than ever before. Matching the appropriate architectural approach with the kind of business analytics solutions needed by a midsize company has the potential to deliver benefits such as reduced time to insight, greater agility, increased self-service and lower overall IT demands.

References:

  1. Open source In-memory Analytics – YellowFin
  2. Extinction of traditional Business Intelligence: Elasticube Chronicles
  3. In-Memory Data Management by Plattner/Zeier

Don’t forget to leave your comments below.


Srikanth Chintamaneni is a manager in the Information Management service line of Deloitte Consulting India Pvt. Ltd. He has over 13 years of experience in providing consulting services involving data warehouse and content management solutions in the Health care, Commercial & Consumer Finance, and Industrial Products industry segments. His capabilities support services involving data profiling, data modeling, report design, and end-to-end data warehouse implementations.