Tag: Software

It’s the End of the Business Analysis World as We Know it? Part 6

Written by Steve Blais on August 20, 2013. Posted in Articles.

Being the serialized story of Brian Allen and Ann Brady, business analysts, and their Adventures in the New Oder of Agile

Excerpted from the forthcoming book from John Wiley, The Agile Business Analyst due out the end of 2013

Chapter 6: Wherein Verna addresses the business analysts and their final fate is known while Brian searches for the right theme song.

Brian saw Ken and Sandra walking slowly toward them from the end of the corridor with measured pace. He and Ann were walking toward them with the same measured pace. The corridor had emptied out. Brian heard the whistling trill from the theme song to “The Good, The Bad, and the Ugly” echo in his head as they approached each other. He immediately thought that he was at a disadvantage since he still had his backpack slung over his shoulder. “I got my updated resume in my backpack, just in case,” Brian whispered to Ann as they walked. She nodded.

Then Ken and Sandra turned to their left down an intersecting hallway. Brian and Ann turned left seconds later to follow them. Still no words were spoken. It was the way of Verna. The whistling in Brian’s head was now replaced by the bubbly, somewhat inane tune, “We’re off to see the Wizard…” and he looked at the floor to see if there were yellow tiles.

They turned left again and then right along passageways and halls that seemed to become narrower as they proceeded. Brian realized that he and Ann were in a sector of the Organization’s Headquarters that they had never been in before and didn’t even know existed. The theme song to :”The Twilight Zone” snaked into Brian’s brain replacing the jovial image of Oz with the dour visage of Rod Serling.

Finally they came to The Door. Sandra pressed a button and passed her badge over a green light and The Door opened. Inside was a large well lit antechamber with several desks with people busy doing things people do at desks, a couple of conversation areas and three doors leading to other offices. Brian wondered if he were going to get to pick which door Verna was behind.

Sandra led them to the left hand door and opened it after knocking. When the door opened everyone behind the desks stood up at attention. Ann, Ken and Brian entered the room. Sandra did not. She closed the door behind them. This was Verna’s office.

“Welcome, I’ve been expecting you,” came a disembodied voice that was strong and authoritative but seemed to carry with it a smile of understanding and empathy. The three looked around the large office, darkened by heavy drapes over the windows, and could not immediately see the source of the voice.

“Please, have a seat.” There were three chairs in front of the executive desk in the center of the room and an empty black executive swivel chair behind it. Then they saw Verna standing at the corner of the desk.

When Brian saw Verna he realized why she was rarely seen around the organization. She was 4’9” in height and was barely visible behind the large desk. He imagined her sneaking unseen around the corridors of the organization, and the theme song to The Pink Panther crept into his head. Verna pulled herself into the executive chair and they could see her clearly. She had short page-boy length hair and was of indeterminate age although her face spoke of volumes of experience in the corporate political wars.

“First of all,” Verna said. “I want to let you know that I am completely in favor of the entire concept of agile: agile software development, agile management, agile organization. I have been in favor of it long before it was agile. Back in the day being agile as you describe it now was just business as usual. We in the corporate world have lost touch with our ability to respond, to be flexible, to take chances and risks, and to make things up as we go. Management has a few set backs or failures along the way and suddenly we are all worried about risk management and making sure that there is a paper trail we call documentation to absolve us of blame. We all had the agile spirit at least once, when we were start ups – as organizations or when we first started up in the business field. But as things went on we lost it. So, now if we have to call it agile and take a slap in the back of the head or a glass of cold water in the face to wake us up, then I’m all for it. And that includes all of the precepts, such as removing the business analyst from between the business person with the problem and the development team with the solution.”

This pronouncement deflated Brian and Ann. Brian let his backpack slide to the floor next to his chair.

“You don’t need to pull your resume out of the backpack as yet, Mr. Allen,” said Verna with a smile.

Brian looked up and over at Ann. His face said “how did she know that?” Her faced responded with “Lucky guess, maybe?” He noticed out of the corner of his eye that Ken was smiling in apparent victory.

“I am talking about removing requirements as we know them: This absurd process of creating long documents followed by lengthy confirmation processes and approvals by executives who don’t even read the bloody thing, and then making changes which also require confirmation and approval are absurd. We need to be more flexible, fluid and responsive, and requirements documents do not get us there. There is too much time spent in arguing about how a requirement is written and the meaning of the words in the document than in defining what is necessary to be done to solve the problem.”

Now Brian could see Ken’s grin growing wider. Ann was looking down and nodding. Brian found that there was nothing that Verna was saying that he could disagree with. He had felt for a long time that defining requirements was a small and sometimes trivial part of his job, but he had always assumed it was necessary. After all, what is a business analyst without requirements? The Business Analysis Body of Knowledge seemed to be all about requirements as the Project Management Body of Knowledge was all about structure and documentation. And he knew; he had his PMP although he wasn’t sure anymore why he had it.

“But,” said Verna, interrupting Brian’s reverie, “That does not mean at all that we are considering eliminating the business analyst. The business analyst is vital to the success of this organization, more so than the technologists who develop the software solutions.” She aimed her last comment at Ken whose smile froze on his face and then fell into his lap. Ann looked up at Verna with a question on her face.

“Let me highlight the job you two have been doing for the organization. It hasn’t gone unnoticed,” Vern continued. “You have helped Marketing, especially Dmitri, many times over the past by ensuring that their initiatives mesh well with the rest of the organization and are aligned with our overall strategic goals. Marketing has a reputation for focusing only on what is necessary to achieve their goals without regard for the rest of the organization. And that is good. Without them we don’t go anywhere. But we need someone to provide an overall business view to what they are doing, and you have been doing that.

“The information you assembled and the analysis you provided on our strategic build or buy situation is going to save us a lot of money and contribute significantly to the success of Backbone. There will be less coding that has to be done…sorry, Ken…which means that parts of the overall system will come up sooner than expected giving us significant competitive advantage and return. And we thank you for those efforts.”

“We were just doing our job,” muttered Ann, still bewildered at Verna’s comments.

“And that, my dear, is my point, you are doing your job,” responded Verna. “I also personally like the way you keep asking questions, even in the face of rampant enthusiasm, especially on the part of Marketing. You also challenge the technologists’ assumptions and the ‘no problem syndrome’. You apply upfront critical thinking which has saved this organization thousands in miscommunication and rework costs. I know anything “upfront’ is considered suspect by the agile authorities, but I tend to be old school and believe that any process that makes things work more efficiently is worthwhile no matter where it occurs or what you call it. And I particularly like the way you do it without a lot of paperwork, overhead or ceremony.

“You facilitate meetings and get problems addressed and solved. You effectively deal with conflicts, especially between IT and the business units and between the business units themselves. Unfortunately many of those conflicts are about how requirements are phrased, but we may have eliminated that source of conflict with this agile product development approach. You seem to be able to get the best out of people when you are around, helping them achieve their goals. You were instrumental in providing the information and argument that convinced Carl that his precious system needed to be replaced and not maintained.

“You have taken the time to keep up to date not only on technology so you can understand the technologists, but also the business side so you understand business jargon. And I’ve noticed that you don’t spend time just translating one set of jargon to the other, but facilitate the interactions between the suits and the nerds so that each will become more familiar with the other’s domain. Very good. Of course, it has led to the current demand by Ken and his cronies that you be removed from the ‘middle’. So you unwittingly were part of your own demise in that regard.

“I also noticed that people around the organization seem to call on you when they have problems and you are able to help them define what their real problems are and help them come up with solutions, even when those solutions do not involve software development or even IT at all.”

“But they work for IT,” Ken pointed out. “They should be defining IT solutions.”

Verna affixed a withering stare at Ken. He seemed to shrink in his chair until he seemed to be shorter than Verna. “They do not work for IT, sir. They work for the organization.”

Brian and Ann shared glances. They never assumed anything else.

“The bottom line is that business analysis is here to stay. What you do is the real backbone of the organization. You provide independent and objective information to the decision makers, define and solve business problems, improve business processes and make a multitude of other contributions..”

Ken appeared to have one more protest up his sleeve. “I was given the direction by Vince and others that we all have to be agile. The entire organization is going agile.”

“Yes, I believe that is true, and as I clearly stated, I am in agreement with everyone going agile. But these business analysts, for all intents and purposes, sir, are already agile. They are responsive, flexible, focus on the problems of the entire organization, collaborate at all times and facilitate collaboration wherever possible, and work to get the business customers with problems and the IT or other teams with solutions together to add value to the organization. How much more agile can you get, sir? I believe if you go back and look at your recent history, you will see that business analyst laid the groundwork for successful agile organizations.

“I would say, Mr. Allen and Ms. Brady, that you are certainly agile as far as the organization is concerned, at least as agile as the developers working with Scrum. So, since IT has decreed that they don’t seem to have any use for you, replacing your requirements definition duties with the agile process, then what I want is that you work directly for me. You will remain business analysts. And your scope will be the entire organization. I envision an entire team of business analysts moving the organization strategically, tactically, and competitively forward”

There was a long pause as Brian and Ann let the directive sink in.

“I have spoken,” commanded Verna, ending the meeting. \

Outside in the hallway, Ken nodded to them and bid them well and said that it was clear that he would be working with them on upcoming projects, including Backbone. He did not seem to bear any ill will. In fact, Brian almost sensed that Ken now welcomed their participation in the agile community as business analysts. But then, that remained to be seen.

As they stood in the lobby of Verna’s offices, Brian looked at Ann. “Do you know the way back?”

Ann replied, “I left the bread crumbs.”

“Looks like everyone is safe.”

“Yes, we did it. And we got to meet with Verna.”

“Hey, did you notice? No strange silences and eerie feelings when you said her name.”

“Yes. I wonder if it’s just because we met her or because we now know her real stature. She’s small, you know?”

A disembodied voice came out of hidden speakers someplace near where they were standing in the lobby of Verna’s area. “I heard that, Ms. Brady.” It was unmistakably Verna. They looked at each other and scurried out of the area and through The Door.
“How did…?” whispered Brian. Ann shrugged.

“Well for one thing,” said Brian, still keeping his voice soft as they moved down the hall. “This is going to be really challenging and very interesting.”

“Yes. And we love it that way. We are, after all, business analysts.”

Don’t forget to leave your comments below.

It’s the End of the Business Analysis World as We Know It Part 2

Written by Steve Blais on May 14, 2013. Posted in Articles.

Being the serialized story of Brian Allen and Ann Brady, business analysts, and their Adventures in the New Oder of Agile

Excerpted from the forthcoming book from John Wiley, The Agile Business Analyst due out the end of 2013

Chapter 2: Wherein the doomed business analysts figure out a plan to embrace change

When we last left Brian Allen he was facing the Perils of Pauline the head of the organization’s PMO.

Brian was worried, not just for himself and Ann, but for the other business analysts in the group. There were the senior business analysts, Stan, Shelly and Summer. And there were the junior business analysts: Jennifer, Jocelyn, and Juan. What would become of them if all the business analysts were eliminated from the software development process? The other business analysts worked on other projects not currently threatened by Ken and the Agile Forces. But they soon would be, especially if Ken’s project were successful. Brian turned his attention to Pauline who sat framed in her office window with the sunlight bouncing off her black curls.

“I have gotten the word not to assign a project manager to the new Backbone Project. They are going to be using this agile thing to develop the software and manage the project. Do you know anything about it?” Pauline asked with her eyebrows arched. She spoke through compressed lips.

“Yes,” Brian replied. “I was aware it was coming down the pike, but thought that cooler heads would prevail and we might start with a small project first. Maybe a new label printing project or something equally benign.”

“Well, it is happening and as you are probably aware, that also means that the PMO will not be assigning any business analysts either.”

“You never assigned business analysts before.”

“That is irrelevant. We will not be doing it now.”

“I suppose those of us originally assigned to the Backbone project will have to be reassigned to other projects now?”

“That would normally be the situation. However, most of the new projects are in Marketing, and the Vice President has decreed that all his projects from now on will be run as agile projects. There is no place for business analysts there either.”

“In other words…?”

“In other words, there are no projects for you to go to that are not agile right now.”

Brian looked down between his feet. “Any idea of what we can do?”

“Not in the least,” snipped Pauline. “I have my own problems. I need to figure out what happens to the PMO and my job when there are no project managers and software developers start working directly with the business owners.” She was being dismissive.

Brian got up to leave wondering why Pauline called the meeting. As he reached the door he realized that she was actually asking for his help, but would not do so directly. “I’ll see what I can do, Pauline,” he said as he left noting that she smiled for the first time. He guessed that she was coming to him since he was instrumental in evolving the processes that developed into the first PMO at the organization.

As he walked down the hall he passed Ann who was with Raj on their way to talk to Pauline.

“Any luck?” asked Ann apprehensively.

“No. But we’re not fired immediately. Basically, we have to figure out how we will fit within the New Order,” replied Brian.

“Not a good proposition,” said Raj. “I perceive we are up against the wall on this. Ken does not like business analysts and I doubt he ever has, although I only know the man for a few years. He thinks they add no value to the process of developing software. He says that they get in the way; they are an obstacle. He says that business analysts only copy down what the business people tell them as requirements and then reformat them and give them to his developers.”

“We do more than that!” said Brian emphatically. Then after pausing and getting no supporting reaction from either Ann or Raj, added, ‘Don’t we?”

“Well,” said Raj. “I have to recall that his comrade, Scott, says that ‘BA’ also stands for ‘Band Aid’. I would not take that as a positive sign.”

As if on cue, Ken and Scott rounded the corner and came upon the three business analysts. Ken was tall with short cropped blonde hair and a stubbly beard on his face, and Scott had long, dark-haired, was shorter and rounder, and wore sliver glasses that were rectangular shaped.

“It’s the requirements gatherers,” Ken said with a sneer. “Hello, Gatherers, let the hunters through. We have projects to do. We’ll get them all done and have dinner on the table before you even get your first set of interviews done.”

“Speed,” echoed Scott, ”is what it is all about. You gather while we produce.”

“We don’t gather requirements,” protested Brian. “We gather information and then analyze the requirements from that information to make sure that the requirements solve the business problem. How are you going to know you’ve solved the business problem if you don’t even analyze enough to determine what the business problem is before you start?”

“We adjust. We adapt. We learn. We let the problem and the solution emerge through our collaborative activities. We don’t do the Waterfall thing with each group creating a document to pass on to the next group. Everyone works together in agile.”

“So, why not have business analysts involved with the collaboration? I think we have plenty to offer. After all we are the organizational communicators and problem solvers. We are part of the team.”

“Not in Scrum you are not. Scrum recognizes no titles for Development Team members other than Developer, regardless of the work being performed by the person; there are no exceptions to this rule.* And we don’t want any groups of business analysts lurking about either. Development Teams do not contain sub-teams dedicated to particular domains like testing or business analysis.* In other words, there is no place on our development for anyone but developers. Tough luck, Gatherers, you need to find a new job.”

Ann, Brian and Raj looked at each other and then back at Ken and Scott as they walked down the hall. As the two of them reached the corner, Scott turned to the business analysts and shouted, ”Oh, we’re having a memorial service for the Waterfall later this week. You’re invited. And the business analyst might be next to go.”

Ann pushed her blond hair across her forehead in her signature gesture and asked, “So, what are we going to do? The eight of us had been assigned to the Backbone project and Helen over in HR said that she has already gotten the official request for transfer for all of us. And she said she doesn’t really have anywhere to put us since marketing is supposedly going all agile and they have most of the new projects.”

“What are our options, Raj. You know more about this than we. You’ve been working a bit with the web guys”

“There are only three roles in Scrum: product owner, scrum master and developer. The product owner represents the business and provides the work to be done by means of the product backlog. The scrum master helps the team with the Scrum process and removes obstacles to the team’s success. And the developers write code and test.”

“That’s it?” said Ann. “I think we are doomed. I need to get my resume dusted off.”

“No. Not yet.” Said Brian. “I have an idea.”

“To get them to back off agile?” asked Ann.

“No, I think that ship has sailed.”

“Then perhaps to sabotage the ship so that it sinks, and they call us back in to rescue the project and survivors?” suggested Raj.

“Excepting of course, Ken and Scott,” Ann added.

“No, not that either, although that does sound appealing.”

“Then what?” asked Ann as they entered the lunch room and saw the rest of the business analyst team huddled together in the corner over their lunches and looking very worried.

“I recall something about the definition of the development team that Scott didn’t mention: Individual Development Team members may have specialized skills and areas of focus, but accountability belongs to the Development Team as a whole. *” Ann stepped away from the counter to take a cell phone call.

“Yes, Brian, that is true,” responded Raj, picking out a salad, “But everyone on the development team must be able to code. Everyone is a developer. You are not supposed to have someone who is just one specialty.”

“Right. But here’s the thing, Raj. The development team is supposed to be cross functional and consist of “generalizing specialists” as Scott Ambler says, right?” Raj nodded. “Then we suggest that each of the three pilot teams take Jennifer, Jocelyn and Stan. They are all recent programmers who have moved into business analysis. They have the coding skills, and Jocelyn also has testing skills, so they are already generalizing specialists. They would bring their facilitation skills to the mix and make it easier for the teams to talk to the business. They also bring some business acumen that the developers don’t have since they have been working on this side of the house for a while.”

“But they will not be business analysts anymore?”

“No, that’s true. I don’t think any of us here will be business analysts after this. It is after all a New Order.”

“That is fine,” said Raj helping himself to a bowl of watermelon and mixed fruit while Brian picked a slice of double chocolate cake and wished the organization’s cafeteria served beer or wine with lunch. “But that is only three of the business analysts. What about the rest of us? Certainly you cannot code anymore. It has been at least twenty years since you last coded.”

“True. And no one uses PL/1 and ALGOL anymore anyway. I mentioned those three because I know they have technical skills and probably would not mind being back in the technical fray again. There may be many business analysts who are capable of applying their technology training or experience and can leverage that technical ability with the facilitation and communication skills they picked up as a business analyst. They would actually be doing more true business analysis work than if they just transcribed requirements from the business community.”

“OK,” smiled Raj. “That is a solution. But only a partial solution.”

“Guys,” interrupted Ann as they were paying for their lunches. “I just got off the phone with Helen. She says that the organization is instituting some belt tightening measures. There will be layoffs. They are going to start with all the older employees and move to those who support the older technologies. Anyone who is not specifically spoken for and that means business analysts who are not attached to a project.”

“That means all of us.”

“Except Jennifer, Jocelyn and Stan and any other business analyst who can morph into the role of Scrum Developer.”

“Brian, you have any other ideas?”

“Looks like you will be burning the midnight oil again tonight, Brian.”

Now there was an even more pressing need for action and ideas. The organization has announced There Will Be Blood. Will our own Captain Midnight be able to rescue the rest of the business analysts (including himself)? Will the Bell Toll for business analysis as well as Waterfall? Will Ann get to eat lunch? Join us next month for another thrilling episode of Brian and Ann in the New Order od Agile.

Don’t forget to leave your comments below.

* Quoted from Schwaber & Sutherland, The Scrum Guide: the definitive Guide to Scrum: the Rules of the Game, published by Scrum.org., July 2011

It’s the End of the Business Analysis World as We Know It

Written by Steve Blais on April 16, 2013. Posted in Articles. 3 Comments on It’s the End of the Business Analysis World as We Know It

Where Does the Business Analyst Go From Here?

Being the serialized story of Brian Allen and Ann Brady, business analysts, and their Adventures in the New Oder of Agile

Chapter 1 How it began

Now he was concerned. Brian had weathered pretty much everything the organization could throw at a business analyst from the early days when he had to fight just to make the role a viable position in the organization to the past few years when he was under constant pressure to take over project management jobs in addition to his business analyst work. Brian was flattered that they felt that he had the capability to do both jobs simultaneously, but his heart was in business analysis and he was afraid that if he diluted his efforts by also managing he would seriously impair the enterprise level project he was on. Besides he had already done the project management thing. It was interesting and challenging, but it wasn’t what he wanted.

Now things were changing once again. The project he was on was a long term, several year reconstruction of one of the primary internal systems. It was one of the first systems the company had built to provide internal control over sales, marketing, order processing and just about everything associated with the organization’s backbone. The system was the pride and joy of the CIO, Carl, who was on the team that developed the system twenty years ago and it took some convincing, and a bit of arm turning, for the CIO to finally admit that the system needed an overhaul. Since he identified so closely with the system, it was as though he himself needed an overhaul. As a result, Carl also looked at the way the organization had been developing software over the years and despite a highly successful record of on-time delivery, at least on the internal mainframe based systems, the CIO decided that perhaps the organization should segue into agile.

The web site maintenance had converted to an agile approach to software development a few years back and been successful after some bumps. There were some teams here and there on the periphery of the internal systems that were run with an agile approach, mostly in maintenance activities. But the primary systems, based on the trusty IBM computers and written in Good Old COBOL, with a smattering of Assembler Language Code, were still maintained in a structured software development approach patterned after the linear, waterfall, model. Major new efforts, like the overhaul of the purchase order system which took two years was also done the traditional way.

Since the entire backbone system of the organization was going to be rewritten – code, databases, architecture, even some new networks – Carl had decided that everything might as well get more “modern’ as well. They were even relocating various offices and departments.

So Brian was worried. He and Ann were the lead business analysts in the organization and from his understanding of agile and its mystiques; he was looking at the end of his career as a business analyst in the organization. Neither he nor Ann wanted to leave. Their roots ran deep in the organization. But Ken, the leading Scrum advocate, who had led successful Scrum teams and Theresa, the technical lead, who was a staunch Extreme Programming (XP) advocate, were both onboard with the software development conversion and had advised Brian and Ann independently that the services of the business analyst were no longer required in the New World.

It wasn’t pretty. Brian still shuddered when remembering the show down meeting when the CIO decided to give the green light to the New Order of Development. What was ironic was that Brian had been a vocal proponent of the overhaul of the old system and was instrumental in influencing the CIO and executive committee to launch the new program that appeared to signal the death of his own job. Brian in fact, with Ann’s help, had written the successful business case!

Brian, of course, protested loudly in front of Carl, Vince, the Vice President of Marketing and heir apparent to the CEO’s job, and Olivia the Vice President of Operations, asking how the teams were going to determine what they had to do for the new system.

“Do any of your programmers have experience with inventory turns or balancing a cash receipts journal?” Brian had taken accounting courses and even a CPA overview from the state CPA association to keep up to date on accounting.

“They don’t have to be experts,” Ken countered. “We have the experts in the business who can tell the developers (he emphasized the word to remind Brian how out of touch he had become) what needs to be done. It’s not a problem. It’s time the developers and the business cut out the middle man, er, person (acknowledging Ann) and let the solution meet the problem directly.”

Ann spoke up. ”I’ve spent years with both the developers and business people. I don’t think the developers care to sit in long meetings and discuss the intricacies and ambiguities of supply chains and customer service. They have always preferred to just get the facts: what they have to do to solve the problem.”

Ken responded before Ann could finish. “This is the New Order, Ann. The developers are going to get with it and learn about the business. They are going to learn negotiation and interaction. Things are changing. We’re taking the developers out of their development hovels and moving them across the aisle.”

“Even if they don’t want to go? Even if they prefer just to code?” Brian asked remembering the days when he started out as a programmer and worked with many peers who really didn’t want to talk to anyone in the business.

“Yes,” replied Ken. “That is the Way of Agile.” He said it with a slight reverence as though talking about an Eastern Philosophy like Zen. Brian thought he saw Ken put his hands in front of his stomach and bow slightly with a Yin and Yang symbol hovering over his head.

Ann recovered the floor. “And I have not seen many of the business people who will jump at the chance to spend all their time with the developers and sacrifice their daily jobs. They don’t understand how the developers need to hear the specifications. They don’t understand how to specify requirements.”

Ken, in his enthusiasm, jumped in again, “They don’t specify requirements. Requirements are dead.” Brian thought he heard an underlying meaning in that statement: “Business analysts are dead along with the requirements.”

Ken explained, “We use user stories now. We don’t do requirements. Not only are requirements inefficient, tedious, generally unworkable, and unreliable, they are also boring. User stories are easy enough even users can do them. And we are also making the whole burdensome change management system that has infiltrated everything in our culture obsolete. No more green and pink forms, no cumbersome on line system that people forget to update, no endless change management meetings with the CCB that doesn’t want to be there in the first place, no politics,” Ken’s voice was raising, sounding like a politician in front of a Labor Day political rally crowd. “There will be no more risk management exercises with a Risk Register maintained by a Project Administrator, no Earned Value graphs and charts and reports, and, consider this, sir,” he said addressing Vince, “no more project managers or PMO.”

Ken, and everyone on the table, knew that the PMO and to a large degree the whole project management structure of the organization had been a thorn in Vince’s side for years. Vince hated the paperwork and time consuming processes that the PMO had established to provide some semblance of control and prioritizing of the projects in the organization. Vince was known to protest loudly to anyone who would listen, especially his golfing buddy, Charles Evans, the CEO, “We’re losing ground to the new and faster and more agile competitors out there. They get to market much faster than we do. They have a more agile approach to product development. When we come in with a change, it’s either in response to a competitor’s move or to market demand. If we don’t act fast we lose market share (market share was an important issue to Charles). If I have to go through all this paperwork and committee approval and business analysis and requirements and stuff…it takes longer to get something approved than to just get it done!” And unfortunately, Brian knew that was true to a large degree, but he was also around in the old days when everyone in Management Information Systems (MIS), as it was called then, seemed to be working on the demand du jour with no order or priority or sense. Efforts were introduced and dropped, replaced by other initiatives from executives with higher political clout, or urgency, and then brought back out of their hiatus with the admonition that they should already have been done. It was chaos. Projects failed not because they were mismanaged but because of the conflicting priorities and the disorder in the assignment of resources and the frenetic communications channels. Everyone in the organization could talk to any programmer at any time to get anything done. Brian was somewhat proud of the work he and Ann had done in bringing order to the business process of managing organizational change. He never took Vince’s complaints too seriously because the system was working fine for the rest of the organization.

It was then that Brian realized that Ken was in cahoots with Vince and had Vince’s complete support for the New Order. And Vince, as the next CEO, had the political clout. It looked like the Organizational Change Management system that had run successfully for these many years was as doomed as well.

“And as for the business people, they will love being able to get their needs met immediately in the order that they want them met. They will be able to change their mind, change their needs, change their approach, try new things and then try something else without the imprimatur of fighting the system to get things done. We see a major increase in innovation because the barriers to creative failure will be removed.”

Brian wondered where Ken had picked up the phrase “creative failure”. He knew it won points with Vince.

Brian and Ann fought to hold down their frustration. They lodged all the objections they could muster, but to no avail. The New Order was coming to the Organization. And that meant that there was no place in the New Order of Things for either of them or any of their cohorts on the Business Analyst Team that he and Ann had struggled so long to set up.

Brian’s thought, unexpressed to Ann who was lost in her own downtrodden thoughts, was “I’m way too young to retire, and way too old to start over again in a different organization or a different career. Besides, I really, really, really love being a business analyst.”

Brian was now being summoned into Pauline’s office. Pauline was the head of the PMO to which the Business Analyst Team reported. As he walked down the corridor which seemed a lot longer this morning than ever before, feeling like he was walking to his own execution, he was wondering, “what happens now? Is there no place for a business analyst in the New Order of Agile?”

With a shudder, Brian gathered himself together and opened the door to Pauline’s office.

“Ah, Brian,” said Pauline, not really smiling. “We have to talk. I’ll be talking to Ann next.”

What is going to happen to Brian facing the Perils of Pauline? Will the Business Analysis Team and the Change Management System survive? What about Ann and the rest of the business analysts in the organization? Join Brian, Ann and the rest of the Business Analyst Team in the next episode of Brian and Ann and the New Order of Agile next month.

Don’t forget to leave your comments below.

Excerpted from the forthcoming book from John Wiley, The Agile Business Analyst due out the end of 2013

Agile Business Intelligence

Written by Srikanth Chintamaneni on May 7, 2012. Posted in Articles. 2 Comments on Agile Business Intelligence

An iterative methodology for fast, flexible and cost-effective Business Intelligence

Can Agile Business Intelligence finally deliver analytics and insights to the people that need it? Or is it potentially a distraction? Agile philosophy has been around for more than a decade, and it looks like BI is catching up to it, with increasing talk of Agile BI. While Agile development emphasizes technology and business teams working side by side, Agile BI puts forth the notion that business end-users should be free of the technologists altogether – with more self-service tools and ability to design interfaces and analytics to explore information without having to first go through the IT department.

Traditional approaches to BI cannot deliver solutions and reporting fast enough in this era of hyper-competitiveness. Companies spend a lot of time modeling data, and that’s precisely what IT team does very well. They collect requirements and transform those requirements into data models. But the problem is it takes too long. By the time the development is done, the requirements have changed. If the IT team didn’t foresee some of the requirements, and didn’t model them, well then, the organization really cannot analyze the condition. So we definitely need environments that run on the data itself, not from data models. Given the difficulty that many organizations have faced in delivering the BI applications their managers and executives need to understand performance and make critical business decisions, it’s not surprising that an alternative development approach is being embraced. Indeed, there is a broad and growing consensus that Agile BI’s time has come.

Some of the key traditional challenges that the Agile BI addresses are as follows

Growing Demand:

Demand for information about business performance has risen dramatically. (The Information Age could just as well be called the BI Age.) BI delivery teams have a large backlog of projects from business users looking for more information to support their decisions. But it’s not just more information users want; it’s more information faster. Agile BI helps IT meet the imperatives for quantity and speed in unlocking the full value of data assets.

Flexibility:

The agile methodology is designed to adjust to changing requirements – and BI requirements change more frequently and profoundly than those for nearly all other types of software projects. In fact, in a 2011 survey of 200 businesses and IT executives conducted by Forrester, 67% of respondents said that BI requirements change at least monthly. A full 20% of respondents said their BI requirements change on a daily basis. Such changes wreak havoc on the traditional waterfall delivery cycle, yet they are inevitable during the lifetime of any BI project.

User Engagement:

The great strength of the agile methodology is that it fosters collaboration between IT and the business. While traditional approaches have struggled to place user needs at the core of the process. Agile BI is all about giving users faster access to functionality and more opportunities to provide feedback. Ultimately, user engagement equates to higher user satisfaction and adoption rates.

Manageable Scope:

Budget overruns and blown schedules can damage IT’s credibility, besides costing the company real money. Because Agile BI focuses on the delivery of smaller sets of functionality in shorter time periods, projects are driven by business defined scope and value. Project timelines and budgets can be tracked in smaller units, and users pay for the value defined. Avoiding scope creep is good news, but it’s better news that these budgets are significantly smaller and the project timelines much shorter.

Lower Costs, Higher Value:

Agile methods in BI have a strong track record in reducing project costs and shortening timelines. Further, because project budgets are aligned to high-priority deliverables and outcomes – that is, high-powered, easy-to-consume applications that users like and that meet real and urgent business needs – overall technology ROI also increases. Conventional SDLC approaches are poorly suited for BI. Traditional waterfall methodology for SDLC calls for collecting user requirements, documenting them, transforming them into specifications, and then turning specifications over to developers, who then go through the design, build, test, implement cycle. While this approach is often successful for traditional enterprise application implementations, it is almost guaranteed not to work for the majority of BI requirements. The “build it, and they will come” mentality is directly applicable — and recommended — for BI, since only once an end user sees something in front of him, something he can touch and feel and “play with,” will the real requirement materialize.

Clearly a different approach is needed to make BI applications more flexible and able to react much faster to ever-changing business and regulatory requirements. Agile BI is first and foremost a different approach to designing and building BI applications.

The purpose of Agile BI is to: 1) get the development done faster, and 2) react more quickly to changing business requirements. Mostly Agile BI is no different than any agile development methodology that calls for incrementally delivering products versus a big-band approach; for rapid prototypes versus specifications; for reacting versus planning; and for personal interactions with business users versus documentation. The Agile BI methodology differs from other agile approaches in that it requires new and different technologies and architectures for support.

The key to driving an Agile BI project is minimizing project management overhead, reusing existing assets, and automating inefficient manual tasks. Aligning your project budgets to deliverables and outcomes generates more value – that is, high-powered, easy-to-understand, easy-to-consume business intelligence solutions that users like and that meet real business needs.

How to achieve agile development

Agile BI projects should focus on people over process. This does not mean that Agile BI is inherently opposed to thorough and careful development processes, but is guided by a minimalistic business user focused approach, to streamline development cycles. There are three key agile processes that should be adhered to ensure an Agile BI rollout:

Iterative ‘Sprint’ development cycles

Systematize ongoing BI processes
Implement Barely Sufficient Processes

Iterative ‘Sprint’ development cycles

In Agile development methodology, teams work in ‘sprints’ to produce bite-sized deliverables in an iterative manner. Sprints can be one to four weeks long depending on the size and complexity of the project. At the end of each sprint, the business has a working deliverable, such as a new report or dashboard, delivered to them in a production setting.

By contrast, the waterfall development cycle used in traditional BI rollouts, is cemented in a regimented, sequential progression. It is inflexible to changing reporting needs and costly to make changes to. Agile BI, as a process, is about delivering functioning software regularly in short weekly or monthly timeframes – the shorter and more frequent the better (working software is the measure of BI success). Agile BI is about responding to the immediate needs of the BI user, rather than working to establish and deliver ALL potential reporting needs upfront. To establish an approach that facilitates Agile BI, reporting objectives should be readjusted regularly based on available resources and intermediate business goals to help focus attention on what is really important. Working in this manner ensures the relevancy of reporting to business goals and enables a faster, more flexible approach to changing reporting needs.

Systematize ongoing BI Processes

Agile BI development teams must automate any repetitive tasks/processes to allow more time and focus to be spent on developing and delivering end-user features. For example, whilst critical, testing the BI system manually takes up unacceptable resources each and every sprint cycle. Automated testing conducted by the users actually involved in the development of new reports, means that new changes can be quickly tested within the sprint, rather than waiting to be processed by a separate testing team. This way, accountability resides within the team.

Implement Barely Sufficient Processes

Minimizing the amount of ‘ceremony’ associated with BI development reduces the length of development cycles and allows development teams to concentrate on the work that matters. This minimalistic approach does not suggest that careful planning is unnecessary during the development process, but that formal planning and documentation should be aimed at satisfying the practical needs of the project. For example, a concept document for each sprint should focus on business user requirements and nothing more. Additional verbiage simply adds no value. Agile BI is just as much about maximizing the amount of work not done.

Summary

Successful Agile BI deployments enhance organizational flexibility and responsiveness. Increasingly, businesses and their personnel are exploiting the benefits of Agile BI, allowing them to respond with immediacy to business demands. To survive and prosper in a competitive marketplace, businesses from all industries and sectors have to be able to scan their external environment, review their internal processes and make appropriate proactive and reactive changes. Modern companies are striving to spread fact-based decision-making throughout their organizations. Agile BI solutions, with their end-user centric approach, enable organizations’ to anticipate and adapt to shifting market conditions.

References

Information Management Blog : Agile BI
Yellowfin : Agile Business Intelligence
Forrester report on Agile BI Out Of The Box
Balanced Insight: Enabling Agile Business Intelligence with Balanced Insight

Don’t forget to leave your comments below.

Business Analytics with In-Memory Databases

Written by Srikanth Chintamaneni on January 24, 2012. Posted in Articles.

Abstract

The Business intelligence (BI) and Data Warehouse vendors are increasingly turning to in-memory technology in place of traditional disk-based storage to speed up implementations and extend self-service capabilities.

For years, it has been noticed that the process of creating customer data queries and building business intelligence reports has been a prolonged activity. This is because the information needed must be pulled from operational systems and then controlled in separate analytical data warehouse systems that can accept the queries. Now, however, with the advent of true ‘in-memory analytics’, a technology that will allow operational data to be held in a single database that can handle all the day-to-day customer transactions and updates, as well as analytical requests – in virtually real time.

Starting Questions

Successful Business Analytics project implementations start by asking the right questions. Here are a few that should be on your short list.

· How do I manage and maintain the performance of my existing reports with the ever increasing data?

· What is the cost effective alternative to data warehouses that provide the ability to analyze very large data sets, but is much simpler to set up and administer?

· What can I do today to support near-real time reporting requirements and not relying heavily on IT departments?

· How can I demonstrate value to my company to extend real-time ad-hoc query capabilities for high volume transaction functionalities such as Financial Services?

· How do I minimize the administration overhead and yet provide a transparent reporting environment to end user?

The purpose of this article is to put both BI technologies in perspective, in-memory and disk-based, explain the differences between them, and finally explain, in simple terms, why disk-based BI technology is not on its way to extinction. Rather, explain the requisites for considering an in-memory database BI solution.

But before we get to that, let us understand the differences between disk-based and in-memory databases.

Disk-based and In-memory Databases

Database irrespective of disk-based or in-memory, we are talking about where the data resides while it is actively being queried by an application: with disk-based databases, the data is queried while stored on disk and with in-memory databases; the data being queried is first loaded into RAM (Random Access Memory).

Disk-based databases are engineered to efficiently query data residing on the hard drive. At a very basic level, these databases assume that the entire data cannot fit inside the relatively small amount of RAM available and therefore must have very efficient disk reads in order for queries to be returned within a reasonable time frame. On the other hand, in-memory databases work under the opposite assumption that the data can fit entirely inside the RAM. The engineers of in-memory databases benefit from utilizing the fastest storage system a computer has (RAM), but have much less of it at their disposal.

The fundamental trade-off with disk-based and in-memory technologies is faster reads and limited amounts of data versus slower reads and practically unlimited amounts of data. These are two critical considerations for business intelligence applications, as it is important both to have fast query response times and to have access to as much data as possible.

Fast analysis, better insight and rapid deployment with minimal IT involvement!

What is it?

As the name suggests, the key difference between conventional BI tools and in-memory products is that the former query data on disk while the later query data in random access memory (RAM). When a user runs a query against a typical data warehouse, the query normally goes to a database that reads the information from multiple tables stored on a server’s hard disk. With a server-based inmemory database, all information is initially loaded into memory. Users then query and interact with the data loaded into the machine’s memory.

BI with In-memory databases may sound like caching, a common approach to speeding query performance, but inmemory databases do not suffer from the same limitations. Caches are typically subsets of data, stored on and retrieved from disk (though some may load into RAM). The key difference is that the cached data is usually predefined and very specific, often to an individual query; but with an inmemory database, the data available for analysis is potentially as large as an entire data mart.

In-memory database is designed specifically to take advantage of the immense amount of addressable memory now available with the latest 64-bit operating systems. In-memory technology uses the multi-gigabytes of memory space available in 64-bit servers as for its data store. In-memory analysis is designed to improve the overall performance of a BI system as perceived by users, especially affecting complex queries that take a long time to process in the database or when accessing a very large database where all queries are hampered by the database size. With in-memory database, it allows data to be analyzed at both an aggregate and a detailed level without the time-consuming and costly step of developing ETL processes and data warehouses or building multidimensional OLAP cubes. Since data is kept in-memory, the response time of any calculation is lightning fast, even on extremely large data sets analyzed by multiple concurrent users.

This kind of immediate, interactive analysis is particularly important when people are trying to discover unknown patterns or learning new opportunities.

Who is it for?	Know your challenges	Finding the right mix
When selecting an in-memory solution consider one that operates seamlessly within an end-to-end BI platform where its usage is completely transparent to users and report developers Ideal for setting up departmental BI applications and for meeting the BI needs of small to medium sized businesses as it requires very little up-front effort, and no ETL Populated quickly from any database source, users can seamlessly use in-memory databases and associated meta-data layers as a source for many reports, dashboards, and analysis Look for technology that has been designed to avoid the excessive administrative burdens and can scale to enterprise levels in terms of user number, data security and data governance The leading benefits of Business analytics with in-memory databases are to deliver decision insight with the agility that businesses demand. It is a win for business users, who gain self-service analysis capabilities, and for IT departments, which can spend far less time on query analysis, cube building, aggregate table design, and other time- consuming performance-tuning tasks	Regardless of what fancy algorithm is used with an in-memory database, storing the entire dataset in RAM has a serious implication: the amount of data one can query with this technology is limited by the amount of free RAM available, and there will always be much less available RAM than available disk space Limited memory space means that the quality and effectiveness of the BI application will be hindered: the more historical data to which we have access and/or the more fields we can query, the better analysis, insight and, well, intelligence one can get to One could add more and more RAM, but then the required hardware becomes exponentially more expensive. Beyond 64GB, we can no longer use what is categorized as a personal computer but will require a full-blown server which brings us into very expensive computing territory Note that the amount of RAM required is dependent on the number of people simultaneously querying it. Having 5-10 people using the same in-memory BI application could easily double the amount of RAM required for intermediate calculations that need to be performed to generate the query results.	A key success factor in most BI solutions is having a large number of users, so we need to tread carefully when considering in-memory technology for real-world BI. Otherwise, the hardware costs may spiral beyond what the organization is willing or able to spend Some of these databases introduce additional optimizations which further improve performance. Most of them also employ compression techniques to represent even more data in the same amount of RAM The future of BI lies in technologies that leverage the respective benefits of both disk-based and in-memory technologies to deliver fast query responses and extensive multi-user access without huge hardware requirements. These types of technologies are not theoretical anymore and are already utilized by businesses worldwide. Some are designed to distribute different portions of complex queries across multiple cheaper computers (this is a good option for cloud-based BI systems) and some are designed to take advantage of 21st-century hardware (multi-core architectures, upgraded CPU cache sizes, etc.) to extract more juice from off-the-shelf computers

Summary

Business Analytics with in-memory database provides companies with a faster, more flexible, and arguably lower-cost way of accessing and processing information allowing users to get answers to business questions in seconds rather than hours. By virtue of its high performance architecture in-memory has the potential to help midsize organizations become more informed, agile and respond quicker to changing market conditions.

In addition, advances in technology and lower costs of memory and CPU make this type of technology more attractive than ever before. Matching the appropriate architectural approach with the kind of business analytics solutions needed by a midsize company has the potential to deliver benefits such as reduced time to insight, greater agility, increased self-service and lower overall IT demands.

References:

Open source In-memory Analytics – YellowFin
Extinction of traditional Business Intelligence: Elasticube Chronicles
In-Memory Data Management by Plattner/Zeier

Don’t forget to leave your comments below.

Srikanth Chintamaneni is a manager in the Information Management service line of Deloitte Consulting India Pvt. Ltd. He has over 13 years of experience in providing consulting services involving data warehouse and content management solutions in the Health care, Commercial & Consumer Finance, and Industrial Products industry segments. His capabilities support services involving data profiling, data modeling, report design, and end-to-end data warehouse implementations.