3 min read
Data Warehousing for Dummies: Agile, Big Bang or Automation?
Written by: TimeXtender - February 9, 2015
In many cases a typical waterfall or "Big Bang" approach to data warehousing fails before the warehouse is even completed.
Why does my business need to follow an agile approach to data warehousing?
There are a couple of factors to consider when developing and implementing a data warehouse. First of all it takes a lot of time, people and effort to spec a warehouse based on business requirements, business rules and user input. And this is based on the current state of the company, seldomly taking into account where the business is heading. Not to mention the cost involved to get a pretty dashboard or report delivered an exec or board members.
Big Bang vs. Agile
Based on the complexity of the warehouse it can take anything from 6 to 18 months and even in some cases several years, to build this warehouse. Where in this time the business has not seen any reports or dashboards for their quite expensive investment. On top of that, the business has evolved and might not have the same requirements they did 6 to 18 months ago.
By following an agile approach to data warehousing, allows the data warehouse to evolve with the business, by having relevant information based on current business requirements and answers current business pains. This allows the business to see results in a short space of time, instead of waiting 6 to 18 months to see results. In many cases a typical waterfall or "Big Bang" approach to data warehousing fails before the warehouse is even completed, due to business pulling the plug on an investment they feel they are not getting any return.
The Big Bang approach is also not Industry, company size or vertical specific. Many companies have tried this approach, ending up with long development cycles, costly projects and worst of all very little to no delivery. I came across a great article that illustrates this scenario perfectly. A well-known Retailer went for the Big Bang approach only to discover; reports were often delayed, data quality was hampered by hand-coded business rules, there was inconsistent data quality, data architecture remained in silos instead of being integrated, and there was a lack of full disaster recovery for data.
After pumping up the infrastructure to support all of the above pains, they were none the wiser. In addition to all of this it would also take another 3 years to fix this and have a full functional warehouse in place.
The Agile Approach
According to Wikipedia, Agile BI is defined as the following:
"Agile Business Intelligence (BI) refers to the use of the agile software development methodology for BI projects to reduce the time-to-value of traditional BI and helps in quickly adapting to changing business needs. Agile BI enables the BI team and managers to make better business decisions.
Agile methodology works on the iterative principle; this provides the new features of the software to the end users sooner than the traditional waterfall process which delivers only the final product. With Agile, the requirements and design phases overlap with development, thus reducing the development cycles for faster delivery. It promotes adaptive planning, evolutionary development and delivery, a time-boxed iterative approach, and encourages rapid and flexible response to change.
Agile BI encourages business users and IT professionals to think about their data differently and it characterized by low Total Cost of Change (TCC). With agile BI, the focus is not on solving every BI problem at once but rather on delivering pieces of BI functionality in manageable chunks via shorter development cycles and documenting each cycle as it happens. Many companies fail to deliver the right information to the right business managers at the right time."
So What Does This All Mean?
By following an agile approach to data warehousing, the retailer covered nine key business areas namely: sales, products, customers, prices, promotions, buyers, brands, calendars and locations. And all of this to be completed in just six-months compared to the previous three-year forecast.
Agile data warehouse development is performed in multiple sprints, involving the business user throughout the development for constant feedback. The process is iterative, allowing user feedback to be taken into account and the development addressing the users input on a frequent basis. Agile data warehouse development has proven in many cases that is has less Total Cost of Ownership (TCO) implication compared to the traditional Big Bang approach.
A Data Warehouse Automation (DWA) tool can get you there in six-weeks or less. That allows you to take full advantage of your ERP data with pre-built business analytics. TimeXteder's DWA platform allows end users to plug into their ERP and build a Star schema warehouse within a day. With a drag and drop interface, based on the Microsoft platform, a staging area, warehouse and subject area cubes that allows you to pivot, slices and dice and drill into your data for complete business analytics.