Why do Data Warehouses Take so Long and Cost so Much?

Background

Data Warehouses, particularly Enterprise Data Warehouses (EDWs), are increasingly necessary for organizations to compete in today’s marketplace. The insights and analyses provided by EDWs are extremely difficult, if not impossible to obtain from on-line transaction processing systems (OLTPs). These systems are geared toward a particular business process, usually duplicate data in other TPSs and have different data structures for the same business concept.

What is an EDW?

An EDW is simply a single, integrated, complete and consistent data store that contains data from most, if not all, major business processes. The data is made available to end users in a way that they can understand and use it in a business context.[1]

Unlike OLTPs, an EDW:

• Integrates data from many different OLTP systems into a single database
• Contains both current and historical data
• Organized by major business processed (Sales, Marketing, Finance, HR, etc.)
• Once data is in the EDW, it is not changed in place; rather new records record original content and historical content
• Detailed as well as summarized data
• Creates open environment to turn raw data into actionable information that is easily accessible to decision makers and analysts
• Helps the company make better business decisions making it more competitive, profitable and able to respond to changes much faster

What Organizations Typically say about Traditionally Developed EDWs

An EDW is a very powerful business tool. However, like many powerful tools, the traditional way of building an EDW takes a long time, requires a great deal of money and resources, and is very risky.

Here is some feedback from organizations that have built EDWs using traditional, manual methods:

• Development takes too long
• They cost too much
• They required too many resources
• It’s agonizingly difficult to integrate multiple transaction system data
• They can’t find people with the right skills and experience
• Once built, the EDW cannot respond to change quickly enough
• EDWs fall short of user expectations
• It is difficult to get information out of the EDW
• EDWs are horrifically costly to keep running after they are built due to inadequate architecture and design, lack of documentation and team turnover
• EDWs are just too complex due to so many tools and technologies involved
• EDWs are a high risk undertaking

The Shocking Truth about Data Warehouse Failure

It’s not easy to build a good enterprise data warehouse. EDWs are large, complex systems. Several industry analysts have stated that two-thirds of large-scaled projects fail to achieve their stated business goals, were delivered late or were substantially over budget. EDWs fall in this category.

What Organizations Want from a DW or an EDW

In today’s rapidly changing business climate, organizations want an EDW that gives them a competitive advantage or at the least makes it possible for them to survive. The main needs expressed by business leaders for an EDW include:

• A guarantee of success
• Need it yesterday
• Business data integrated into a single, trusted data store
• Want answers to any question across business processes
• High quality data
• Low total cost of ownership (TCO)
• 360° view of any person or organization that touches the company
• Respond quickly and cheaply to changes in business conditions or acquisitions
• Scalable
• Fast response

[1] Paraphrasing Barry Devlin, a thought leader in all things Data Warehouse.