Spread Marts

spread mart (sprěd märt) noun.

1. a spreadsheet that is used for data integration

Spread marts solve some problems but introduce others.

---

In the latest issue of DM Review, there is an article Business Intelligence The Self-Service Way by Shailesh Kosambia from Tata Consultancy Services. It talks briefly about the data issues inhibiting self-service, saying "Business users do not have access to consolidated, unified data..." Shailesh continues, "When a data warehouse does not adequately cover all lines of business, the users have to get data from different sources and consolidate it in one place, leading to several spread marts within the same organization."

Claudia Imhoff discusses ways to solve the issues of "spread marts" in How to "Excel" in Your Business Intelligence Environment:

1. Being able to "follow the data"
2. Scheduled data updates
3. Expiration dates
4. Securing certain data
5. Preserving formats, formulas
6. Link back to live data
   

Why are people so hung up on using Excel?

Neil Raden of Smart (enough) Systems explains the infatuation with Excel, saying it's because it is "subversive". To paraphrase, spreadsheets allow users to do things behind the back of IT.

What if users could get all the benefits of using Excel (easy to use; data manipulation and combination; no IT needed) without the problems around transparency, latency, authority, and consistency?

To the man with a hammer, every problem looks like a nail. For another tool, see my previous article on Data Mashup.

Pervasive BI by TDWI

pervasive (pər-vā'sĭv) adj.

1. spreading or spread throughout
2. having the quality or tendency to become spread throughout all parts of

When pervasive is used as a modifier with BI, it is always this second definition. Often it is also just wishful thinking more than reality.

---

Wayne Eckerson of TDWI just published a new report: Pervasive Business Intelligence - Techniques and Technologies to Deploy BI on an Enterprise Scale. Along with the 35 page report, he also gave a webinar today highlighting some of the key ideas and points he raised.

One of the main ideas he put forward, as a way of expanding usage of BI tools, was that of information sandboxes. In these interactive dashboards, users can explore and analyze data, without performing traditional ad hoc report creation. I feel that Wayne is playing catch up, because Boris Evelson at Forrester published his research on BI Workspaces (he initially called them "analytical sandboxes") last month, and vendors have been providing this capability for over a year. Better late than never.

Also, to my pleasant surprise Wayne, Mr. Data Warehouse, acknowledged that the advent of new technologies can make a data warehouse optional. I am paraphrasing, but I am glad some of these BI traditionalists are no longer in denial.

Self-Service vs. IT?

symbiosis (sĭm'bē-ŏnt') noun.

1. any interdependent or mutually beneficial relationship between two persons, groups, etc.

Users need IT and vice versa.

---

The question mark in the title is on purpose. I believe it is an open question, with no right answer.

Just a few short years ago, I don't think anyone would have doubted the requirement for the IT department. Nowadays, with SaaS, appliances, and Web 2.0, many people/companies are.

Specifically in the BI space, Gartner published an article: Emerging Technologies Will Drive Self-Service Business Intelligence. Techworld cited it in a release: IT's role in business intelligence to lessen. The Gartner publication is also the focus of an article published today by DM Review called Gartner Says Emerging Technologies Will Marginalize IT's Role in Business Intelligence.

Aside: The focus of the article is on interactive visualization, and alternatives to data warehousing. I couldn't have made a better case for InetSoft products if I had written it myself.

I don't think the role of IT will be reduced. Rather, I believe that IT will spend less time on the mundane tasks like producing reports, and be able to focus on other, more advanced, BI projects. For example, predictive modeling, decision automation, and embedding BI in business processes are all still firmly in the realm of IT.

Post-pre-aggregation

heuristic (hyŏŏ-rĭs'tĭk) noun.

1. of, pertaining to, or based on experimentation, evaluation, or trial-and-error methods

The definition of multi-dimensional cubes is an exercise in successive approximation.

---

The traditional bottom-up design of pre-aggregated data is based on user input collected at the beginning of the project. Then, as they start to use the system, their feedback is incorporated into re-engineering efforts. This type of system has a lot of inertia, and can easily frustrate the user population.

Let's take a step back and remember why pre-aggregates are employed in the first place. Running a report or viewing a dashboard that is based on a large dataset will be slow if it is done on the fly. If we can anticipate the queries that will be executed, and run them before they are needed, we can give the appearance of a faster response.

I think the trouble enters in the interpretation of anticipate. The traditional model aims to address future usage in the data design stage. Consider an alternative of preparing at runtime.

Let me expand on this to clarify what I mean. In the traditional model, data is pre-aggregated, then reports are created. In the alternative model, reports are created, then data is pre-aggregated.

This means that the end products can be used to define what data is pre-calculated and how. If a report displays quarterly sales information, the sales data can be pre-aggregated at the quarter level. This takes the guesswork out of the process, guaranteeing that all scheduled summations are used, and that all end products can take advantage of preprocessing.

Build vs. Buy

opportunity cost (ŏp'ər-tōō'nĭ-tē kôst) noun.

1. the cost of an alternative that must be forgone in order to pursue a certain action
2. the benefits you could have received by taking an alternative action

The choice between build and buy requires more than simple arithmetic.

---

In many organizations, the choice between purchasing a software package and building your own comes down to license costs. This is a very myopic/nearsighted/shortsighted view.

Hopefully some of you are already thinking in terms of TCO. This is Total Cost of Ownership, and represents the entire cost, which is not so easy to calculate.

First, it is important to understand the difference between cost and outlay. An outlay is cash exchanged for acquisition or usage (license cost) and the concept of cost is a superset that includes outlays and other expenses (maintenance, salaries).

Next, we also have to consider opportunity cost. Consider the following two scenarios: purchasing a product that is easy to deploy and maintain; or having an in-house developer use a couple open source packages to build the desired functionality. It is clear that the second option wins the "lowest license cost" battle. In some cases, a new developer must be hired, in which case his or her salary must factor into the TCO of the second option. In other cases, when a developer is already on staff, there is the opportunity cost to consider. What else could the developer work on, instead of reinventing the wheel? What benefits could that project have yielded? If there is nothing else the developer could have done, then costs can be saved in the first scenario by reducing staff. There is also opportunity cost for the user population, because they must wait for the homegrown solution.

The last item that is a little more difficult to quantify is risk. This applies to the initial creation, where developing a project internally is more likely to fail or suffer from underestimated costs. It also applies to the risk going forward of supporting and maintaining the homegrown system without the help of a vendor.

I am not saying that buy is always better than build. It is important to accurately assess the options, considering the total costs over time.

Sea Change in BI

sea change (sē chānj) noun. idiomatic

1. profound transformation: "Full fathom five thy father lies: / Of his bones are coral made: / Those are pearls that were his eyes: / Nothing of him that doth fade / But doth suffer a sea change / Into something rich and strange." -Shakespeare
   
2. a striking change, as in appearance, often for the better

The data warehouse has gone from a means to an end.

It's time to step back and look at the forest.

---

In thinking about data mashup, I asked myself why more vendors don't provide it. My initial thought was that it was an innovation that was never thought of before, but I then humbled myself and returned to reality. I now believe a combination of two factors cause the lack of availability of data mashup.

First, my vision of user-driven data mashup would not be an effective standalone product. Think of using a separate application to pull/massage/transform/combine disparate data sources. Now, what is the result? In only a few cases is the query the end of the story. Typically that dataset now needs to be presented in an interesting/useful/appealing way. To have this dataset be the input to a separate BI product would require some integration point, like a web service, and a generic way of communicating a recordset of any size in both dimensions. This starts to get complex, and require skills that are possessed only by IT professionals.

Second, let's ask the question of why other BI vendors don't provide data mashup. It is not very hard to see that it is not because they can't, but rather they choose not to. The remainder of this article is devoted to this topic.

Tom Gonzalez wrote a blog entry called What is wrong with the Business Intelligence Industry?. As an "outsider", Tom brings a fresh perspective from his experience applying next generation technology to business problems.

Some industry analysts are even starting to challenge the status quo, which is refreshing.

Colin White wrote an article, Is Data Warehousing Essential to Business Intelligence?, that concludes, "No, it's not." He reminds us of the 5 issues that data warehouses address, and that these issues are solvable through other means.

At the risk of spreading an unconfirmed rumor, Claudia Imhoff, co-author of books on data warehousing, has allegedly admitted that some BI implementations do not require a DW. A baby step, but at least it's in the right direction.

More on this later.

Data Mashup Defined

mashup (māsh'ŭp') noun.

1. an audio recording that is a composite of samples from other recordings, usually from different musical styles: Danger Mouse created the Grey Album, which is a mashup of The Beatles' White Album and Jay-Z's Black Album.
2. the creation of a new work from two sources that were not initially designed to be combined
   

Apply this idea to the internet and you get things like Google Maps with Subway lines.

Apply this idea to data and you have an alternative to data warehousing.

---

Some have tried to define data mashup as data federation that makes use of web services, screen scraping, and even Yahoo! Pipes. This is all well and good, but doesn't really live up to the spirit of mashup because of the dependence on IT skills. After all, you wouldn't label ETL as "data mashup". I think that the key component that is missing is user involvement.

From a Business Intelligence perspective, users have had ad hoc query abilities, to some degree, for years. Imagine for a moment, a business user pulling together data from a sales system, a marketing database, and a local spreadsheet, creating a dataset that will help guide future decisions.

This is certainly possible if IT populated a data warehouse with all this information, or if IT built the federation infrastructure to present all this data as a virtual warehouse. Either way, it is a static definition that IT needs to build and maintain. Changes of a permanent nature are passed down from users to developers, and changes of a temporary/hypothetical nature are left unaddressable.

Enter what I call true Data Mashup. The user-defined variety that takes ad hoc query to the next level. It gives users the lego bricks, and lets them build a car, house, or Taj Mahal.

The IT staff will always be responsible for defining the lego bricks, but end users can play with the data in ways that are unanticipated. They can combine datasets that were not initially designed to be combined.