Wikipedia explains “Cloud Computing” as;

Cloud computing is Internet-based computing, whereby shared resources, software, and information are provided to computers and other devices on demand, like the electricity grid. However, the analogy to utility computing is not entirely correct, as discussed here. Cloud computing is a paradigm shift following the shift from mainframeto client–server in the early 1980s. Details are abstracted from the users, who no longer have need for expertise in, or control over, the technology infrastructure “in the cloud” that supports them. Cloud computing describes a new supplement, consumption, and delivery model for IT services based on the Internet, and it typically involves over-the-Internet provision of dynamically scalable and often virtualized resources. It is a byproduct and consequence of the ease-of-access to remote computing sites provided by the Internet. This frequently takes the form of web-based tools or applications that users can access and use through a web browser as if it were a program installed locally on their own computer. NIST provides a somewhat more objective and specific definition here. The term “cloud” is used as a metaphor for the Internet, based on the cloud drawing used in the past to represent the telephone network, and later to depict the Internet in computer network diagrams as anabstraction of the underlying infrastructure it represents. Typical cloud computing providers deliver commonbusiness applications online that are accessed from another Web service or software like a Web browser, while the software and data are stored on servers.

Most cloud computing infrastructures consist of services delivered through common centers and built on servers. Clouds often appear as single points of access for all consumers’ computing needs. Commercial offerings are generally expected to meet quality of service (QoS) requirements of customers, and typically include SLAs. The major cloud service providers include Microsoft,Salesforce, Skytap, HP, IBM, Amazon and Google.

The real question is; actually most of the companies were doing it for years. But they did not name these services after “Internal Cloud” or “application host”…

Data Warehouse is a repository (collection of resources that can be accessed to retrieve information) of an organization’s electronically stored data, designed to facilitate reporting and analysis. More simply, a data warehouse is a collection of a large amount of data.  However, the means to retrieve and analyze data, to extract, transform and load (ETL) data, and to manage the data dictionary are also considered essential components of a data warehousing system. Since 1950′s companies are collecting all kinds of data, to create their foundation of base information. Collecting all these data and generating information islands out of it, is enough to complete this complex task, or do we need to go one more step?

After all this precious efforts, some of these important collections are going to trash, or loose its integrity, through the fast changing medias. Some of them archived to somewhere, that we did not even know that, they exist! What are we creating for all these years…. Most important question is “Why are we collecting them for”?

a) Just for the sake of having the “data” in house? or, b) To own the “structured information” for our future benefits?, or c) To structure the information, to create “useful and classified intelligence” for specific needs?

This is an expensive investment, isn’t it? Who is guiding this initiative? So many questions to answer, here. I’d like to open this discussion to all of you out there, to see how far we can prove this is a hard task to accomplish… Or it’s impossible to build it, the way that it’s been described. Please find the detailed explanation of Data Warehousing below, that’s been collected from different resources, brfore posting your comments…

*************************************** / ******************************************

History

The concept of data warehousing dates back to the late 1980s when IBM researchers Barry Devlin and Paul Murphy developed the “business data warehouse”. In essence, the data warehousing concept was intended to provide an architectural model for the flow of data from operational systems to decision support environments. The concept attempted to address the various problems associated with this flow, mainly the high costs associated with it. In the absence of a data warehousing architecture, an enormous amount of redundancy was required to support multiple decision support environments. In larger corporations it was typical for multiple decision support environments to operate independently. Though each environment served different users, they often required much of the same stored data. The process of gathering, cleaning and integrating data from various sources, usually from long-term existing operational systems (usually referred to as legacy systems), was typically in part replicated for each environment. Moreover, the operational systems were frequently reexamined as new decision support requirements emerged. Often new requirements necessitated gathering, cleaning and integrating new data from “data marts” that were tailored for ready access by users.

Key developments in early years of data warehousing were:

1960s — General Mills and Dartmouth College, in a joint research project, develop the terms dimensions and facts.[3]

1970s — ACNielsen and IRI provide dimensional data marts for retail sales.[3]

1983 — Teradata introduces a database management system specifically designed for decision support.

1988 — Barry Devlin and Paul Murphy publish the article An architecture for a business and information systems in IBM Systems Journal where they introduce the term “business data warehouse”.

1990 — Red Brick Systems introduces Red Brick Warehouse, a database management system specifically for data warehousing.

1991 — Prism Solutions introduces Prism Warehouse Manager, software for developing a data warehouse.

1991 — Bill Inmon publishes the book Building the Data Warehouse.

1995 — The Data Warehousing Institute, a for-profit organization that promotes data warehousing, is founded.

1996 — Ralph Kimball publishes the book The Data Warehouse Toolkit.

2000 — Daniel Linstedt releases the Data Vault, enabling real time auditable Data Warehouses.

Architecture

Architecture, in the context of an organization’s data warehousing efforts, is a conceptualization of how the data warehouse is built. There is no right or wrong architecture, but rather there are multiple architectures that exist to support various environments and situations. The worthiness of the architecture can be judged from how the conceptualization aids in the building, maintenance, and usage of the data warehouse.

One possible simple conceptualization of a data warehouse architecture consists of the following interconnected layers:

Operational database layer

The source data for the data warehouse — An organization’s Enterprise Resource Planning systems fall into this layer.

Data access layer

The interface between the operational and informational access layer — Tools to extract, transform, load data into the warehouse fall into this layer.

Metadata layer

The data directory — This is usually more detailed than an operational system data directory. There are dictionaries for the entire warehouse and sometimes dictionaries for the data that can be accessed by a particular reporting and analysis tool.

Informational access layer

The data accessed for reporting and analyzing and the tools for reporting and analyzing data — Business intelligence tools fall into this layer. The Inmon-Kimball differences about design methodology, discussed later in this article, have to do with this layer

Conforming information

Another important fact in designing a data warehouse is which data to conform and how to conform the data. For example, one operational system feeding data into the data warehouse may use “M” and “F” to denote sex of an employee while another operational system may use “Male” and “Female”. Though this is a simple example, much of the work in implementing a data warehouse is devoted to making similar meaning data consistent when they are stored in the data warehouse. Typically, extract, transform, load tools are used in this work.

Master Data Management has the aim of conforming data that could be considered “dimensions”.

Normalized versus dimensional approach for storage of data

There are two leading approaches to storing data in a data warehouse — the dimensional approach and the normalized approach.

In a dimensional approach, transaction data are partitioned into either “facts”, which are generally numeric transaction data, or “dimensions”, which are the reference information that gives context to the facts. For example, a sales transaction can be broken up into facts such as the number of products ordered and the price paid for the products, and into dimensions such as order date, customer name, product number, order ship-to and bill-to locations, and salesperson responsible for receiving the order. A key advantage of a dimensional approach is that the data warehouse is easier for the user to understand and to use. Also, the retrieval of data from the data warehouse tends to operate very quickly. The main disadvantages of the dimensional approach are:

In order to maintain the integrity of facts and dimensions, loading the data warehouse with data from different operational systems is complicated, and

It is difficult to modify the data warehouse structure if the organization adopting the dimensional approach changes the way in which it does business.

In the normalized approach, the data in the data warehouse are stored following, to a degree, database normalization rules. Tables are grouped together by subject areas that reflect general data categories (e.g., data on customers, products, finance, etc.). The main advantage of this approach is that it is straightforward to add information into the database. A disadvantage of this approach is that, because of the number of tables involved, it can be difficult for users both to:

join data from different sources into meaningful information and then

access the information without a precise understanding of the sources of data and of the data structure of the data warehouse.

These approaches are not mutually exclusive, and there are other approaches. Dimensional approaches can involve normalizing data to a degree.

Top-down versus bottom-up design methodologies

Ralph Kimball, a well-known author on data warehousing,is a proponent of an approach to data warehouse design which he describes as bottom-up.

In the bottom-up approach data marts are first created to provide reporting and analytical capabilities for specific business processes. Though it is important to note that in Kimball methodology, the bottom-up process is the result of an initial business oriented Top-down analysis of the relevant business processes to be modelled.

Data marts contain, primarily, dimensions and facts. Facts can contain either atomic data and, if necessary, summarized data. The single data mart often models a specific business area such as “Sales” or “Production.” These data marts can eventually be integrated to create a comprehensive data warehouse. The integration of data marts is managed through the implementation of what Kimball calls “a data warehouse bus architecture”. The data warehouse bus architecture is primarily an implementation of “the bus” a collection of conformed dimensions, which are dimensions that are shared (in a specific way) between facts in two or more data marts.

The integration of the data marts in the data warehouse is centered on the conformed dimensions (residing in “the bus”) that define the possible integration “points” between data marts. The actual integration of two or more data marts is then done by a process known as “Drill across”. A drill-across works by grouping (summarizing) the data along the keys of the (shared) conformed dimensions of each fact participating in the “drill across” followed by a join on the keys of these grouped (summarized) facts.

Maintaining tight management over the data warehouse bus architecture is fundamental to maintaining the integrity of the data warehouse. The most important management task is making sure dimensions among data marts are consistent. In Kimball’s words, this means that the dimensions “conform”.

Some consider it an advantage of the Kimball method, that the data warehouse ends up being “segmented” into a number of logically self contained (up to and including The Bus) and consistent data marts, rather than a big and often complex centralized model. Business value can be returned as quickly as the first data marts can be created, and the method gives itself well to an exploratory and iterative approach to building data warehouses. For example, the data warehousing effort might start in the “Sales” department, by building a Sales-data mart. Upon completion of the Sales-data mart, The business might then decide to expand the warehousing activities into the, say, “Production department” resulting in a Production data mart. The requirement for the Sales data mart and the Production data mart to be integrable, is that they share the same “Bus”, that will be, that the data warehousing team has made the effort to identify and implement the conformed dimensions in the bus, and that the individual data marts links that information from the bus. Note that this does not require 100% awareness from the onset of the data warehousing effort, no master plan is required upfront. The Sales-data mart is good as it is (assuming that the bus is complete) and the production data mart can be constructed virtually independent of the sales data mart (but not independent of the Bus).

If integration via the bus is achieved, the data warehouse, through its two data marts, will not only be able to deliver the specific information that the individual data marts are designed to do, in this example either “Sales” or “Production” information, but can deliver integrated Sales-Production information, which, often, is of critical business value. An integration (possibly) achieved in a flexible and iterative fashion.

Bill Inmon, one of the first authors on the subject of data warehousing, has defined a data warehouse as a centralized repository for the entire enterprise. Inmon is one of the leading proponents of the top-down approach to data warehouse design, in which the data warehouse is designed using a normalized enterprise data model. “Atomic” data, that is, data at the lowest level of detail, are stored in the data warehouse. Dimensional data marts containing data needed for specific business processes or specific departments are created from the data warehouse. In the Inmon vision the data warehouse is at the center of the “Corporate Information Factory” (CIF), which provides a logical framework for delivering business intelligence (BI) and business management capabilities.

Inmon states that the data warehouse is:

Subject-oriented

The data in the data warehouse is organized so that all the data elements relating to the same real-world event or object are linked together.

Non-volatile

Data in the data warehouse are never over-written or deleted — once committed, the data are static, read-only, and retained for future reporting.

Integrated

The data warehouse contains data from most or all of an organization’s operational systems and these data are made consistent.

Time-variant

The top-down design methodology generates highly consistent dimensional views of data across data marts since all data marts are loaded from the centralized repository. Top-down design has also proven to be robust against business changes. Generating new dimensional data marts against the data stored in the data warehouse is a relatively simple task. The main disadvantage to the top-down methodology is that it represents a very large project with a very broad scope. The up-front cost for implementing a data warehouse using the top-down methodology is significant, and the duration of time from the start of project to the point that end users experience initial benefits can be substantial. In addition, the top-down methodology can be inflexible and unresponsive to changing departmental needs during the implementation phases.

Hybrid design

Over time it has become apparent to proponents of bottom-up and top-down data warehouse design that both methodologies have benefits and risks. Hybrid methodologies have evolved to take advantage of the fast turn-around time of bottom-up design and the enterprise-wide data consistency of top-down design.

Data warehouses versus operational systems

Operational systems are optimized for preservation of data integrity and speed of recording of business transactions through use of database normalization and an entity-relationship model. Operational system designers generally follow the Codd rules of database normalization in order to ensure data integrity. Codd defined five increasingly stringent rules of normalization. Fully normalized database designs (that is, those satisfying all five Codd rules) often result in information from a business transaction being stored in dozens to hundreds of tables. Relational databases are efficient at managing the relationships between these tables. The databases have very fast insert/update performance because only a small amount of data in those tables is affected each time a transaction is processed. Finally, in order to improve performance, older data are usually periodically purged from operational systems.

Data warehouses are optimized for speed of data analysis. Frequently data in data warehouses are denormalised via a dimension-based model. Also, to speed data retrieval, data warehouse data are often stored multiple times—in their most granular form and in summarized forms called aggregates. Data warehouse data are gathered from the operational systems and held in the data warehouse even after the data has been purged from the operational systems.

Evolution in organization use

Organizations generally start off with relatively simple use of data warehousing. Over time, more sophisticated use of data warehousing evolves. The following general stages of use of the data warehouse can be distinguished:

Offline Operational Data Warehouse

Data warehouses in this initial stage are developed by simply copying the data off of an operational system to another server where the processing load of reporting against the copied data does not impact the operational system’s performance.

Offline Data Warehouse

Data warehouses at this stage are updated from data in the operational systems on a regular basis and the data warehouse data are stored in a data structure designed to facilitate reporting.

Real Time Data Warehouse

Data warehouses at this stage are updated every time an operational system performs a transaction (e.g. an order or a delivery or a booking).

Integrated Data Warehouse

These data warehouses assemble data from different areas of business, so users can look up the information they need across other systems.

Benefits

Some of the benefits that a data warehouse provides are as follows:

A data warehouse provides a common data model for all data of interest regardless of the data’s source. This makes it easier to report and analyze information than it would be if multiple data models were used to retrieve information such as sales invoices, order receipts, general ledger charges, etc.

Prior to loading data into the data warehouse, inconsistencies are identified and resolved. This greatly simplifies reporting and analysis.

Information in the data warehouse is under the control of data warehouse users so that, even if the source system data are purged over time, the information in the warehouse can be stored safely for extended periods of time.

Because they are separate from operational systems, data warehouses provide retrieval of data without slowing down operational systems.

Data warehouses can work in conjunction with and, hence, enhance the value of operational business applications, notably customer relationship management (CRM) systems.

Data warehouses facilitate decision support system applications such as trend reports (e.g., the items with the most sales in a particular area within the last two years), exception reports, and reports that show actual performance versus goals.

Disadvantages

There are also disadvantages to using a data warehouse. Some of them are:

Data warehouses are not the optimal environment for unstructured data.

Because data must be extracted, transformed and loaded into the warehouse, there is an element of latency in data warehouse data.

Over their life, data warehouses can have high costs.

Data warehouses can get outdated relatively quickly. There is a cost of delivering suboptimal information to the organization.

There is often a fine line between data warehouses and operational systems. Duplicate, expensive functionality may be developed. Or, functionality may be developed in the data warehouse that, in retrospect, should have been developed in the operational systems.

Sample applications

Some of the applications data warehousing can be used for are:

Decision support

Trend analysis

Financial forecasting

Churn Prediction for Telecom subscribers, Credit Card users etc.

Insurance fraud analysis

Call record analysis

Logistics and Inventory management

Agriculture

The future

Data warehousing, like any technology, has a history of innovations that did not receive market acceptance.

A 2009 Gartner Group paper predicted these developments in business intelligence/data warehousing market.

Because of lack of information, processes, and tools, through 2012, more than 35 percent of the top 5,000 global companies will regularly fail to make insightful decisions about significant changes in their business and markets.

By 2012, business units will control at least 40 percent of the total budget for business intelligence.

By 2010, 20 percent of organizations will have an industry-specific analytic application delivered via software as a service as a standard component of their business intelligence portfolio.

In 2009, collaborative decision making will emerge as a new product category that combines social software with business intelligence platform capabilities.

By 2012, one-third of analytic applications applied to business processes will be delivered through coarse-grained application mashups.

References

Inmon, W.H. Tech Topic: What is a Data Warehouse? Prism Solutions. Volume 1. 1995.

“The Story So Far”. 2002-04-15. http://www.computerworld.com/databasetopics/data/story/0,10801,70102,00.html. Retrieved 2008-09-21.

Kimball 2002, pg. 16

Kimball 2002, pg. 310

“The Bottom-Up Misnomer”. 2003-09-17. http://www.intelligententerprise.com/030917/615warehouse1_1.jhtml. Retrieved 2008-11-05.

Ericsson 2004, pp. 28-29

“Data Warehouse”. http://www.tech-faq.com/data-warehouse.html.

Yang, Jun. WareHouse Information Prototype at Stanford (WHIPS). [1]. Stanford University. July 7, 1998.

Caldeira, C. “Data Warehousing — Conceitos e Modelos”. Edições Sílabo. 2008. ISBN 978-972-618-479-9

Abdullah, Ahsan (2009). “Analysis of mealybug incidence on the cotton crop using ADSS-OLAP (Online Analytical Processing) tool , Volume 69, Issue 1″. Computers and Electronics in Agriculture: 59–72. http://dx.doi.org/10.1016/j.compag.2009.07.003.

Pendse, Nigel and Bange, Carsten “The Missing Next Big Things”, http://www.olapreport.com/Faileddozen.htm

“Gartner Reveals Five Business Intelligence Predictions for 2009 and Beyond”, http://www.gartner.com/it/page.jsp?id=856714

We are professionals. What is that mean, for crying out loud! Can I do anything, because I’m a professional? Absolutely not.  But insurance companies are doing the same thing for years and selling the invisible future by scaring the hell out of us!!! As my father always said; (god rest his soul) “Bad example is not an example”. Without going any further, let’s stop here for the insurance of any kind.

Well, nobody is in the business to loose money. That’s for sure. What are we missing here? Are we corrupting the meaning of the technology and our professions here? That is not my intent. I’d like to point some un-resolved pain points here.

By the way, what is technology? Let’s have a quick look;

Technology is the usage and knowledge of tools, techniques, crafts, systems or methods of organization. The word technology comes from the Greek “technología”

Technology has affected society and its surroundings in a number of ways. In many societies, technology has helped develop more advanced economies (including today’s global economy) and has allowed the rise of a leisure class. Many technological processes produce unwanted by-products, known as pollution, and deplete natural resources, to the detriment of the Earth and its environment. Various implementations of technology influence the values of a society and new technology often raises new ethical questions. Examples include the rise of the notion of efficiency in terms of human productivity, a term originally applied only to machines, and the challenge of traditional norms.

Philosophical debates have arisen over the present and future use of technology in society, with disagreements over whether technology improves the human condition or worsens it.  Indeed, until recently, it was believed that the development of technology was restricted only to human beings, but recent scientific studies indicate that other primates and certain dolphin communities have developed simple tools and learned to pass their knowledge to other generations. So be careful from now on. There is another group challenging the human race here… Ok, I need to stop now. This is going too deep….. Let’s focus on Computer Technologies….

Computing is described as, ability to generate a process of tasks with in an assigned protocol stack. A computer is a programmable machine that receives input, stores and manipulates data, and provides output in a useful format. The main idea of computing is, to take human factor out of the equation from day to day processes. Tie these processes to a continuous routine and assign it to a computer…. Did you ever realize that, your quartz watch is a computer. Your timer on the oven. Or,  your radio… Doing the same thing over and over again, without adding or subtracting any comments or complaining about it. No comments about married couples here… At least, you know what you will get, at the end of the day. Everything is black and White. No shades of Gray. This is what we called “infrastructure” or “foundation” at the real life. Foundation… This has to be rock solid to build anything on top of it. Isn’t it?

Can you imagine a society controlled by the machines, that they have their artificial learning curve in place? The logic will change from “may be” to “True or False” statements. Similar to executive management’s mind set “why fix it, if  it’s not broken? What about the progress, learning, productivity, creativity kind of aspects? People can make mistakes. Thank you, but no thank you. Just keep the status Quo. No reason to improve, please come and join me on my hamster wheel of “Corporate America”. That is the bottle neck of Corporate America. It came to a point that we, shoot ourselves at the foot, by trying to create ourselves a job security in this dry process of “keep the lights up”. Yes, we are talking too much but, not delivering the result oriented technology to our clients to ease the processes….. That is why “Human Resource” departments are adding lines to their job applications like;

“Are you a Computer User?” …….

Have you ever seen a question on your application form like, “Are you an elevator user?” or, are you a telephone user? You just pick up the phone and dial the number, or press the numbered floor button on the elevator. It is a part of our daily life. That SIMPLE !!!!!

To make the long story short; as all IT professionals, we need to focus on one thing, and one thing only; Being able to create the simple’s best ! Don’t be mistaken, simple is the hardest thing to create. Focusing on this initiative will serve the purpose of positive technology and may be, we’ll be forgiven by the Executives, Wall Street, Nasdaq or Small businesses and get back to track after “Y2K” fiasco.

————————————————————-

Resources: http://en.wikipedia.org/wiki/Technology

Do we really realize that, everything that’s generating traffic in between electronic circuits on the board are virtual ….. All protocol stacks are re-configured and re-structured from assembly to GUI (graphical user interface) for “us” to understand.

Let me simplify the word “virtualization”. Hardware virtualization is simulation of the physical characteristics of a computer, operating system and/or application (all different topics) on a physical computing platform. Basically, putting different fruits or vegetables on the same basket, by segregating them from each other. This is done by, carving out spaces from a physical hardware and consolidate more than one “server and/or workstation attached to a simplified protocol stack (kernel).  This HAL (hardware abstraction layer) controls the way that data travels in between the resources. This idea came into existence in 1960 by IBM experimental IBM M44/44X system.

A hardware abstraction layer (HAL) is an abstraction layer, implemented in software, between the physical hardware of a computer and the software that runs on that computer. Its function is to hide differences in hardware from most of the operating system kernel (masking), so that most of the kernel-mode code does not need to be changed to run on systems with different hardware. With this capability, we can generate platform agnostic operating systems on the same physical platform.

Keep in mind, we used to be designing virtualization at the software layer. In these days, we are configuring the foundation at BIOS (basic input output system) level, to allocate more resources to detail and expand the capabilities of the Server Virtualization layer.

IBM “P”/”Z”, Intel x86/x64/itenium, AMD and RISK processors are supporting the hyper-vision on the processor level now. This gives us power to MIGRATE…  Magic word for the CFO’s and CIO’s do not forget. ROI and TTM is the key. Maintenance ratio goes from 1 to 15 to 1 to 55… What a big saving…. Not so fast! What about the storage? Oh, may be that’s why EMC bought out VMware…

I can go so deeper on this topic. But first of all, I’d like to know, if there is a group out there, really would like to read about this initiative. Because, it’s a very detailed and bulky subject to digest. Virtualization is the growing platform for Cloud Computing foundation. I’d like to combine these two topics under this umbrella. Anyhow, I’ll post more by dividing it into small chapters. if, I receive enough request for it.

Let’s see…

This is our Global Gateway  Site. According to the level of our service contract, our customers have their own secure portal sites, which they can share, store, index, and collaborate. These portals are physically segregated information portals, and they have connected to their own “Operational Database” ODB’s…

Please leave your connection information. Our team will reach out to you and, fill you in with the detailed information.