These days it seems like when people aren’t spending their free time trying to figure out what the Cloud really means to them and their organizations, they are spending the rest of their time trying to determine what “Big Data” will mean to their organization and to them. Okay, maybe that is an exaggeration but it is obvious to the casual observer that Big Data gets a significant amount of airtime in the media these days.
This is evidenced from some “Big Data” related quotes from a Dilbert cartoon
People talk about Big Data in business circles but it is clear that many people don’t really understand what the concern is all about. So what is Big data and why could it be so important? Big data is a collection of data sets that are very complicated and are of such a large scale that they become difficult to work with using standard database management tools. Big Data can be identified by following characteristics, Volume, Variety, Velocity, Variability and Veracity. Volume, Variety, and Variability are pretty much self explanatory (How much data, how many different data types, and how much variation do we have within each type of data?). Velocity has to do with the speed of generation of data. Veracity has to do with the quality of the data actually being collected. Veracity is one of the most important elements of your data because even if you have collected lots of data, if it is not good quality data, then any analysis you may do on it may not be that helpful.
We have been warned that we must pay attention to Big Data because it is about to change or shift the way many organizations do business. Today’s computers and servers have the ability to manage more transactions, handle larger amounts of data, and process a greater number of calculations than ever before. While that in itself is significant, it begs the question, what does an individual or company do with all this data? It seems great we have all this data, but when we have so much of it what does it really do for us?
Getting value from this data requires us to have some vision or purpose for the data we are collecting. The people interpreting the data need to understand the underlying fundamentals of the business as well as what the available data is able to tell us. Just collecting more and more data in itself may not provide any additional value. Hiring an expert in data such as a data scientist/specialist who cannot properly interact with your employees, who actually understand the business, may not provide the value your organization is looking for. Gartner recently predicted that “through 2017, 60% of big data projects will fail to go beyond piloting and experimentation and will be abandoned.” This is significant in that just because you invest in Big Data doesn’t mean you are guaranteed to get value from your investment.
So what does this all mean for Commodity Trading organizations and their Commodity Trading and Risk Management (CTRM) systems ? They are already collecting lots of data internally in the CTRM systems and additionally the Trading Companies may be accessing external data providers and amassing even more useful data. As mentioned above the data must have a purpose in order for it to become useful information. We know that some of the larger oil and gas companies believe they can trade and make more money if they have more and better information. In a recent Bloomberg article it stated that even with the current low oil prices, BP’s trading arm was able to boost profits. One interesting quote from that article was the following: “The in-house trading arms of oil companies take speculative positions from time to time, profiting from what BP describes as the “huge amount of information” about supply and demand that others do not have access to. Oil prices rose during the quarter as U.S. shale companies idled oil rigs, giving traders with better intelligence on the industry an opportunity to profit.”
That of course brings us to another question. Can every trading firm obtain that same amount of data in order to generate worthwhile information? Can the cost to obtain that data actually outweigh the potential benefits? I would suspect that it takes a certain critical mass of trading operations to justify the cost of obtaining the required data to give the organization a significant competitive advantage. So maybe big data isn’t for everyone. Perhaps in some sectors Big Data is really only for the Big Players in that industry.
- On May 25, 2015