Lesson 7: Introduction to Data Mining

Motivate...

As we near the end of this course, I’d like to provide a bigger picture of how we can utilize all of the tools we’ve learned so far. For this lesson, nothing new will be presented. Instead, we will combine all the statistical metrics we have already learned into one process - a simplified yet effective template for data mining. Please note that there are many ways to go about data mining; this lesson will only present one method.

Before we talk about data mining, I want to present a general motivation for the case study we will focus on. In this lesson, we are going to look at corn. As an agricultural product, you might think that weather (and climate) play a big part in the market. But there are two reasons why this is not necessarily true.

The first is that corn has been genetically altered many times to withstand the extremes of weather. In particular, ongoing research has created drought-tolerant corn hybrids. The second part of this problem is that corn, and in particular corn prices, are largely governed by the futures market. Understanding futures markets is not a requirement for this class, but if we want to ask the right questions, we need to learn a little about all the aspects of the problem. Check out this video that describes the futures markets.

Click for transcript of Futures Market Explained.

PRESENTER: A $3 box of corn cereal stays at roughly the same price day to day and week to week. But corn prices can change daily, sometimes by a few cents, sometimes by a lot more. Why does the cost of processed food generally stay quite stable, even though the crops that go into them have prices that fluctuate?

It's partly thanks to the futures market. The futures market allows the people who sell and buy large quantities of corn to insulate you, the consumer, from those changes without going out of business themselves. Let's meet our corn producer, this farmer. Of course, she's always looking to sell her corn at a high price. And on the other side, our corn user. This cereal company is always looking to buy corn at a low price.

Now the farmer has a little bit of a problem because her whole crop gets harvested at once. Lots and lots of farmers will be harvesting at the same time, and the huge supply can send the price falling. And even though that price might be appealing to the company that makes cereal from corn, it doesn't want to purchase all of its corn at once because, among other reasons, it would have to pay to store it.

[CASH REGISTER SOUND EFFECTS]

But it's fortunate that corn can be stored because that means it can be sold and bought throughout the year. And this is where the futures market fits in. Buyers and sellers move bushels around in the market, though actual corn rarely changes hands. Instead of buying and selling corn, the farmer and cereal maker buy and sell contracts. Now, we're getting closer to peace of mind for both sides because a futures contract provides a hedge against a change in the price. This way, neither side is stuck with only whatever the market price is when they want to buy or sell.

These contracts can be made at any time, even before the farmer plants the corn. She'll use the futures market to sell some of her anticipated crop on a certain date in the future. Of course she's not going to sell all of her corn on that contract, just enough corn to reassure her that a low price at harvest won't ruin her business. The contract provides that security. The cereal company uses the same market to buy bushels. Their contract protects against a high price later. Contracts will gain or lose money in the futures market.

If the price goes high, the farmer loses money on that futures contract. Because she's stuck with it. But that's OK. Because now, she can sell the rest of her corn-- what wasn't in that contract-- at the higher price. That offsets her loss in the futures market. If, at harvest time, the price of corn is low, well, that's exactly why she entered the futures market. The low price means her contract makes money. So that profit shields her from the sting of the low price she'll get for the bushels she sells now.

The corn cereal company doesn't like those higher prices, and that's why they have a futures contract. They make money on it and can use that profit to cover the higher price of the corn they now need to buy. The futures market serves as a risk management tool. It doesn't maximize profit. Instead, it focuses on balance. And in this way, it keeps your cereal from breaking your weekly shopping budget.

The futures market is regulated more by contracts than by supply and demand. Weather, however, does play an important role (source: "Investing Seasonally In The Corn Market(link is external)"). If we can predict the amount of corn that will be produced or the amount of acreage that will be harvestable (not damaged from weather), that information could be used to help farmers decide how many contracts to sell now or whether a company should wait to buy a contract because the supply will be larger (greater harvest) than anticipated.

Our goal for this lesson is to determine how much weather impacts the corn supply (in the form of harvest, yield, price, etc.), and if certain weather events can be used to predict the corn harvest, providing actionable information on whether to buy or sell a futures contract.