Hi, in this video I'm going to go over the data set that's going to be used for the next several hypothesis testing videos and used throughout all the different demonstrations of different forms of hypothesis testing that's coming up. So, this data set comes from ERCOT and contains the power generation for different sources for two days from the Texas power crisis in February of 2021, those two days are February 14th through 15th. It's sort of at the height of the crisis in on the 14th ERCOT started shedding load basically in response to the winter storms. And so, we can see how power generation fluctuates during that time. In this data set, the data are containing 15-minute intervals throughout that time period so a lot of data here and aside from this data set we know from other sources that the forecasted peak capacity during this time for wind was 6.1 gigawatts, and that forecast to peak capacity for natural gas was 48.4 gigawatts. What this forecasted peak capacity means is, this is basically ERCOT's prediction of how much power or energy would be needed from these two different sources when did natural gas during this time frame to meet demand during this unseasonably cold winter for Texas. So, through the remainder of this video, I want to go through the processing of this data set. So, let's get to it.

So, we'll start by importing the data. I've got to contain the Google Drive. So, I've already mounted my Google Drive, and and before I import it, it's always good to click on the CSV and open it up either Google Sheets or Excel of course, as we've done before. And so, here we see the the raw data format of this data set. We've got our dates in the first column, the fuel type, biomass, coal, gas. So, a lot of different sources here, not just wind and natural gas. And here we see two different forms of natural gas. So, there's natural gas and then combined cycle natural gas, settlement type, total energy, or total power I should say, and then times that throughout the day. So, starting basically at midnight and going out throughout the day in which power was generated, and then in the cells here we've got the different amounts of power generated. And we know from our cut that these values are in megawatt hours. So, let's get down to the processing steps. First off, we'll import the data set using using the usual command here. Also, good to look at just a preview of the data we can do gen.head to see what we have here. Look at the first five rows and last five rows, and we see what we saw in the csv, basically. So, it looks like everything was imported correctly. And also, it's not showing all the columns, here. So we're sticking out, you know, some of the middle columns. But we can see that it goes up from, you know, 15 minutes after midnight, all the way up to midnight of each day. Let's take a look at the column headers here. So, gen. columns.

Always take a look at the column names again date, fuel, settlement type, and then all the times throughout the day. Okay, all well and good. So, our first order of business is to filter out some of these rows. So we want to focus on just those two days every 14th and 15th. And additionally, we only want to focus on two power sources, wind and natural gas. So, we can filter based upon those two criteria. So, let's make a new data frame. We'll call it gen.fil for filter. Well, we can call it anything here. And our data frame is currently called gen standing for generation, power generation that is. And then we want to filter this. So, we'll have one filter or set of filters for date, and then we'll have another set of filters for power source. And so, in date we'll have we have two dates to filter. And we'll separate those by and/or statement. So, it could be February 14th or February 15th. And so here we just need to say gen.date equals, and then we need to make sure we get the formatting exactly right here. And here it's useful to have this preview in here it's given by head here. We know that it takes on this format, so month, year. And we want to edit this to be either the 14th of 2021, or the 15th of 2020. Right. I'll do the power source on the next line here. Let me organize this a little bit better. And so, our power sources as we saw in the file could be wind. So, our fuel could either equal wind with a capital W, or it could be just gas. Or we have that third type of gas dash CC for combined cycle. Okay so let's make sure that that worked. Make sure we go back to preserve indenting here, keep it organized. Oh, we have an error here, series object is not... Oops, I forgot an or statement here. There we go.

Okay then, we have it. So now our fuel is only limited to gas, gas-CC, and wind. Okay, that's all well and good. Next step is that we need to rename one of our columns here. Now this is just a particular set for this data set, but it turns out that what ERCOT calls night, zero, zero, zero this is going to be, they allocate this to say, for example, February 14th when really in reality you know midnight belongs to February 15th, right? So, let's just change this to to
11:59 p.m. We're 23:59 when you're using military time. So, that it's attributed to the right day. The reason for this is that the built-in date time tools in Python are going to think that zero zero zero should belong to the 15th, when really that data value that we have, there is attributable to the 14th, for example. So, we'll just make this quick edit, gen.fill. And we just need to rename. And we'll rename columns. And so, our existing name zero colon zero. Zero should instead become 23 59. So, even though we're you know, taking it back one minute, that's really not substantial enough to ruin our data set.

So there we have it. Oh this should be encased in squiggly brackets, of course. There we go. So, moving on here, we've you're done with that, now we can remove some superfluous columns here. So, fuel, sorry not fuel but settlement type in total we don't need. So, let's go ahead and get rid of those. So, we'll continue just writing over gen.fill. Instead, we'll say we'll either a drop command to drop from gen fill some columns that are equal to and in squiggly brackets settlement type and total, so we don't need those for the rest of the analysis, and we'll just check on this always good to check and make sure that that worked and there we go we don't have settlement type we don't have total now this next step is a little bit tricky but um essentially we need to think about so these first steps for you know fairly straightforward we can get rid of some extra stuff here and um you know filter out the things we don't need basically But ultimately, we need to think about how we want this data organized. So. And to do that, a good place to start is to think about what are the real cases here. Well, the case, or each individual case, is the power generation. And this is power generation from each source, on each day, at each time. So, for following our standard format of the data frame having one row for each case. We need to have these generation values all in one column. And we can have a separate column that indicates the field type in a separate column that indicates the date, and a separate column that indicates the time. Turns out we're going to combine date and time so that we don't have, so we have this in one column here. But basically we need to distinguish each value here by those other columns, the fuel, the date, the time. Okay. So in order to get it into that right format, to reshape what we currently have here into something that has those columns, we're going to use this melt function. All right, so how we do this is, again, we'll just write over genfil. We'll replace this and we'll call our dataframes genfil.melt. So, we're going to use our melt function here.

We start with our id_vars, so the vars that are identifying the cases. And so, in square brackets, this will be the date, and the fuel, of var_name will be time, so that'll be unique for each one, and then our values. So, our value name is going to be, sorry the time is going to be a new variable that's going to create, that's going to take all of these current column headers a different time and put them into this new variable called time. And then value name we can call this generation. And we know from beforehand that these values are in megawatt hours. So, we'll keep the units there. So, it's going to take all the values that are under these time columns and put them into this generation card. So, in other words just to summarize, so these id_vars are the current ID variables: date, and fuel. So those are static. We're going to keep those. All the other columns are going to go into a new variable called time. And the values underneath those are going to go into a new variable called generation, defined by value, meaning okay. So, let's take a look and see how this has worked.

And there we have it. We've got our new structure. So instead of a wide fat table, we've got a skinny, narrow table. We've got date, we've got the fuel type, we've got time, and we have generation. So, each of our cases there has a single row. And that's exactly what we want. Now, as I said before, we can combine date and time so that we have something called DateTime that's going to be unique for every case here.So, we need to first import a package datetime. Then we will create a new variable. We'll just call it DateTime. We could call it anything we wanted but DateTime is pretty straightforward. And then we'll say from this pandas package, we need to, we're going to use this function to_datetime, so transform something to a datetime object. And the thing we're going to transform is genfil dot date column, date here, plus, and here I'm going to put in a blank space and another plus and then our time. So, here we're we're not only pulling in the values under date and under time, those current columns here, but we're also specifying the format. We're going to take those and we're going to separate them by a single space. So, let's see what this looks like here. Go ahead and run it. And there we go. We've got a new column, DateTime, that's got the date, space, time, which is handy.

Okay, so if we wanted to, we could remove date and time from this but that's not necessary. One other thing we need to do is we need to convert our generation, which is actually in units of energy here, so megawatt hours, and convert this to power. And ideally, we want this in gigawatts. The reason being is because we have our forecasted peak capacity in gigawatts here, and what we're going to want to do is compare the actual generation to these peak capacity values. So we need to have them in the same units. So let's go ahead and convert megawatt hours to gigawatts. And this is going to be here, we're basically using a state of the known formulas or unit conversions that we're going to enable here. Okay, so we'll start by taking these existing generation values, and we can see the first thing we need to do is we need to get rid of these columns, the commas that are in here. So if we took these directly, Python's not going to recognize them as numbers. We need to remove the commas in order for it to recognize it as a numeric variable. In order to do that, we will replace these strings. We're going to replace the commas with nothing. So the way this replace function works is we take our variable, we say dot str. This is referencing a string package and we're going to use this replace function. We're going to take what we want to replace, the commas, and we're going to give it a second argument of what we want to replace it with, in this case nothing. So that'll just simply remove the commas. We can go ahead and make sure that worked. And there you have it.

Now we can convert this to be numeric. I didn't do the, didn't look at D types before, data types before, but if you did that you'd see that this is actually a character column, character variable. So we'll say to_numeric, so we can convert that to a number now. Because ultimately we're going to want to do some quantitative operations on this so we need it as a number. All right, so now that it's a number, now we can actually do the unit conversion. So we'll start here, instead of generation and megawatt hours, and we can say the units. You're not going to be gigawatts. And what we'll do is we'll take our existing variable generation to megawatt hours and we will divide by 0.25, because there's, 15 minutes is a quarter of an hour. So we're dividing by these hours to take it out of the numerator, here. And also additionally, we're going to divide by a thousand because there's a thousand megawatts in a gigawatt.

So we can make sure that that worked. And there we go. So we've got our generation in gigawatts here and then we can now, we can clean up a little bit. Also, we can drop some of these unneeded columns now. So we'll drop columns. We can get rid of date, we can get rid of time, and we can also get rid of generation in megawatt hours. This is going to give me some problems because I've already read that, but anyway we can, here we can put these in a new chunk of code I'll call columns. And there we go. So anyway, now we have our cleaned up data set. The reason I got this error before is because I had already run these. And so now it can't find, I can't do this replacement because I already replaced it. So anyway we've got our cleanup data set here with just the variables we want; the field type, the datetime and generation. One other thing we need to do is we've got gas and gas combined cycle, let's combine these into just natural gas, because these are two different forms of natural gas. We're going to lump them together because ultimately that forecast to feed capacity is considering both forms of natural gas, and the generation data is segregating those two sources. So, in order to do this we need to first pivot our data frame. So let's make a new object called genpiv, and what we're going to do now is we're going to, we have something  that's tall and skinny here, and we're going to make it a bit fatter. We're going to have separate columns for a fuel type here. So we're going to essentially do the reverse of that melt so that we can then do some operations on the two different columns. So we'll say we'll define an index here. Our index is DateTime, sorry, index is DateTime because that's a unique identifier for these rows basically, well, combined with the fuel type, I should say. and then we're going to make some new columns that are based on fuel. And then the values for this new data frame are going to come from our generation in gigawatts. Okay, so let's take a look at what that looks like.

So there we go. Now we have a data frame that has indices. So this is not a column right now. These are indices of DateTime. And then we have gas, gas combine cycle, and wind. And the values that are all in generation in units with gigawatts. Okay. So now we can combine gas, and gas CC, simply by adding those two columns together. So I'll make a new column here. Let's call it Natural Gas, and we'll take our gas column and our gas CC column and we're just simply going to add those two values together. So we're going to aggregate them. Now after we do that, the gas and gas CC are going to be extra, so we can get rid of those. So get rid of gas and gas CC since those are no longer needed, and let's see what we have now. So there we go. We've got now just two columns; wind and natural gas. And you know, we see that gas, gas CC, 7.5, 26.2, and add those together 33.7. That's about right. So there we go. Now we want to get it back into the formula we had before where fuel was a column, not two separate columns. So, let's melt again. Before we melt, we need to re-establish DateTime as a column.

So what we're going to say is, we're going to say from genpiv, take to index and make it a view column called DateTime. Let's see what that looks like here. So we still have DateTime as an index but we've also separately made it as a column here. And then from genpiv, we can say melt again, our id_vars in this case are going to be, is going to be just DateTime and our values are going to be in generation gigawatts. Go ahead and run that. And there we go. That's what we want. So we have a column of DateTime that has a case in the times, fuel type, and then our generation values.

Lastly, let's go ahead and visualize this. This is an important step for hypothesis testing, and seeing what the data looked like this gives us a good idea of how to set up our subsequent hypotheses. So, let's make a line plot here, where we will have a line, one line for each energy source and plotted over time here. So, GG plot, we'll take our data frame genpiv and we'll have a line where X is going to be our DateTime and Y is going to be our Generation. We need to get this name exactly right so it matches the column there, and also we want to specify the color in this case as fuel. So let's see what this looks like. There we go. So we've got our different fuel sources; natural gas, and red and blue plotted over DateTime here. We can see natural gas is a lot higher than wind, so there's a lot more energy coming from natural gas than wind just because there's a larger natural gas infrastructure in Texas. Although Texas does have comparatively large wind infrastructure compared to other states.

We will, let's clean this up a little bit. I don't really like how you can't see the date and times here very well. So, let's adjust our theme and make our axis text on the x-axis a little more readable. We'll say element text, and change the angle to be 75 degrees. You can see the effect of that. There we go. So now we can read the dates here a bit better. But it's cutting off the time, so let's before we do the theme, I like putting the theme at the end. Before we do that, let's adjust the the date labels. And we can control the formatting of this as such, so we can say we want month and day, so we're only going to extract the month and day because they're all the same year, it's all 2021. So it's also all the same month. We could get rid of the months, but let's keep the month and day. And we also want to include our, and separated by colon, the minute. We'll put a plus here. So let's see the effect of that. There we go. So, now we've got some time information there. So, we have have the hours and also the minutes displayed.

So that's great. Let's also put in the capacities. So, we have the actual production plotted here. Let's see how he stacked up against our capacities. And so, we can insert those capacities, since those are essentially constant values over the whole time frame here. We'll put those in as horizontal lines, and if we want to do an H line, we need a y-intercept argument. So basically, what's the y-value for that horizontal line? And so, for natural gas this is going to be 48.4, and our color here can be red and, in our line, type will be dashed. Let's make it a dashed line. And then we'll also do this for wind at 6.1 gigawatts, and let's make this blue. Let's go ahead and see that, so there we go now. Yeah, I don't really like how these are different shades of red and different shades of blue. So, I'm pretty picky here. If we wanted to control that a little bit better, we can say we can define our color scale for the curves, so for the lines, and we'll set some values let's switch this up. Let's say, so we'll have natural gas be the blue and wind be the red. It doesn't really matter. There you go. So now the colors match perfectly in terms of shade. Just how I like it. So, but the main takeaway here is that, well, we can see natural gas is consistently below its capacity over this two-day time frame. Wind is below its capacity sometimes, and sometimes it's above it. So, that's interesting. So, let's retain that thought as we go forward in the subsequent hypothesis testing. Okay?

All right, in this series of videos we're going to shift back into hypothesis testing. And in particular, we're going to focus on the one-line tests that you can do to do all of the hypothesis testing that we learned in the previous lesson, but in just a single line. And while a lot of times we'll still want to do the randomization procedures, these one-line tests can be a good way to to check your answer. Because although they may be slightly different, a lot of times they should actually follow the same pattern as your randomization distribution. So, let's go ahead and jump right in. So, we're here in the code for hypothesis testing. And in particular, I'm going to focus on the hypothesis test for a single mean in this video.

And so, when we write our hypotheses, we say that mu equals in this case 6.1, the capacity. And then our alternative is that mu is less than that. So in essence, we're going to test whether or not wind actually met the capacity in, during the Texas 2021 cold snap. And so, in the previous lesson we went through these steps: we shifted the data, we initialized the variable, and we implemented this randomized random choice function where we set replace equal to true and focused on the shifted data. And then, eventually we calculated the p-value as the proportion of data that was less than our original mean. And we got a p-value of zero, which leads us to reject the null hypothesis in favor of the alternative, that the capacity of wind, that the average wind was less than the capacity. And so, this one-liner test, we're going to use a library, the stats library in Python. And in particular, this library is the stat SCI Pi stats, which is the same library that we used when we were writing our normal or working with the normal distributions in the previous videos.

And so, in order to do the single mean hypothesis test, I'm going to put the results into a variable called results. And we say stats dot ttest underscore one - the number one - samp. So effectively, a one sample ttest. Then we give it our data, which is wind - which is the observed wind generation. We give it our population mean, which is what we expect. So this is our capacity in this case, or whatever value you have set in the null hypothesis. And then we give it the alternative. And in this case, we specify the alternative as less. Meaning that we are doing a test in which we're looking at less than, we're using a left-tailed test. And then to actually see the p-value we can just say results dot p-value at the end. And so, we can run this. And we can see that the p-value is still less than our significance level. It's very, very close to zero. But as what often happens in these one-liner tests, that is more specific. So, instead of just staying zero, we've got you know quite a few digits, but still very, very, very close to zero given the e to the negative seven. And this is a very common result when we're using these one-liners and comparing them to randomization procedures, because what this one-liner test is actually doing is conducting that same randomization procedure that we went over, but with a lot more data. And so, essentially, it does something very similar to this. And instead of being a thousand, is maybe a million, 10 million. It essentially generates a lot more iterations, which allow it to become a lot more specific in the results.

Hi. In this video, we're going to go over how to do a hypothesis test for a single proportion using randomization distributions in Python. And we're going to continue on with this Texas Cold Snap example from the Power Crisis in 2021. And we're still going to be focusing on just when because this is a single proportion, so we have to look at a single sample here. Now, the proportion that we're going to look at, is the proportion of time that wind generation fell below the forecasted peak capacity. So, recall that this forecast to peak capacity is really, reflects the demand on the system. And so, we're going to look at the proportion of time that essentially supply was not meeting demand during this crisis. And again, just for wind here. Okay, so let's get to it.

So, our hypotheses for this single proportion. The null is that the forecast speed capacity was met half the time, or at least half the time. And the alternative is that this proportion, the proportion of time below the peak capacity, is greater than 0.5. That is, more than half the time wind was not meeting that capacity level. Okay, so that's kind of, that's our failure criteria here in the alternative. So, we're testing to see if either the null was met, or that null was not met. Again, we're not, we never accept this alternative hypothesis. So, let's start off. The first thing to do always in step one, is to calculate our sample statistic. So, what is our actual proportion from the data? So, let's calculate wind p-hat. So, our p-hat from the wind sample is the length of the elements in our wind vector. Recall that with the hypothesis test for the single mean, we extracted the generation from wind into a single vector called wind. So, this wind object is a series of the generation in gigawatts. And we're going to divide by little n, little n being the sample size of when p-hat. We calculated those back up in the hypothesis test for the single mean. So, n is the is the length, effectively, of wind here. And wind, again this generation of just the wind source.

So, let's see what our sample statistic is here. So, when we'll print this out 0.64. So, 64 of the time the actual generation fell below the forecasted. Keep forecasted peak capacity of 6.1 gigabytes. So, then the question is, well is that a lot? Is that statistically significantly different than half the time? So, that's what the hypothesis test here is going to answer. So, for a randomization distribution, this is going to be a bit simpler than what we did with the with the mean. We're not going to use random choice. Instead, our key function here is going to be random.binomial. We're going to use this binomial distribution to simulate conditions under the null hypothesis. So, if the null hypothesis were true, then half the time the generation would be above or equal to 6.1, and half the time be below. That'd be the, that's essentially what's going on in the null. And maybe those portions are a little bit different, but let's see here. So, with the binomial distribution, how we can simulate from that is that, well, we can essentially let the binomial distribution, is essentially doing a series of coin flips. So, we can flip a coin at every time during our two-day period here of the Texas cold snap. And you know, if it's heads, then will say that wind falls below the capacity. If it's tails, it falls above. So, that's that's essentially what's going on with this binomial distribution.

So, let's put it into action in Python here. We'll establish our little n is the length of the wind, and we already have that established above. But we'll reproduce that here, and again we can always specify a capital N and we'll keep it the same as what we had for the hypothesis test for the single means. And what our random dot binomial distribution is going to do is, it's going to return the counts of successes here. So, it's going to turn, return essentially, our numerator in this proportion. So, the number of times that wind falls below 6.1. And, what we need to give it is our sample size. So, we want to do n different flips, or trials, and our proportion here is going to be 0.1. I'm sorry 0.5. That's going to be our null value. We take this value straight from the null hypothesis up here. And our size is going to be n. So, we're going to do little n trials, capital N number of times.

So, let's go ahead and run that and we'll see what's contained in counts here. So, we see that we have a collection of a thousand counts, a thousand numbers of successes. And so, if we want to get our p-hats for means, so our simulated p-hat values will be just the counts divided by in this case, a little n. It's not letting me. Every time I hit return, it to capitalize it, yeah there we go. So there it is in proportion format. So these are p-hats, what we're looking at right now, the series of numbers. This is our randomization distribution. Let's visualize this.

So, we'll do p-hat data frame is equal to PD dot data frames. We're just going to transform p-hat into a data frame here, and give it some column names. We'll say p-hat, and then use our good friend GG Plot to visualize this. I'm going to use a different visualization than histograms. We did a histogram last time for the single mean. Let's use a, let's do a dot plot now. We'll see what this looks like. The reason, well, I'll explain, the reason why I'm using a dot plot here when we take a look at it. And as we had before, let's also put our sample statistic here, as a vertical dashed line. So, we'll have our sample statistic when p-hat is going to be a vertical-dashed line. And this x-intercept value, the color again will have the red, and line type can be dashed.

So, there we go. There's our randomization distribution. This is the dot plot. So, instead of having bars, we have a collection of dots stacked on one top of one another. And the reason why I like doing this for the randomization distributions, is that each dot represents one randomization. So, a single randomized sample. So, we can see basically, the number of randomized trials, in this case, that fall within each bin, within each category of p-hat here, which is nice. It gives us a sense of the scale, and how many trials have been running. Again, this vertical red dashed line, this is our true sample statistic. This is what the data say. And similar to what we saw with the hypothesis test for the single mean, this falls outside of our randomization distribution. In this case, it falls above it. But let's go back and reference our alternative hypothesis here, alter our closest. Our alternative hypothesis is that p is greater than 0.5, and here we see no randomized trial that is at least as great as this red dash line. So, this red dash line is pretty extreme here. So just by inspection, this week we know beforehand that we can reject the null hypothesis. We can formalize this into a into a p-value. Let me just copy and paste our code for printing the p-value here.

Our p-value here is not going to be based on random mean. Instead, we have p-hat is our series that contains a randomization distribution. And it's not wind mean this time, but rather wind p-hat. And we've got to flip this around. It's greater than this time. But still our p-value is zero. So, there's zero probability of getting a randomized trial at least as great as our sample statistic if the null hypothesis is true. Again, this randomization distribution is reflecting the condition under which the null hypothesis is true, and showing what sort of variability we would have in that. In our data these would be our p-hat if that normally boss is true. So, what should we write for a concluding statement? Again we can say since the p-value of zero is less than 0.05, we can reject the null hypothesis, that wind met or exceeded the forecast to peak capacity. I'm just going to write the capacity half the time. But the key term here, is rejecting the null hypothesis. So, we reject the null hypothesis, and we provide the reason, the reason being that our p-value is less than our significance level, our default significance level of 0.05. Okay, so that is our test for a single proportion.

Hi. For the remainder of lessons and videos under hypothesis testing, we're going to look at two-sample comparison. So comparing the wind sample that we have here, is red bumpy curve versus the natural gas sample, just blue bumpy curve. Now before we get into actually, any actual hypothesis tests here, I want to talk a little bit about an additional variable that we're going to calculate out of this. So, let's get to it. So, the reason for calculating an additional variable here, is to have something by which we can compare wind versus natural gas on an even basis. We can see clearly here, that natural gas is producing a lot more energy than wind. And it also has a much larger forecasted peak capacity than windows because of these natural differences. And these natural differences arise because natural gas is a much larger system in the state of Texas than wind is. In other words, Texas relies way more on natural gas energy than wind energy. But because of that, we're kind of comparing apples to oranges here. And we want to level this playing field and come up with a metric by which we can compare fairly, wind versus natural gas and directly answer the the question. Well, which one has underperformed more? If we just looked at generation alone, well, we'd say wind is underperforming simply because it's not making as much energy as natural gas is. But that's not fair, because there aren't as many wind turbines where there isn't as much wind energy to begin with in the state of Texas than there is natural gas. In other words, a lot Texas is way more dependent on natural gas here. So, in order to facilitate a fair comparison between wind and natural gas, we're going to look at the percent deficit.

And this percent deficit is the generation. So, that bumpy curve minus the forecasted peak capacity, divided by the forecasted peak capacity. So, in other words, this is basically looking at how much each energy source has either gone above or below  that dashed line on a proportional basis to that capacity level. So, how are we going to calculate this? So let's just make a couple of new variables here. Recall that the data object that we're working with is genpav. That's our data frame. We're going to use this dotloc function to define some new values. 

So, we're going to say anywhere where the fuel is equal to wind, we're going to make a new variable called capacity, and set it equal to 6.1. At the same time, we're going to do the same thing, except when the fuel is natural gas under this new capacity variable, we're going to set it to 48.4. These are gigawatts. These are the forecasted peak capacities in gigawatts that we talked about in the first video. Then with those in place, we can calculate a new variable called deficit that will simply be our generation in gigawatts here minus this capacitive. So, that's our deficit that's the numerator the percent deficit and then if we want to just get this into a percent deficit.

We can do our deficit with a capital D, divided by capacity, times 100 percent. So, let's see what this looks like. So there we go. Here's our percent deficit. Many instances it's negative. In other words, that means it's below the forecast peak capacity. I think some of these rows that are omitted here for when to be positive, let's visualize it. So, to start off with our visualization, let's go back up to our previous visualization, borrow from that. And so, we'll still do a line plot. We'll still have the same colors here, except we don't want to plot generation, we want to plot our percent deficit. And we'll color by fuel type. We'll sell blue and red, except now since we're looking at percent deficit, these capacities they're not as relevant anymore. So, let's delete those. Instead, we want to compare. Two is zero percent, right? Is there a zero percent deficit? Let's look at this as a black line dashed, both. I'm going to do it as black because regardless of whether it's wind or natural gas, we want to compare it to zero percent deficit.

There we go. So, the red, wind, blue, natural gas, black, dashed line at zero percent deficit. We see that natural gas is always in the negative, and wind is sometimes perhaps most of the time, well, 64 of the time, as we learned from the test of single proportions, below zero. Okay, so that's what we're going to use. We're going to use the percent deficit in the remaining two sample tests. So, comparison of means both paired and unpaired, and a comparison proportions.

But in this video, we're going to go over how to do a difference of means test, or a comparison of means for two samples. So, let's get to it. So, we're going to continue on using this percent deficit that we calculated in the last video. And our hypothesis test for these two samples in the comparison means, or difference in means test, is going to be the null, is going to be that the mean for gas, minus the mean for wind, is equal to zero. Or, in other words, that these two means are equal. The alternative here is that the mean for natural gas is less than that for wind. Or, in other words, the difference between those two is less than zero. The reason I set it up this way is because just by looking at this visualization that we developed in the last video because natural gas is consistently below zero percent deficit. I have the suspicion that its mean will be less than that for wind. So, I set it up for the hypothesis based on that. But, let's go ahead and calculate our sample statistic here. And we can confirm whether the mean of the data conformed to that. So first, let's pull out our wind sample and recall that our data frame is genpiv, that we're working with here. We've calculated percent deficit in gen cap, so we want to pull out that values from that variable here. So, genpiv.loc. And anywhere where genpiv as a fuel value that's equal to wind, we want to pull out the associated percent deficit values. Okay, and that'll be our wind sample. Same thing for natural gas, we'll just replace wind with yes.

So, we'll just pull out those two samples. And then our sample difference is going to be our sampgas.mean. So, the mean of sampgas for in minus sampgas, sorry, sampwind.mean. So, here's our sample statistic because the difference between the mean of the gas percent deficit and the mean of the win percent deficit. Note that this order is important here. It's got to conform to what we have in the hypothesis test here. It's really critical. Let's print some values here so let's print our mean gas value is this let's also print our mean win value this is just for inspection. And then finally, the difference between those. We see that our average percent deficit for our gas sample is negative 28 percent. The average percent deficit for a wind sample is negative 15.5 percent. So indeed, from the samples the gas is less than the wind. And so, this would justify having the hypothesis set up this way. And then the difference is indeed negative, or less than zero. So again, justifying this setup here. But still, the question remains is this negative twelve and a half percent, is this statistically significantly below zero? In other words, is this completely out of the realm of possibilities even if the null hypothesis were true? So that's what we want to test with the randomization distribution and the resulting hypothesis test here.

So, moving on to a randomization distribution. We already have imported numpy previously, but let's do it again anyway, just for, as a reminder that numpy is needed for this. One thing that we're going to do here is we're going to copy the genpiv data frame over to something called sim. So, we're going to make a new data frame sim, that's just going to be an identical copy to genpiv. The reason being is that we're going to be scrambling up some of the values there. I'm going to want to mess up our original data frame. Again, just a reminder, capital N will be a thousand, as will consistently use through these hypothesis tests. Although you could do more when you're doing this on your own, or sample size is going to be the number of rows in genpiv, or in sim, here. So, this is going to be the sample size of wind, plus the sample size of gas. This shape function gives the first, the number of rows, then the number of columns. By putting the index here we're getting, we're extracting the number of rows from that. And then, as we did in the test for single means, we want to store our resulting statistics from the randomization distribution into some variable. So, we'll have xbardiff, this is just going to be an empty variable, capital N, 1000 blank spaces in it. And as we go through this for loop we will fill in those blanks.

So from our sim data set, data frame, what we're going to do is we're going to scramble up this percent deficit color. So, we're going to make a new percent deficit column, or overwrite the percent deficit column, and we're going to use random choice to do the scrambling. So, we saw random choice first in the hypothesis test for a single mean, where we were drawing with replacement from that wind shift variable here. We're going to draw from our original percent deficit column, and we want to use size equals N. So, the same sample size that we have originally, and here we don't want to do this with replacement. So, we're going to do replace equals false. The reason being is that if we go back up to our data frame, really we just want to scramble up these values and reorder them in the same column, and we don't want to repeat any values we really just want to reallocate these values across the two groups, wind and natural gas, and do that in a random sense. So, some will stay within, when some will stay within natural gas, but we're going to switch out the order and potentially regroup some of them because under our null condition if there isn't any difference between these two groups then it wouldn't really matter what group the observations follow. So, that's the philosophy under the null hypothesis here.

So, once we do that scrambling, well then, we just need to execute our calculation of the statistic again. So, I'm going to copy and paste everything from from step one into here. And really, we just need to replace all these samps with say, sim, for example. So, I'll have our simulated wind sample, it's coming from sim replacing genpiv with sim after we've scrambled up percent deficit here. And then the difference will be the difference in these means. So, the same steps that we had before in calculating the sample statistic from the data, we'll use after we've done the randomization and scrambled up this percent deficit column here. Okay — now. Oh, sorry, I should recall this xbar diff. And we want to, of course, store this difference. Okay. So, we can go ahead and run our randomization distribution and then visualize it. So, I'm going to go back up I'm going to just copy and paste our visualization from our proportion, because of a single proportion bring it back down here, it's the same structure except we have a different, we have different names of things. But still our randomization distribution we want to turn into a data frame, and our x-intercept should be our sample statistic with samp diff here.

So, we can go ahead and visualize that. So, under our null condition, if the null hypothesis are true, if there really wasn't any difference between the two groups, we would see a difference of means a percent deficit that ranges from about negative 10 to about positive 10. So, pretty big swing, but nevertheless, our actual sample statistics. So, from our data, what we observed in reality is quite a bit less than that range, falls outside of that range. So what do you think our p-value is? You're thinking zero, you've got it. But we can go ahead and calculate it here. Again, just replace our p-hat with this xbardiff, the series, not the data frame. Our sampdiff replaces wind p-hat, and the direction of this inequality should be less than because that's our alternative hypothesis. And we have zero here. So, our concluding statement, again since the p-value is very low 0.0, and less than the significance level of 0.05, that's the default value. We can reject the null hypothesis that there is no difference between the means of the two groups. The two groups being percent deficit of gas, versus percent deficit of wind. Again, the critical language here is rejecting the null hypothesis because our p-value is less than 0.05. But it's always good to contextualize that result in the actual problem. So what do you learn from particular null hypothesis? Well, we know then that there actually is a difference between these two groups. So, there is evidence that natural gas underperformed more than wind. Okay. So, that's our test for a difference of needs.

Hi. In this video we're going to do a hypothesis test for two samples comparing means. But this time we're going to look at the mean of differences, that is, a paired comparison. This is in contrast to the difference of means that we did in the last video. So, let's get to it. So, the reason why we're doing a paired comparison here, or we're able to do a paired comparison, is because at each individual time here we have both a measurement of wind percent deficit and natural gas percent deficit. And so, we can compare what's going on at each individual time as we go along. We don't have to just compare entirety of one curve versus the entirety of another curve. There may be valuable information in constraining it, constraining that comparison at the individual time steps that we have here. So, how do we do this?

First thing is we have to reorganize our data a little bit. So, let's just create a new data object where it's called df2 from our original genpiv. We need to pivot this data frame. So, let's do this. And you'll, it'll start to make sense after we've done it. So, our index is going to be date time. Our columns will be created from the fuel column that we have in genpiv. And our values are going to come from percent deficit. So, let's see what this looks like. And it'll be clear what's being done here. So, what was the date time column in genpiv has now become an index. We had a fuel column that contained either natural gas, or wind, or two groups. And those are now two separate columns, here. And the values underneath are the percent deficit now the reason why we organize it as such is because now we have these pairs at each time step spread over two columns here, so this allows us to very easily then calculate the difference of each of these pairs.

So, I'll make a new variable difference that's simply our natural gas minus our wind and why am I doing natural gas minus wind and not wind minus natural gas? Well, this is because of our hypothesis. So, our hypothesis is that the mean of the difference between gas and wind and this is following the same order that we had in our first test of two means comparison of means looking at the difference in means. So, our order there was gas minus wind. So, we're preserving that same order. Again, this is based upon the supposition that we see in the from the data visualization that gas is under performing relative to wind. But the goal of this is to test that supposition test that hypothesis. So, there we have, there's our difference. We can see that it's been calculated. So, now we have a difference column that's simply the difference between those two things. And then finally, to get our sample statistic, which we can do down here. Our meandiff, we'll call it, is from this difference column and calculating the mean.

So, we can see what that value is. Lo and behold, minus twelve and a half percent. Note that this is identical to the greatest degree of precision that we have here, to what we had above with the difference of means minus 12.5628, blah, blah, blah, blah. They're mathematically the same thing. So we have the exact same sample statistic here just calculated slightly differently. However, the randomization test is going to be done considerably different. So again, just as a reminder, we do need numpy here. Include that to be complete. We're going to also copy df2 to something called sim here. So that we don't mess up our original data frame. It's just good practice. Capital N again will be a thousand, and little n again will be the number of rows that we have in our data frame of interest. We do need an empty vector here to store our statistics in after we do the randomization. And again, we'll fill this up as we march through our for loop

So, what goes into our for loop? Well first we need to do our random randomizing, here. What we'll, how we'll do it is we'll calculate this multiplier. The multiplier here is just going to be, and this should be in square brackets, Our multiplier is just going to be either one, or minus one, that is we either flip the sign of the difference, or we don't, because flipping the sign of the difference is equivalent to swapping the values across these two groups, or AKA across these two columns here. So, this is quite simple in terms of code.

We just use random choice again and we can just simply randomly select from the vector 1 or -1. So, we're just going to randomly choose one, or minus one, we're going to do it n times. And we will do that with replacement, of course, so we're going to end up with every time we go through this whole thing because a vector here or a column it's the same length as percent deficit. I'm sorry same length as difference. This can be filled with some random assortments of ones and minus ones. Okay. And then, our new simulated randomized difference column is going to be simply this sim multiplier times our same difference. So either we to look at the sign, or we don't. And then finally we store our sample statistic in this xbardiff vector, this empty vector here, and this will just be difference. And we'll find the mean of it. Okay.

So, there we go. And then we can of course visualize the randomization distribution xbardiff or we're still using that variable name. We're cycling that variable name, but we're not looking at the difference. It means we're looking at the mean of differences. So, we'll change that name up. And our sample statistic is actually mean diff. We'll call that something else here. Let's see what this looks like.

There we go. So immediately you might say, okay, well the mean diff, which is equal to this minus 12.5 percent still outside the realm of our randomization distribution, right? Again this randomization distribution is the realm of this is the the suite of possible events that we could see if the null hypothesis were true. And so, again, you can immediately tell from this that our p-value is still going to be zero. But there is one difference I want to tease out of this. So, in comparison to what we did with the difference of means, so take a look at this randomization distribution. Most of it's concentrated between about you know, minus seven. Deposit minus seven and a half to positive seven and a half. Whereas, we have a bit of a wider sweep of possible events when we did the difference of means.

So, in other words, this randomization distribution from the mean of differences is a little bit more concentrated around the null hypothesis value. In this case a zero percent deficit. Now the reason for that is because we're constraining the randomization to take place within each row before with a difference of means essentially these values could swap rows. They could end up anywhere. But here they can only stay within the rows. So, because we're maintaining that pairing because we're saying that there's added information somehow in keeping these values at the same times. But there's something valuable about that time information. We're getting a narrower range of possibilities in this randomization distribution. And thus a more definitive answer in our hypothesis test, even though the p-value is still zero. I encourage you to go take a look at a subsequent video effective sample size, in which we'll compare these. If we have a more limited sample size you can see some interesting differences there. Okay. Just around finish the discussion here. Let's calculate our p-value and arrive at a conclusion, we'll just borrow our p-value calculation code up here. It's not xbardiff anymore. Oh it is still xbardiff, but here we have new diff, and again, it's still less than, and again, we get a p-value of zero, and of course, our concluding statement would be the p-value is low. It's zero. It's less than 0.05. We will reject the null hypothesis. Same conclusion that we had under the difference of means. Okay. Thank you.

In this video we're going to go over the one line test that can be used for a two sample mean hypothesis, a comparison of the means. And so, when we wrote these null hypotheses, our null hypothesis was that the mean of gas, minus the mean of wind was equal to zero. Thus, there was no difference between the two stated. And our alternative is that gas minus wind is less than zero, suggesting that there is a difference between the two. And here we calculated mean for the percent deficit as, generation minus capacity, divided by capacity. And so, we did all of this prep work and then we got into the actual randomization procedure where we calculated our sample difference, did our reallocation procedure where we sampled from percent deficit with replace equals to false, and recalculated these differences, and ultimately got a p-value of zero. Which suggested that we reject the null hypothesis in terms of this one-liner test. I'm going to call the results, results. And again, using that stat dot scipy library we can say stats dot t-test. And in this case we say IND for independent samples. And then we give it our first sample, which is sample gas. And a key point here is to make sure that you're matching the order that you write your alternative hypothesis in. And so, we did gas minus wind. So our first sample is gas. And I want to also point out that we are using the actual data here. So our sample is located from the samp.loc which we calculated in step one. We're not using any of the reallocation samples which we calculate in step three. So our first sample is samplegas, our second sample, samplewind. Here we need to say equal there equals false. And this just tells it that the variance is not equal between the two data frames to variables. So, it'll conduct a slightly different version of the t-test than if they were equal variance. And then we specify the alternative equals less, since we are using a left-tailed test. And then after that, we can say results dot p-value to print the p-value. And so, here we can see that we are very, very close to zero, following that same pattern that we have seen in the earlier two tests. It still has the same conclusion as our randomization procedure, but with more specificity. So again we reject the null hypothesis.

Hi. In this video, I want to show you what the effective sample size would have on our hypothesis tests and the results.

So, for starters, just to summarize, we saw in all five hypotheses tests that our p-value was zero. And this was using all available data, which one should use normally. You should always use all the data that you have available to yourself. But we had pretty large sample sizes there. So we're using all the data, good amount of data. Data collected every 15 minutes during that that two-day duration. So there's a wealth of data and because of that we got really low P values. We got, we had lots of evidence to definitively say that, well first with the single sample test that, both wind and gas didn't meet capacity. But then when we did the two sample tests and compared them, natural gas performed even worse than wind, so both failed. But natural gas failed more effectively, is what's being said in the results here. But we didn't see much variation in our p-values. Every time the p-value was zero, and we had these very definitive results, we always saw that the sample statistic fell well outside of the randomization distribution.

And so, I want to show you what the effect sample size has here; one to give some more variation in the results and what statements you should make if your P values are not zero, but then also to really give you motivation and show you the importance of collecting your data. Okay, so in order to do this we're gonna imagine some situations here. So imagine the situation that we have far less data than we actually have. Say for example, suppose during this two-day event across February 14th and 15th, in February 2021 we, maybe there was some sensor failure in the grid and we weren't able to record data every 15 minutes. Maybe we only had a handful of data in actuality. So, in order to kind of simulate that hypothetical or imagined situation, let's insert some code here and alter reality. So we're going to down sample the data set. No, this is something that you shouldn't normally do in practice. Again, in practice, you want to use all the data that's available to you. But I'm doing this here just to show you how these results would change if we weren't fortunate enough to have this much data.

So, in order to do this down sampling we're going to go back to genpiv and we're going to pivot this. I'm just going to, actually we've done this before so I'm just going to copy and paste where's our pivot. So we're going to pivot so that we have this situation. Now that we have, you know, values paired here, we can easily down sample. Essentially we can just select some of these times in which say, for example, the sensor would actually be working. And we can do this quite easily using a built-in function in pandas called sample. So this sample function is from pandas and what we do by specifying n say equals 20, is we're just going to randomly sample 20 values, or 20 rows from this data set. Okay, so instead of having, you know, 192 samples, we're going to go down to 20. So, far less data.

Then we're going to, we're going to reconstitute this in its original form. So we're going to unpivot AKA melt. And we did that here, so we can copy and paste this here. So we'll date time is now in index will bring us back to a column, and then we will melt this into this manifold here. So, let's actually go to runtime and we will run everything before up to this code. Now we can run this again and here we go here's our new down sample data set. We've got date, time, fuel generation. Note that we've got 40 rows here, so starting at 0 to 39, because our sample size is 20 and we've got two groups. Okay, so that's our new genpiv down sample. Now let's see what change this has on all our results. So we have a slightly different wind mean, although it's basically the same, 5.15. We'll redo a randomization distribution.

Oh, this randomization distribution has gotten quite a bit larger now, still centered on the null value of 6.1 is this for the single mean here, and it overlaps with our sample statistic, so our p-value now is not zero. 0.042 it is still less than 0.05 though, so we're still less than our significance level, our default significance level. But we should change this. Our P value is not zero anymore, it's 0.042. We would edit this statement. I'll just delete this right now. Therefore, we can still reject that. We still reject the null hypothesis, but 0.042 is less than 0.05, but just a little bit less. It's not dramatically less. Okay, let's continue on. What about with our test for single proportion?

Our sample statistic has gone up a bit, 0.7. Can just re-do all this code here again, we've got overlap here, and what's our p-value, 0.061. So this is not, this does not meet the criteria for rejecting normal hypothesis, that is using the default value of 0.05 so we would change this. Since the p-value of 0.061 is more than 0.05, we can no longer reject the null. We would say we fail to reject the null hypothesis. That we met or exceeded the capacity half the time. Nope, really, really important here we do not accept the alternative. We are not saying that the alternative hypothesis right here that this proportion of failure rate is greater than half time. We're not saying that that's true. We're saying that the null hypothesis is wrong. In other words the evidence, I'm sorry we're saying that we're not accepting the null hypothesis, we're just failing to reject it. Okay, and we're not accepting the alternative, right. Furthermore, continuing on, when we get into two samples we'll just recalculate our percent deficit values here note that this is you know much less detailed of a curve we have you can see where the data points are here where the inflection of the curve changes. And we will go ahead and run our test here. Our difference of means is no longer 12 and a half percent, 13.8. So that changes a bit because we've altered our data set.

Again, we've got overlap here, and again our p-value is greater than 0.05 so we would again alter this statement to reflect that this is no longer 0.0. But 0.77 is more than the significance level of 0.05, we fail to reject the null hypothesis. But there's a difference. Okay. So we fail to reject. We do not accept the null. We fail to reject it. Redoing now, going down to the mean of differences, running this again. Note that this python notebook makes it very handy to just rerun the code. We just go through and press play. We still have everything called gen PIV. We just alter gen PIV so it makes it very easy to examine this hypothetical situation. This is interesting. So when we do the difference of means, our p-value is 0.038. So, it's still low, still below 0.05. We still reject the null hypothesis.

But this is different than the difference of means that we did above. So take a look at this randomization distribution from about negative 20 to positive 20. Note that we still have the same sample statistic here, and minus 13.898, blah, blah, blah. When we go up to the difference of means we still have that same sample statistic. But the randomization distribution is fatter. It goes up to maybe minus 22.5, and positive 22.5. The value was 0.077. It's about twice as large as what we got with the mean of differences. So as I said before, when doing the paired comparison with the mean of differences, by constraining the randomization to be at those times within each time interval we're adding information to the test we're saying that the time makes a difference the time is valuable, and in doing that we get a lower p-value. So really interesting here that doing the difference of means we'd fail to reject the null. However, if we do the mean of differences we do reject the null we get a very different outcome. Just to round off the discussion on effective sample size, we'll redo our comparison of proportions. We run our code here, a portion of 1.7 agrees with what we had, with the single proportion difference 0.3, much wider outcome with the randomization distribution p-value quite low, 0.007.

So, we would still reject the null. In this case we still have the same outcome. So effective sample size, if we have a much smaller sample size, we do see these non-zero p-values. And in particular, some of those tests fail to reject the null. The p-value is larger than our default setting against level 0.05. And also in the two sample comparison with means, our difference in proportions fails to reject. Whereas our, I'm sorry, our difference of means fails to reject. Whereas, our mean of differences still rejects. So really interesting there. Okay, so sample size really important, larger sample size always better. More data is always better. Really important to repair that in mind. Okay. Thank you.