Variable Renewable Energy Generation Forecasting and Integration with Dispatching

Speeches Shim

This webinar explores how Indonesia, Kazakhstan, and Pakistan deploy variable renewable energy generation forecasting tools to ensure effective grid management and electricity system reliability.

A cityscape at night with high power energy transmission towers in the foreground. A network of lines and dots of light are superimposed over the image to evoke the concept of the smart energy grid.
As prices for clean energy and storage technologies continue to fall and nations explore ways to cut emissions, integrating higher shares of variable renewable energy becomes more urgent and complex.
© urbans78 / Adobe Stock

Investing and upgrading prepares the grid for evolving needs while raising investor confidence and making renewable energy projects and large-scale transmission expansions with long lead times more bankable. SURE works with partner countries to transform power systems and integrate higher shares of renewable energy while reducing costs and curtailment, preventing stagnation of large-scale renewable energy deployment, and enabling new business models such as electric vehicles, aggregation, demand-side management, and distributed energy resources. We train partners on modeling, analysis, and implementation skills to drive energy sector transformation.

VRE Forecasting in Indonesia, Kazakhstan, and Pakistan

USAID hosted an 1.5 hour-long webinar, Variable Renewable Energy Generation Forecasting and Integration with Dispatching: Experiences from Indonesia, Kazakhstan, and Pakistan, on Wednesday, October 13, 2021, that explored how Indonesia, Kazakhstan, and Pakistan navigated the complexities of VRE generation forecasting and its integration with dispatching and market operations. Each country’s grid began from a different starting point and is now in a distinct stage of integrated dispatching.

VRE forecasting helps utilities cost-effectively integrate utility-scale VRE and distributed energy resources into the power grid. Consumers will enjoy lower electricity prices and less risk of power outages as a result of the grid’s improved reliability and flexibility.

Mother holding her son in nature. Wind turbines in the background.
Through the Scaling Up Renewable Energy program, USAID helps partner countries power economies with renewable energy, meet international climate commitments, and open markets to private investment and competition.
© Viktor Pravdica / Adobe Stock

2017–ONGOING, GLOBAL – USAID’s Scaling Up Renewable Energy (SURE) program helps partner countries meet bold climate commitments by accelerating their transition to more affordable, reliable, and accessible energy that spurs growth, powers health systems, and reduces emissions.

Through SURE, USAID provides a variety of services to help policymakers, utilities, and regulators plan, procure, and integrate renewable energy, modernize energy sectors, and create policies that enable sustainable energy markets to flourish. As more countries aim to achieve net-zero emissions by 2050 and reduce hazardous waste, SURE’s Innovation Fund supports clean energy technology, energy efficiency innovation, and a circular economy for renewable energy equipment.

Video Transcript 
Hello my name is Kristen Madler. Welcome! I'm the clean energy coordinator at the U.S. Agency for International Development in the Center for Environment Energy and Infrastructure in Washington and I'm pleased to welcome you to the webinar by our program Scaling Up Renewable Energy or SURE. The title of today's webinar is Variable Renewable Energy Generation Forecasting and Integration with Dispatching. Next slide please.SURE is a global project that helps partner countries meet international climate commitments by increasing adoption of renewable energy technologies around the world. SURE's cadre of technical experts provide renewable energy planning, procurement, circular economy, and grid integration support services. SURE develops training, tools and resources to help policy makers, utilities, and regulators modernize their energy sectors and create policies that enable renewable energy markets to flourish. Next slide. As prices for clean energy and storage technologies continue to fall and nations explore ways to cut emissions, integrating higher shares of variable renewable energy or VRE becomes more urgent and complex. A key tool to mitigate the impact of VRE on system operations is VRE generation forecasting. During this webinar we will explore the opportunities and challenges that grids in Indonesia, Kazakhstan, and Pakistan face as they navigate the complexities of VRE forecasting and its integration with dispatching. We will then take questions. Next slide. USAID supports partner countries transform their power to be more green, reliable, and cost effective. A powerful way to achieve this goal is through a comprehensive approach to grid integration, system operations, and planning. We will discuss lessons learned and key takeaways from country experiences that can help market operators and regulators increase system reliability because better forecasting allows for significantly lower levels of volatility in the supply demand balance resulting in higher overall reliability; Improve system flexibility by providing accurate VRE generation forecasts. This allows the system to optimize its entire generation fleet which is often unutilized or underutilized; Reduce curtailments allowing more renewables to be economically scheduled and reduce reservoir requirements needed to manage deviations between dispatch generation and demand helping supply constrained systems. Next slide, please. A few housekeeping items to mention before we begin. Please write any questions you may have in the Q&A box at the bottom of the screen. Feel free to ask questions throughout the presentation and we will address them at the end during the Q&A session. The webinar will be recorded and available on the training and events section of the usaid.gov Scaling Up Renewable Energy site. For more information on grid integration visit the SURE website which can be found in the chat box. Next slide please. Today's webinar will include presentations from Pramod Jain, President of Innovative Wind Energy, one of SURE's subcontractors in the Tetra Tech lead consortium, Arman Azurmanyan, Chief of Party of USAID Power the Future and Farooq Qurban, Manager in Strategy and Market Development at the Central Power Purchasing Agency in Pakistan. Pramod has expertise in renewable energy grid integration studies, power markets, digitalization of power systems, wind and solar resource assessments, and renewable energy zones. Armen has expertise in energy sector reform, power system design and engineering, power systems stability and synchronous operations and utility operations and regulations. Farooq has expertise in power system operations and control, generation planning, regulatory framework, competitive markets design product development and simulations. He is also a certified energy risk professional. Without further ado, I will turn it over to Pramod, our first speaker. Thank you Kristen and good morning and or good afternoon and good evening to everyone. Next slide please. Let me start with some context about integration of renewable energy generation into the grid. It's pretty common knowledge now that the cost of procuring renewable energy has fallen below the cost of energy from a new conventional plant. In a lot of countries it has fallen below the cost of operating fossil fuel plants. So we're also seeing a sharp rise in country-specific RE targets and governments are pushing to make the power sector net zero emissions. So these two forces are leading to further acceleration of renewable energy deployments and so the question is no longer, "Is renewable energy cheap or is it at cost parity with conventional generation?" -- which it is -- so that question has been answered. Now the next question or the real question is what is the total cost of RE to the grid and to make my point let me provide an illustrative example. So in most auctions we are seeing renewable energy prices come in at about two to three cents per kilowatt hour for a 20-year fixed price PPA. Now let's look at the other cost component which is the grid integration cost and through prior history we've seen costs in the range of half a cent to five cents per kilowatt hour. So most grids like the U.S, EU, and China started out with very high costs of grid integration. They were in the range of 5 cents per kilowatt hour but over time they have reduced it through a variety of strategies. For instance the grid and ERCOT now in Texas in the U.S. is reporting costs as low as half a cent per kilowatt hour. So when we look at the total cost of grid integration we see that sorry the total cost of re to the grid we see grid integration can be a key key component of this cost and and a large component if it is not managed properly so the question really is what strategies can be used to reduce the cost of grid integration and USAID's SURE project which is sponsoring this webinar is providing technical assistance in this area to countries to essentially scale up renewable energy generation in the grid. Next slide please. This is a famous and often used chart with cost of grid integration on the y-axis and different interventions in the grid in order to integrate renewables and the interventions are on the x-axis. The lowest cost and the least difficult to implement interventions are renewable energy forecasting or generation forecasting and sub hourly dispatching and these are the two topics that are the focus of this webinar. Next slide please. So in simple terms renewable energy forecasting is short-term prediction of generation of a renewable energy plant. So the prediction has a lead time meaning we are doing a prediction either a week ahead of time day ahead or it could be an hour ahead. The other property of a forecast is the forecast interval. Are we forecasting generation for an hour, for 15 minutes or five minute intervals? Okay so, as an example if my lead time is one hour and the forecast interval is five minutes then at 08:00 AM in the morning I will be forecasting generation for 09:00 AM and it would be for 09:00 AM to 09:05, 09:05 to 09:10 and so on. So those are the parameters that will be repeated again and again in our presentations — in the following presentations. I wanted to just clear that up. So the inputs to forecasts are weather forecast, historical generation data, and available capacity of the RE plant. By available capacity, here we mean the number of turbines that are working as opposed to under maintenance in a wind farm and same thing with a solar farm the number of inverters that are actually operational in the plant. Next slide please. Now the question is why are we doing renewable energy generation forecasting and what is the purpose? So the purpose is essentially to create a dispatch plan which is to create a schedule of all the generators on the grid, right. So that is the purpose and the input to that process is what is the load going to be? What is the load forecast? What is the renewable energy generation forecast? And typically the difference between these two, the load forecast and the renewable energy generation forecast, is called the net load. and that is the net demand that the grid is going to see. And dispatching is done to schedule conventional generators to meet this net load demand. Next slide please. So now let's talk about the importance of renewable energy forecasting. As we know a dispatcher cannot really command a renewable energy plant to produce energy. Now this is very different from a conventional power plant that can be commanded and the reason is that a renewable energy plant may not have the resource, meaning a solar plant cannot be commanded to produce energy at 09:00 PM at night when there is no sun right. So the renewable energy plants are basically treated as must-run generators and so all the energy that is produced has to be absorbed by the grid, and because they have to be absorbed by the grid, the grid can manage the other conventional generation only if it gets an accurate forecast from the renewable energy plants. And this is the key point that when there are un-forecasted renewable energy fluctuations then the grid needs larger amount of reserves because it has to take care of these imbalances that renewable energy generation is going to create on the grid and the same thing happens with un-forecasted ramping of renewable energy plants which can affect system reliability and just require larger amount of reserves which are expensive. So what has been observed is that the value of renewable energy forecasting is very high compared to its cost and that's why it showed up in the lower left-hand corner of the NREL chart that I was showing in the previous slide. A good forecast allows a dispatcher to better plan and hence reduce the curtailment of renewable energy generation as well and the curtailment typically happens when the forecasts are of poor accuracy and the grid is unable to plan for this high injection of renewables. Next slide please. I want to make a few points about the benefits and sorry the benefits have been covered which is renewable energy forecasting reduces reserves, improves reliability, reduces curtailment of the grid, and improves system flexibility. Next slide please. So now I'd like to switch to some of the characteristics of forecasting that are important. So next slide. So one of the primary things that one can do to improve the accuracy of generation forecasting is to reduce the lead time. So this is a chart of the California Independent System Operator and a display of a solar plant. The day ahead forecast is the outer light blue line and hour ahead, the 30 minute ahead forecast is the red line and the actual is all the way inside. The point is that as you get closer to the delivery time of solar generation, the forecasts become more and more accurate. So this is the primary way in which accuracy of a forecast can be improved. So next slide please. So now that we know that shorter lead times, example 15 minutes or five minute ahead forecasting leads to higher accuracy. The next question then becomes "How do we reduce the lead time for dispatching?" Because you could have shorter lead times for forecasting, but if you don't use it in dispatching it's basically of no use. So the answer is that significant upgrades are required to the dispatching centers, processes, software tools, data integration SCADA systems and in general a higher amount of automation and digitalization in the dispatch center to be able to do faster dispatching. However international practices from grids with high penetration of renewable energy have demonstrated that faster forecasting with faster dispatching is the key to reducing the cost of grid integration although this is difficult but this is the key and this is how the European and the U.S. grids have been able to reduce the cost of integration. Next slide please. So for a power systems person and a renewable energy person who's interested in cloud computing, machine learning algorithms, parallel processing, so this area of renewable energy generation forecasting sort of is an exciting field because it's sort of at the— it is at the cutting edge of all these disciplines. And this is why I found it really really interesting. I just wanted to highlight the large number of inputs that go into it and the large amount of computing power that goes into creating not only weather forecasts but also creating renewable energy generation forecasts. Next slide please. So here I want to highlight some of the major IT components that go into the total system starting from the renewable energy plant, where the plant SCADA system collects all the data and then it's sent to a forecasting service. The forecasting service then combines it with a weather forecast and sends it to the dispatch center and the dispatch center's energy management system is the one that creates a security constrained unit commitment which commits the generators day ahead and then hour ahead or 15 minutes ahead. It also creates a security constraint economic dispatch which is the actual dispatch of the generators. Next slide please. So there are two approaches to renewable energy forecasting. First is the centralized approach. So in a centralized approach a single entity is taking on the responsibility for creating a generation forecast for all the solar and wind plants that are connected to the grid. So typically it is the dispatch center which would take this responsibility or some other entity within the system operator. So for VRE forecasts since the purpose is to improve dispatching and to reduce the amount of reserves this is the approach that has been adopted by by all the U.S. grids and the EU grids which is to go with centralized forecasting and so typically the costs are shared meaning and the system operator actually levies a charge to the renewable energy plants for doing forecasting on their behalf and the system operator hires two or more forecasting service companies and has in-house staff like a weather scientist and and other people who basically consolidate the forecasts from the two service providers. Two or three service providers deliver it to the dispatch center in a decentralized forecasting system. Each plant hires a forecasting service company to provide a forecast. Now you would ask, you know, as you can imagine, you know, without incentives or without penalties the dispatcher is going to get a poor quality forecast from the plant. There has to be an incentive that has to be provided to the plants. So this is typically required and then it becomes a a question of a carrot and a stick. However countries like India, Philippines, and others have implemented a decentralized forecasting method with penalties. Next slide please. So I have been fortunate to be a consultant on forecasting projects on these three USAID funded forecasting projects. So it supported forecasting for grid connected RE plans. So in Kazakhstan we did it through the Power the Future regional program and the 22 plants are part of the forecasting system. Now it is a hybrid approach meaning it has both a centralized component which the USAID project is providing and then it also gets forecasts from the renewable energy plants in Pakistan through the Sustainable Energy for Pakistan project. 21 wind plants are being forecasted and there we are doing it in a cluster so there are two clusters because the plants are located in specific geographical areas and that's how the forecasts are being done and this is a centralized approach. And the last one is South Sulawesi in Indonesia. This is one of the islands with wind generation and this was done through the SURE program and we're doing forecasting for four plants again in a hybrid mode. So with that I'm going to now turn over to the exciting part of the presentation which is, we will hear from these projects and their journeys through forecasting and integration of forecasting with dispatching. So now I'll turn it over to Armen to tell us about Kazakhstan's forecasting effort. Thank you. Bravo Pramod. Good morning, good evening everybody. Yeah let me share our experience in piloting of renewable energy forecasting in Kazakhstan. This has been done under Power the Future, a USAID funded project and we're lucky to have Pramod on our team to help us on this. Next please. So let me give you a short background on the power sector of Kazakhstan. The peak load is 12,000 megawatts. Total installed generation is around 22-23,000 megawatts out of which eighty percent is coal, ten percent is hydro, seven percent gas, and three and a half percent are renewable energy. The total renewable energy installed capacity is close to 2000 megawatts, out of which 30 percent is wind, and 70 of solar. Promoting renewable energy in Kazakhstan the government established a centralized off-taker of renewable energy, the Financial Settlement Center which is managing and administering the project which are signing power purchase agreements under the renewable energy development program and those plants are accounting for about 1,300 megawatts out of which 33 percent wind and 67 percent solar. And renewable energy targets for Kazakhstan is a leading country in the Central Asia region which is developing renewable energy and has very aggressive renewable energy targets. So by 2025 starting almost from zero actually in 2018 the target is to reach six percent of output from renewable energy, 15 by 2030, and 50 by 2050 but this 50 includes also alternative energy which might be nuclear generation and that decision is not made in Kazakhstan yet. Next please. So before involving the renewable energy forecasting and integration of forecasting into dispatch operations, we have initiated many tasks to help the Kazakhstan power system. To accommodate the increased renewable energy owners and the power sector of Kazakhstan lacks flexibility. In his presentation Pramod stated that traditional power plants can be dispatched yes —and no I mean Kazakhstan with 80 percent of coal-fired thermal power plants which are mostly old, outdated CHPs they almost lack any flexibility to be dispatched. So those are must-run so the room for flexibility in Kazakhstan is very very narrow and we are working with the government implementing market instruments—new products at the market—in order to boost development of flexibility, including flexibility in transmission and of course VRE forecasting and dispatching. So what was the status or still status of forecasting and dispatching in Kazakhstan? The dispatch procedures in Kazakhstan it is day ahead and we are reforecasting is done for 24 hours in one hour intervals. Dispatch is performed on a day ahead for 24 hours as well. The REF submission time is 10 am of previous day so some hours of VRE forecasts have lead time of 36 38 hours and as Pramod presented the accuracy of such forecasting is not good and final dispatch schedule is issued by NDC at 4 pm of the previous day. Again we're talking 30 hours for last hours of the operational day and the the accuracy is very poor. So what we are doing in in the pilot we are planning to move to 1 hour ahead and that information is provided by a centralized forecasting agency and that information is being provided to the Financial Settlement Center which transfers it to NDC. Unfortunately we are not at the stage of integrating that forecast into the dispatch schedules yet so that information is used for accuracy calculation. So where we want to get with changes we're proposing in Kazakhstan, so we will be still on day ahead schedule of planning but our schedules will be closed hour ahead of the operating hours. So the Financial Settlement Center will develop this forecast and submit them to the National Dispatch Center and that information will be integrated into into the balancing market which will start most likely on July 1st of 2022. And when the early plants are participating in the balancing market, the financial settlement of imbalances will be done by the Financial Settlement Center as a centralized entity for administering renewable energy in Kazakhstan. Next please. So here's the timeline of the pilot. So the pilot was initiated under ADB first: pilot one solar and one wind project was very involved and four forecasts per day were provided. That pilot lasted from April 2018 through the end of 2019. In 2019 we picked up that pilot to continue analyzing results and design a new pilot project involving four plants with hourly forecasts to improve understanding and planning and forecasting. In December of 2019 we have chosen two international vendors to provide forecasts. In February 2020 the USAID pilot forecasting begins based on the results we have seen in February 2021. The Financial Settlement Center and KEGOC requested USAID to expand the list of renewable energy plants included into the pilot. So in February we included 20 plants with one forecasting vendor. We scaled up our project to 22 plants in March of 2021 and we're working with Financial Settlement Center and KEGOC, National Dispatch Center of Kazakhstani power system to prepare integration of renewable energy forecasts into the dispatching. Next please. So here's the flowchart of how the forecasting process is organized. As you can see the Financial Settlement Center is in the core of data flow and organizing this forecasting process. So meter data flows from VRE plants to the Financial Settlement Center. The Financial Settlement Center posts availability information on the website and that is being transferred to the RE forecast service provided daily with VRE daily generation data and as Pramod mentioned the technology used by service provider, machine learning, they're developing forecasts for every hour for the next 24 hours. And that and—of course they are including the weather model in their calculations and the final forecast for the next 24 hours is available. And as we are planning, it shall be integrated into the dispatch of the national power system of Kazakhstan every hour. Next please. So what are the results of our pilot project so far? As I said 20 power plants were involved into the pilot up until April of 21 and then two more plants were added. From these curves we see we have achieved significant drop in mean absolute percentage error in respect—with respect to installed capacity and three lines are demonstrating day ahead provided by plant; red line is a Financial Settlement Center our ahead forecast and grey line is Financial Settlement Centers day ahead line. So you can see we dropped by more than twice or about twice from 12 percent. Four hours ahead to six percent below six percent and from 16 to 10 for day ahead. This achievement is used for benchmarking right now but as I said, as I said this will be integrated into the dispatching in the next steps of the pilot project. Next please. On this chart we are showing the improvement in forecasting for two specific plants. One is solar and another is wind for solar. We reached almost three percent of accuracy. We hope to see sustainable three percent because data exchange and availability information is still an issue. But we're improving the quality of forecasting and for wind power plants we reached seven percent which is a significant accomplishment for Kazakhstan as well. Because before starting this we were seeing forecasts close to 25 percent of accuracy. Next please. Lessons learned from this pilot. The quality of the regeneration of the data was a challenge since Kazakhstan did not have strong regulatory and technical standards on provision of measurement and provision of information. It took us six months to streamline and improve accuracy of communicated data from the plants. Plant availability is still an issue. Plants are not required to provide that information under local regulations. So its cooperative approach and based on good will provision but we're improving that communication as well. The quality of forecasts provided by the plants themselves improved in lockstep with the centralized service. It is important for system operators to use the quality, good quality forecast as a benchmark to keep plants accountable for the forecast they are providing. And as I said there are no penalties as of yet in Kazakhstan so it's cooperative and good will approach. But the system we are implementing for renewable energy plants as well, there will be consequences for poor forecasting or for not providing sufficient data. The next step is tighter integration of VRE forecast with day head dispatching. So we're working with National Dispatch Center and we have a work plan with them to develop the road map on how to integrate this short term forecasting in the dispatch operation. We are still working on integration of forecasted schedules into dispatch schedules metric for responsibility of accuracy of reforecasting is unassigned and should be assigned as Pramod stated, there are two approaches centralized and decentralized. So Kazakhstani authorities are still assessing what is the best for Kazakhstan on the methodology of forecasting. Our advice is to go centralized and through implementation of all steps I have described in my presentation, we are expecting that load-following capacity and regulation reserve requirement due to renewable energy plans will drop by half because of forecast errors and other procedures. Next please. That is all for Kazakhstan on a pilot project and with this I'll pass the floor to Farooq for presentation of Pakistani experience. Thank you, thank you Armen and good day everybody. So thanks for inviting me and thank you all for joining this webinar. So can we move to the next slide please. Starting with the country introduction. Let me tell you that in the last year the peak load served in Pakistan was around 24 gigawatt hours and we served around 30 watt hours in terms of energy. In terms of installed capacity Pakistan has a capacity of around 37 gigawatt hours, 37 gigawatts out of which 1200 megawatts is wind and 400 megawatts is grid connected solar. The targets for VRE generation are very aggressive where the government of Pakistan has decided to change its capacity max to 20 from the VRE resources by 2025 and 30 percent by 2030. If you look at the energy mix for the last year so the biggest share individually came from the hydro resources. But if we talk about the fossil fuels jointly the fossil fuels had the biggest share last year the the energy share from wind and solar generation was around three percent. And as I show you on the next slide if we can please move to the next the targets are aggressive and by 2030 55 percent of the capacity is planned to be from renewable resources—and by the way this renewable includes basically the hydro resources out of the total. So as we talked about in the last slide the target is that 30 percent of the capacity in total will be from VRE generation resources by 2030. Next please. So how we basically move towards this VRE forecasting and dispatching parameters probably. Pramod has excellently explained the reasons and the benefits of a need of having this VRE generation forecast and then its integration with the dispatching. Just to give you a small background initially. When we started with the wind generation the first project that became operational was in 2013 and initially basically we did not mandate the power projects to provide a forecast. But as the capacity started to increase and the dispatcher started facing problems with the intermittency of the VRE resources, there was a need to start doing a forecast and subsequently from— I think—the ninth project onward when the project was commissioned the projects were basically mandated under the contract to provide a forecast. And on the poor forecast there is a penalty as well but again this hybrid or you can say the distributed generation forecast from the VRE resources was not adding any value for the system operator. Because in any case the system operator needs to optimize the whole system and it needs to have a total forecast of the VRE resources so that it can optimally dispatch the system. So in 2020 we basically moved towards a central forecast service. What kinds of forecasts are we basically receiving? We are receiving forecasts on four horizons. First is the intraday forecast which is provided at hourly resolution and the forecast is provided by the forecast service provider at top of every hour for the next 24 hours. This intraday forecast is basically used by the control room operators to make their decisions in real time and in the medium term or you can say in the short term for the next couple of hours. The second one is a day ahead forecast for which the resolution is hourly and it is provided every day at 10:00 AM in the morning and its duration is for the next seven days. This forecast is being basically integrated with our day ahead dispatch planning and the system operator is using the forecast from this. Basically using this forecast for making a unit commitment to see and on day head basis. Then is the month ahead forecast for which the resolution is said to be daily and it is due to be provided one week before the start of each month and its duration is for the next month. So on the 23rd day of every month we basically, the control room gets a forecast for the next month on daily resolution and this forecast is basically utilized by the system operator and in making the monthly fuel plans and it basically optimizes the fuel import orders based on the forecast of this VRE generation. Then there is the year head forecast. The resolution for this forecast is monthly and it is due to be provided to the control room one month before the start of each year and it is provided for the next 12 months. This kind of forecast is being utilized by the system operator for preparation of the final production plan and to optimize our plant's generation and transmission outages for the next year. So effectively three of these forecasts: day ahead, month ahead, and year ahead, they are being integrated or you can say integrated about 90 with the dispatching but this intraday forecast. This is because the real time or you can say the EMS system and the SCADA is not fully functional at present so therefore this is only being used as a guide by the operator. But not fully integrated with the dispatching process. Next please. The IT architecture that we are using for managing all this data and because this is a huge data and it has been transmitted or communicated between multiple users in real time or every hour. So an IT architecture in any case was inevitable. So at present it starts with three parameters of information being provided by the wind power projects to this central forecast portal which is being administered and maintained at the market operator. There are four parameters that these wind power plants provide. One is the generation of the last hour second as if there had been any curtailment in the last hour. Third is the availability of its asset for the next four hours. So if none of the turbines are technically unavailable or on maintenance then they give availability that they are fully available. Their turbines are fully available for the next four hours and the fourth information is basically— sorry these are three pieces of information. So one is the curtailment of the last hour, the generation actual, and availability for the next three hours. So this information is being manually punched through a portal which is being transmitted through an automated process and through API to the forecast service provider. The forecast is provided uses this information from the site and it uses its own weather stations and utilizes its neural networks or any other mathematical basically modeling to provide a forecast. This forecast is provided through an API service back to this central forecast portal and from there this is basically provided to the relevant users that is the control room. And we also provide a forecast for each wind power project back to it each one. So in all this process, there is one thing that is manual that is basically data proven from site to the wind forecast portal and there we will be discussing in the lessons learned. And in the next slides that how we basically plan to automate this thing as well. So next please. So since the start of this project on 20 November of 2020. We are showing here the trend of certain forecast basically outcomes so if you look these two graphs are basically from a period of flow and season in Pakistan. So we are seeing that the absolute error basically ranges from around zero percent to twenty percent, but on an average the forecast is typically in the range of around ten percent. So this is basically on a day ahead basis and if we somehow look at the trends so although at times the absolute error goes to a high level but the trend that the forecast service provider is providing is pretty good. So this is being very helpful for the dispatchers to plan for their operations and a couple of hours ahead and also in their head planning and optimizations. Next please. Here we are showing the performance of this forecast service provider for the last, say five six months, and this is basically in the season of high wind season. The overall we are judging the accuracy on a metric of MAPE and that we calculate on on the installed capacity factor. But in this slide you are seeing on instance I mean on hourly resolution what has been the performance. So if we look towards somehow pretty start of the project in May 2021 there were 344 hours out of the total in the total month in which the absolute error was less than 20 and there were 375 instances or hours on which the error basically went beyond 10. So because as I told you that the data is being provided manually from site to the forecast service provider therefore we have divided the evaluation into two you can say—Matrix one is the performance of data, performance of the forecast service provider over the period where data was timely available to the forecast consultant and then there are instances where the second the last column basically shows where it is showing the results over the complete month so, we can say that there were, say, there might be 50 hours in a month in which the data was not timely provided to the to the forecast service provider but still we are using as a mayor of, you can say, to my year the performance of the forecast service provider so if we see especially during the high wind season the forecast I can see has been improving. So starting from 48 percent in May to 70 percent then 73 percent and 78 percent in July. So there were 78 percent of the hours in the month of August where the absolute error was less than 10 percent and this has been pretty helpful for the system operator in making its short-term decisions, especially from a few hours ahead up to day ahead and to optimize the research requirement in the system. So we expect that as we automate this data provision from site to the forecast service provider because we have analyzed instances where the data was timely provided the instances or the absolute error was very low in those instances. So as we move along and we automate the data process we expect that the performance will improve significantly. Next please. So here we are showing basically and a monthly performance so we see that the performance has been from February to this October in the range of six to eight percent on an average. So the map of six to eight percent on day ahead head basis—sorry this is this is calculated on three hours ahead—this is so within the six to eight percent over the installed capacity has been pretty good and especially in the season of high wind the forecast or the accuracy within this range has been, you can say, very useful for the system operator for plants and to optimize the dispatch scene and to optimize the reserves availability in the system at real-time basis. Next please. So probably we have talked about this but I'll quickly skim over the slide. How we are dispatching integration—integrating with the dispatching. So as I told you from day ahead to month ahead and year ahead this the forecast has been integrated with the dispatching system because we are making a dispatch optimization for the medium to long term on an offline tool. So this is fully integrated and the question is now for improvement of the forecast over the next year. So that we can get better results but for real time we are in the process of upgrading the SCADA project and it's allied EMS so as soon as this project has been implemented it will be in a year and a half probably by 2023 it is planned. So we are expecting that we will be, by that time we'll also be able to integrate the real-time data availability from site with the forecast service provider and it will be fully integrated. So that will be, in terms of if we talk about the intermittency of VRE because it has intermittent and short-term basis. So that will be the most helpful and most promising area where we can optimize further the operations of a power system. Next please. The problems that we have been facing or we faced during all this project initially was the availability of historical data because this data was not being maintained centrally by the system operator, market operator at a central database. So it took us around six to eight months to gather the data from all 21 wind farms but somehow this was an exercise a tedious exercise that was to be done once and the good thing is that by an exercise a one-time exercise of six to eight months we were able to collect data of around five years on hourly resolution and we provided it to the forecast service provider. The next challenge that we are facing and somehow because we have been in discussion with the forecast service provider as well that has the availability of data in real-time to the forecast service provider because again this is highly intermittent and it changes in the short-term basis so there is a need to integrate or to provide real-time data to the forecast service provider and in as much shorter shortest time as possible. So we are in the process of automating this data the possibilities or the options that we explored initially was that in the absence of a central SCADA or unless we get at the project, SCADA project fully commissioned one of the option was to integrate with the local DCR system at all sites but again there we faced some problems from the EPC contractors because there were different service providers at when project sites and somehow they were not, you can say, they were reluctant to let us integrate with their local DCS system so the alternate option was that we integrate with the AMR meters or the revenue meters that are installed at all sites. This data is being automated and we get the information at half hourly resolution of the actual energy that has been generated by each wind power plant. So we are now, this data is now available with us in our database. So now we are going to integrate this with the forecast service provider and we expect that within a month's time the forecast service provider will be having the real-time energy for the past hour on ready basis but again there will be two important inputs one as the great curtailment in the last hour. And second is the availability of wind turbines for the next. So this process will continue to be manual but again because these two parameters are not you can say highly intermittent or changing very rapidly in time. So we are expecting that integrating this energy generation from the revenue meters with the forecast service provider will provide us good forecasts. Again as we discussed the poor delivery adversely impacts the forecast model performance and as soon as the SCADA and EMS is available we plan to integrate this service with the EMS and then we are hopeful that we'll be able to further optimize the reserves requirements in the system. So I think this was all from the Pakistan experience and I now hand it over to Pramod for our experience with Indonesia. Let me, let's go to the next slide. So in Indonesia and the South Sulawesi island it is the one where there are wind projects. So this is really endowed with some very good wind resources and the grid statistics are the following and the peak load is 1.5 gigawatts. The low load is one gigawatt approximately and total generation of 2.18 gigawatts. Again a good mix of coal, hydro, and gas generation and it has two wind plants, Tolo which is 72 megawatts and Sidrap 75 megawatts. So as you can see we have about 10 in terms of capacity. When we compare it to peak load is the amount of wind generation. Now the Sulawesi Island has essentially two grids, the North Sulawesi which is still still not connected to the South Sulawesi grid has a peak load of 420 megawatts and it has two solar plants, 15 megawatts and 10 megawatts. So the USAID project SURE helped with the forecasting of all four of these projects. The two wind projects are in South Sulawesi and two in the north. Next slide. So the forecasting setup is the following and compared to Pakistan and Kazakhstan the difference here was we wind with a forecasting interval of 15 minutes and then we are still forecasting over 24 hours. This is both for day ahead and hour ahead and the South Sulawesi grid uses a hybrid model where all plants are required to provide a generation forecast and the dispatch center uses the centralized system that was provided by the USAID project. From a dispatching standpoint they are at a 30 minute interval in the day ahead and redispatching is done only when there is a big imbalance in the grid and this is done an hour ahead and the interval is still 30 minutes. Now in the day ahead dispatching process the renewable energy generation forecast is used in an offline mode meaning the the dispatching software is actually offline so files from the renewable energy generation forecasts are taken into the software and their head dispatchers are created. Let's go to the next slide. This is the timeline of the project. We started in October 2019, selected three three vendors and in June basically we had a long period where all the data flows had to be set up and it took about six months to set up the data flows and for the forecasting to actually start but in the meantime in March we started developing a process to track the accuracy which is to compare the accuracy of the forecast provided by the plants versus the centralized system. And in June 2020 we also designed this offline process of dispatching and integrating it with the dispatching software. In February 2021 and the first pilot ended where we were comparing three international vendors and then in February 2021 we moved to one vendor and this window was chosen based on the accuracy and and the reliability in delivering forecasts over the past year and now this process is continuing. And in the near future we expect to have an online dispatching software which is integrated with forecasting and we expect the software to do both day ahead and our head re-dispatching. Next slide please. So the metrics that we're using for accuracy are essentially two and pretty similar to Pakistan. We are using MAPE which means absolute percentage error and this count we are looking at the number of instances, number of hours when the error is less than 15 number of hours error is between 15 and 25 25 and 35 and greater than 35. So it gives us a good understanding of how inaccurate some of the forecasts are a mean absolute MAPE doesn't quite tell you this distribution. So we wanted to capture this distribution as well because a 35 percent inaccuracy can cause a lot of balancing issues in the grid. Next slide please. So here is some data for 2020 and these are the three vendors that we were using and it is for and the accuracy computation was done week by week for these six months. And as you can see most of the errors are in the range of 15 to 17 percent. Here is some more recent data and as you can see the errors have started to creep back up and the primary reason here was we had some significant data issues so there were data outages that resulted in the forecast accuracy to reduce significantly and more recently we have fixed these issues and the errors should go back down. Next slide please. We face similar issues with solar plants with the two solar plants in North Sulawesi earlier in the year around March and April time frame. But over time it has reduced but nevertheless as you can see the accuracy. Sorry the error is pretty high for solar generation forecasting. It's in the 20 range, 15 to 20 range and the primary reason is because it is an island. The weather patterns are extremely unpredictable and the cloud cover which primarily governs solar generation is extremely difficult to predict. Next slide please. So the process of integration is manual at this point. Because the dispatching tool is itself offline most of the interactions happen through a file transfer process. Now let me go to the second bullet which is hour ahead. So we designed a process, we piloted the process for about a month and then it had to be stopped because it was extremely tedious and it involved a lot of a lot of people spending a lot of time trying to do hour ahead dispatching with these hour ahead renewable energy forecasts. So we are currently waiting for an implementation of the automated dispatching tool which will integrate these feeds and then dispatching will become much easier. Next slide please. So the lessons learned from this project are that the challenges of integrating real-time generation data and getting accurate and timely generation data is one of the biggest challenges here in South Sulawesi and some of this has to do with the standards were not specified as to how the data should be sent when it should be sent and even the standards that were there are not were not quite enforced so we didn't get good data for this project. The next is as we are going to automated dispatching and installing an automated dispatching system and cooperation with the VRE forecasting vendor is crucial to make sure that there is a good handshake between the data exchanges and the data exchanges happen in a timely manner. One of the really exciting things that happened, one of the bright sides of this initiative was that in North Sulawesi a group of engineers sort of started implementing a machine learning algorithm for solar generation forecasting and they basically tied in with the status system of the solar plants. So they were able to get almost real-time generation data and real-time solar radiation data from the pyranometers in the plant and with that they were able to generate a five minute ahead forecast with extremely high accuracy. So it was a very good innovation from this team and the team then now intends to take this software and use it for wind plan generation forecasting and this is a screen off of the forecasting system that was developed by this team homegrown system and I'm very proud to report that it's doing very well. So this is the end of the country presentations and now I will turn it over to Kristen to moderate the questions. Thank you so much Pramod, Armen, and Farooq for those really interesting presentations. Just as a reminder to everyone please post any questions that you might have in the Q&A box. And let's see we'll kick it off with a question around, could you describe the typical cost of a VRE forecasting service? Yes the costs that we are seeing are in the range of 150 US Dollars per month per plant and typically these things are priced for a 50 megawatt renewable energy plant. So the cost is in the range of 150-250 for a plant per month. Fantastic, thank you. And what type of IT equipment is required by the system operators to manage the VRE forecasts if you can take that if you can describe the Pakistan experience please. Right so this basically deals with the database administrator I mean not very IT applications. It's all basically how you manage the data so again I'm not a technical IT technical person but again it I mean it all was basically how to handle large data transfers throughout in real time and it was all about databasing. So not much about the, you can say, a big IT architecture but if you can manage somehow a data center with the one to two software development experts you can manage this thing or you can also basically outsource for the database and you can also use any cloud services. So because in our case the system operator is not managing this all data transfer we at the market operator are basically handling all this thing and we have basically placed it initially. We used a cloud service and now we are moving it on to our own database at that site. Thank you. Could you talk a little bit about the kinds of incentives that are used in different grids that use the decentralized system to encourage higher accuracy forecasting? Yes, so the two examples of one is India and the other one is Philippines where there is a grace, sort of forecast error that is allowed. So for instance a forecast plant is allowed to have an error of say 15 but any errors beyond 15 there is a penalty, so the penalty could be five percent off the PPA tariff. If the accuracy or if the error is between 15 and 25 and then there's a different a higher penalty. If the error is between 25 and 35 percent and then the third level which is greater than 35 percent. So that is the scheme that India employs. The Philippines has a similar sort of stratification of penalties for error forecasting. So based on error for the forecasting right Pakistan has basically also said as I mentioned during this session that initially there were no, for the the plants were not mandated to provide a forecast but as the quantum increase and the number of plants increased we also basically mandated and obligated the plants coming in to provide a forecast and we also set a metric for them and if I read it out from, so there is a penalty basically we evaluate the absolute mean error over the whole year and if the mean absolute percentage error is between 15 percent to 25 percent for the first year. So there is a forecasting at a rebate and it is basically a percentage of energy payment we have linked it to the percentage of energy payment that has been (inaudible) by the seller. So if it is say for example between 15 to 25 percent in the first year there is a 10 percent error, forecasting error rebate. If the forecasting at a rebate is from 25 to 35 percent, so it is 10 percent plus one percent for each one percent variation in excess of 25 and then there is a 30 percent. If the error forecast error is above 35 percent so there are different options that can be used for a central forecast service provider. We are not basically penalizing anybody but if we have basically set a metric for the forecast service provider if it is unable to provide a good forecast. So it will lose this project and that's anyhow a penalty and for decentralized there is a penalty for the projects. Interesting thank you so much and in the Indonesia project it was mentioned that a homegrown machine learning algorithm was implemented by the dispatch center. That was very accurate so why shouldn't each grid follow this example and build their own VRE generation forecasting system? So a great question. So for really short term forecasting which is basically five minutes ahead and so, you can use a lot of what is called persistence method which is you just look at the generation data five minutes in the past and using that generation data and using the trend in the generation data you can then predict using machine learning what is going to happen in the next five minutes. So for really really short term forecasting machine learning algorithms and those work really well but if you have to go out an hour ahead or if you have to go out and a day ahead you really need a forecast a weather forecasting service and for that you have to hire one of these forecasting providers who have super computers running in the background basically churning away global models and downscaling of weather data to come up with good weather forecast and then convert it into a generation forecast. So it is not a substitute you can still have a homegrown system for extremely fast forecasting but for hour ahead and day ahead you certainly need a forecasting service. Thanks and in the Indonesia project why is the forecast error of the solar plant so high it was around 15 percent. Yes, normally we expect solar plants to have an error of around less than 5. I mean some are going as low as two percent error however Indonesia because it is an island and because there is just so much moisture and so much cloud movement and if you focused on the geography this was on the northern island that was going east west. So it's a very small strip of land and so the weather is extremely uncertain, extremely difficult to predict and that's why we are seeing such high inaccuracies. Okay thanks and in Kazakhstan, Pakistan, and Indonesia are the renewable energy plants required to have weather stations that measure the wind speed or solar radiation and provide it to the forecasting service. Maybe I should start with Kazakhstan and I would make this question a little bit bigger Kristen. The policy preconditions for renewable energy, I mean all countries which are getting into development of renewable energy, they start the regulatory and technical framework is not defined. The developers are given all excuses not to spend additional money, no additional risks, but over time systems are learning that without forecasting without specific technical equipment. The system will suffer, so in Kazakhstan there are no such requirements. Some plants are installing weather systems but we're working with the government to develop technical standards and requirements so each and every plant will provide information and that will be absolutely must. So yes Pramod please. So I'll then share the Pakistan experience. So in Indonesia it was required and actually all four plants have weather measurement equipment and they do provide data. However the data transfer doesn't work very well but they do have equipment. Now go ahead. Yes in Pakistan basically all the projects are mandated to install the nanometers and provide this data but this was under the agreement they were mandated to provide. This data to the power purchaser and the power purchaser basically transfers this data to the wind forecasters provider at present. So in short they are mandated to have the meters installed and provide this data. Again, today it is mandated to provide it to the power purchaser and then we can use it for any service that we want. Thank you and then going back to a question that came up during Armen's presentation with regard to the integration process. What were or what are the technological challenges and how were they addressed and what was the impact of that on the budget? We are not at that stage yet, we don't have a budget but the issues we will be dealing with very soon is how to integrate information into the system of scheduling and dispatching and we are talking fast dispatching here. So we will need digitized systems, we will need fast communication channels and we will need fast information from the power plants to implement it all. That is still to be assessed and planned to work with the dispatch center but the big picture is this. Thanks and can you talk about any security measures that have been made to protect the data as well as operations? Armen this was still from your presentation. If you have anything to add on that. We are not tapping into existing IT system of power system of Kazakhstan. So the security safeguards which are in place are very strong and we were not allowed into the system. So we're receiving information outside of the IT system of the Kazakhstan power system. So we're not making it more vulnerable or creating risk for them but once we establish a system of integrated dispatch where information should come from the plants and be integrated that will be a major issue and we still need to work on it. Okay great thanks and across all three of the case studies what is the indicator for the quality of the forecast? The assumption from the person that asked the question was root mean square deviation they asked what is your target? What does the root mean square deviation change with more spatial dispersion as renewable energy installations grow? So I can take that. So root means square is one of the metrics a more preferred metric is mean absolute percentage error, so you're not squaring the error and then taking a square root you're just taking the absolute value and the reason is sort of squaring. It skews the higher deviations in an unusual manner. So it's better to use mean absolute percentage error and then now what is acceptable is a hard thing to say clearly it is something very geography dependent. As I illustrated in Indonesia in an island there's very little you can do and so the best thing that a grid can do is sort of keep seeking better and better weather forecasting services, try out multiple vendors see who can provide better forecasts get their national weather scientists involved, national weather centers involved because hopefully they have better models as opposed to an international provider who's who may or may not have experience in Indonesia. It's better to get a local science group or a local university involved that may have created island-specific models. In Pakistan as you can see I mean the error rate has fallen in Kazakhstan. We are seeing a three percent error rate in solar forecasting so those are the rates to expect. Farooq or Armen, yeah if we just add so there can be multiple matrices but again as Pramod had said, we found it to be a good matrix but anyhow in terms of MAPE. You need to define over what horizon you are going to determine the MAPE because of such a huge data and when we are talking about going down to five minutes of forecast. So in any case you need to define what horizon you are going to determine the absolute mean absolute error and once you define that, then you need to define a probability metric as well that say if you are using one month then at what percentage at what probability at, what instances you expect that the forecast should be between these ranges? One other somehow you can say variable or metric can be how large is your system and what is your otherwise reserve requirement like if you have a system of say 25 mega 25 thousand megawatts and the the largest unit is 1,000 megawatts so you need to have a geometric sum of what can be the difference in your VRE generation and in power ahead or short-term basis. You top it up with your reserve requirement for a generator or transmission outage instance and then based on the system you can define like for our system when we initially started the VRE integration project in 2019. When this forecast service was not available we used a persistent approach, we used data of two years applied on it a persistent approach and then we had an arithmetic, oh sorry, a geometric sum with our reserve requirement for a generational transmission outage instance. So it came out that when you basically merge two to three different requirements. So it comes out that especially because of the wind it might not add a huge quantum but again if your dispatch, the energy mix is skewed towards a high VRE share then in any case you will need to have a stronger matrix or a stronger you can say a tighter limit for forecasting this or having the error range. So that you can optimize the system and you have the least problems and I would just add that is probably the most often asked question. What is the percentage of deviation we should aim for? There is no such number in the world and whereas it varies between countries and regions this is not about numbers. This is about process and methodology and the utilities need to strive to implement the best methods and tools to forecast and to define what is technically feasible in terms of forecasting and only then implement regulatory tools to penalize renewable energy producers or other means to bring the system into stable conditions. So this is about tools and processes which is a constant work for utilities and so were changes to the PPAs required for the penalties for the forecasting errors. And do you think that the application of the penalties has been effective? So in India the application of penalties has been quite effective and the main regulator the Central Electricity Regulatory Commission, sorry Central Energy Regulatory Commission has requested that the bans be reduced meaning initially 15 error was allowed, now they want to reduce it to 10. So no penalties if the error is less than 10 and then have other gradations So this has been I think on board for about two years. So the grid has learned and they are looking to tighten the accuracy requirements and charge a penalty so it has had a very favorable effect because penalties mean money is going out of a wind project and so there's a huge rush to improve accuracy, improve data, improve measurement and so on. Great! Well thank you so much. We're bumping up to our time. I want to thank all of our speakers for the insights and thank you all for attending. Before we adjourn, I'd like to ask you all to complete a brief evaluation survey which you will find in the chat and these responses really help to inform future webinars after this webinar. We will also email you the evaluation survey as well as the link to the SURE website where this webinar recording will be available on SURE's training and events page. So thank you all again for joining us and we look forward to seeing you at our next webinar! Thank you Kristen and thank you Armen and Farooq. It was a lot of fun! Thank you everybody! Thank you. Thank you. Thanks everyone.

Last updated: August 05, 2022

More Videos