Robust and automatic data cleansing method for short-term load forecasting of distribution feeders

Article


Huyghues-Beaufond, N., Tindemans, S., Falugi, P., Sun, M. and Strbac, G. 2020. Robust and automatic data cleansing method for short-term load forecasting of distribution feeders. Applied Energy. 261 (Art. 114405). https://doi.org/10.1016/j.apenergy.2019.114405
AuthorsHuyghues-Beaufond, N., Tindemans, S., Falugi, P., Sun, M. and Strbac, G.
Abstract

Distribution networks are undergoing fundamental changes at medium voltage level. To support growing planning and control decision-making, the need for large numbers of short-term load forecasts has emerged. Data-driven modelling of medium voltage feeders can be affected by (1) data quality issues, namely, large gross errors and missing observations (2) the presence of structural breaks in the data due to occasional network reconfiguration and load transfers. The present work investigates and reports on the effects of advanced data cleansing techniques on forecast accuracy. A hybrid framework to detect and remove outliers in large datasets is proposed; this automatic procedure combines the Tukey labelling rule and the binary segmentation algorithm to cleanse data more efficiently, it is fast and easy to implement. Various approaches for missing value imputation are investigated, including unconditional mean, Hot Deck via k-nearest neighbour and Kalman smoothing. A combination of the automatic detection/removal of outliers and the imputation methods mentioned above are implemented to cleanse time series of 342 medium-voltage feeders. A nested rolling-origin-validation technique is used to evaluate the feed-forward deep neural network models. The proposed data cleansing framework efficiently removes outliers from the data, and the accuracy of forecasts is improved. It is found that Hot Deck (k-NN) imputation performs best in balancing the bias-variance trade-off for short-term forecasting.

JournalApplied Energy
Journal citation261 (Art. 114405)
ISSN0306-2619
Year2020
PublisherElsevier
Accepted author manuscript
License
File Access Level
Anyone
Digital Object Identifier (DOI)https://doi.org/10.1016/j.apenergy.2019.114405
Publication dates
Online07 Jan 2020
Print07 Jan 2020
Publication process dates
Accepted14 Dec 2019
Deposited21 Aug 2023
Copyright holder© 2020, The Author(s)
Permalink -

https://repository.uel.ac.uk/item/8w422

Download files


Accepted author manuscript
data_cleansing.pdf
License: CC BY-NC-ND 4.0
File access level: Anyone

  • 90
    total views
  • 99
    total downloads
  • 9
    views this month
  • 5
    downloads this month

Export as

Related outputs

Machine Learning-Enhanced Benders Decomposition Approach for the Multi-Stage Stochastic Transmission Expansion Planning Problem
Borozan, S., Giannelos, S., Falugi, P., Moreira, A. and Strbac, G. 2024. Machine Learning-Enhanced Benders Decomposition Approach for the Multi-Stage Stochastic Transmission Expansion Planning Problem. Electric Power Systems Research. 237, p. Art. 110985. https://doi.org/10.1016/j.epsr.2024.110985
An integrated planning framework for optimal power generation portfolio including frequency and reserve requirements
Ayo, O, Falugi, P. and Strbac, G 2024. An integrated planning framework for optimal power generation portfolio including frequency and reserve requirements. IET Energy Systems Integration. In Press.
Automatic scenario generation for efficient solution of robust optimal control problems
Zagorowska, M., Falugi, P., O'Dwyer, E. and Kerrigan E. C. 2024. Automatic scenario generation for efficient solution of robust optimal control problems. International Journal of Robust and Nonlinear Control. 34 (2), pp. 1370-1396. https://doi.org/10.1002/rnc.7038
Automatic Scenario Generation for Robust Optimal Control Problems
Zagorowska, M., Falugi, P., O'Dwyer, E. and Kerrigan, E. C. 2023. Automatic Scenario Generation for Robust Optimal Control Problems. IFAC 2023: 22nd World Congress of the International Federation of Automatic Control. Yokohama, Japan 09 - 14 Jul 2023 Elsevier for the International Federation of Automatic Control. https://doi.org/10.1016/j.ifacol.2023.10.1743
Automating the data-driven predictive control design process for building thermal management
Falugi, P., O'Dwyer, E., Shah, N. and Kerrigan, E. C. 2022. Automating the data-driven predictive control design process for building thermal management. ECOS 2022 35th International Conference. Copenhagen, Denmark 03 - 07 Jul 2022 Danmarks Tekniske Universitet (DTU). https://doi.org/10.11581/dtu.00000267
A Modelling Workflow for Predictive Control in Residential Buildings
O’Dwyer, E., Atam, E., Falugi, P., Kerrigan, E. C., Zagorowska, M. A. and Shah, N. 2022. A Modelling Workflow for Predictive Control in Residential Buildings. in: Vahidinasab, V. and Mohammadi-Ivatloo, B. (ed.) Active Building Energy Systems: Operation and Control Springer, Cham. pp. 99-128
MPC and Optimal Design of Residential Buildings with Seasonal Storage: A Case Study
Falugi, P., O’Dwyer, E., Zagorowska, M. A., Atam, E., Kerrigan, E. C., Strbac, G. and Shah, N. 2022. MPC and Optimal Design of Residential Buildings with Seasonal Storage: A Case Study. in: Vahidinasab, V. and Mohammadi-Ivatloo, B. (ed.) Active Building Energy Systems: Operation and Control Springer, Cham. pp. 129-160
Fast and accurate method for computing non-smooth solutions to constrained control problems
Nita, L., Vila, E. M. G., Zagorowska, M. A., Kerrigan, E. C., Nie, Y., McInerney, I. and Falugi, P. 2022. Fast and accurate method for computing non-smooth solutions to constrained control problems. European Control Conference (ECC) 2022. London, UK 12 - 15 Jul 2022 IEEE. https://doi.org/10.23919/ECC55457.2022.9838569
Data-Driven Predictive Control With Improved Performance Using Segmented Trajectories
O’Dwyer, E., Kerrigan, E. C., Falugi, P., Zagorowska, M. and Shah, N. 2022. Data-Driven Predictive Control With Improved Performance Using Segmented Trajectories. IEEE Transactions on Control Systems Technology . 31 (3), pp. 1355 - 1365. https://doi.org/10.1109/TCST.2022.3224330
Predictive control co-design for enhancing flexibility in residential housing with battery degradation
Falugi, P., O’Dwyer, E., Kerrigan, E. C., Atam, E., Zagorowska, M. A., Strbac, G. and Shah, N. 2021. Predictive control co-design for enhancing flexibility in residential housing with battery degradation. 7th IFAC Conference on Nonlinear Model Predictive Control NMPC 2021. Bratislava, Slovakia 11 - 14 Jul 2021 Elsevier for the International Federation of Automatic Control. https://doi.org/10.1016/j.ifacol.2021.08.517
Long-Term Expansion Planning of the Transmission Network in India under Multi-Dimensional Uncertainty
Falugi, P., Giannelos S, Jain A., Borozan S., Moreira A., Bhakar R., Mathur J. and Strbac G. 2021. Long-Term Expansion Planning of the Transmission Network in India under Multi-Dimensional Uncertainty. Energies. 14 (22), p. 7813. https://doi.org/10.3390/en14227813