Energy Meteorology

Information fusion from multispectral sky imagers and weather sensors using computer vision and AI/ML models to improve solar generation nowcasting and coordinate real-time frequency response during cloud-driven irradiance variability. Rapid solar ramps caused by passing clouds create power imbalances that challenge grid reliability. This research develops sensing and forecasting technologies that provide decision support during solar ramps, enabling more efficient deployment of balancing reserves and frequency response resources.

Girasol Machine is a sky imager with far-infrared and multi-exposure capabilities in visible light (fisheye), mounted on a solar tracker so that the Sun remains centered in the sky images throughout the day. The sky imager has a noise-reduction software for the infrared images and an image fusion algorithm to merge fisheye sky images with different exposures. It produces low-noise infrared circumsolar images and high-dynamic-range all-sky images every 15 seconds (Terrén-Serrano et al., 2021). The Girasol Machine is equipped with a pyranometer.

The Girasol dataset is publicly accessible in a Dryad repository. The dataset comprises 300 sample days of multi-exposure fisheye visible-light and far-infrared sky images and irradiance measurements from the Girasol Machine, collected in Albuquerque, NM, USA.

Image processing embedded Girasol Machine: raw infrared image (left), atmospheric scatter radiation (middle), processed image (right).

Building on this hardware, we developed advanced signal and image processing methods to extract cloud dynamics from synchronized sky images and irradiance measurements, enabling integration into solar nowcasting and intra-hour forecasting frameworks (Terrén-Serrano & Martínez-Ramón, 2023). These methods include removing cyclostationary biases in irradiance signals, stabilizing infrared radiometric measurements, and using physics-informed machine-learning models to isolate atmospheric radiation and enhance cloud feature detection, ultimately improving forecast robustness and reducing operational costs.

Geospatial reprojections to account for the curvature of the Earth from the image's perspective from the viewpoint.

We also addressed the geometric distortions inherent to ground-based sky imaging systems by developing geospatial reprojection methods that transform image-plane coordinates into atmospheric cross-sections. These reprojections to account for the Earth’s curvature enable more accurate estimation of cloud position, motion, and spatial extent, particularly for systems mounted on solar trackers where perspective evolves dynamically over time (Terrén-Serrano & Martínez-Ramón, 2022)

This project develops an end-to-end AI/ML framework for solar nowcasting (very short-term forecasting) using a real-time stream of infrared sky images from the Girasol system, enabling continuous extraction of cloud features and dynamic updating of intra-hour forecasts.

End-to-end ML/AI pipeline for solar nowcasting using real-time infrared sky imaging and multi-sensor data fusion.

The embedded architecture comprises three learning modules.

Module 1 (Physics-informed learning): Estimates and visualizes wind velocity fields from image streams, enabling prediction of cloud motion and Sun-occlusion events (Terrén-Serrano & Martínez-Ramón, 2021).
Module 2 (Bayesian learning): Detects clouds and associates each cloud to an estimated wind flow using mixture models in a Hidden Markov Chain (Terrén-Serrano & Martínez-Ramón, 2023).
Module 3 (Probabilistic deep learning): Fuses multi-sensor data—including sky images, weather station measurements, and pyranometer data to generate probabilistic forecasts of solar irradiance over 1–15 minute horizons (Terrén-Serrano & Martínez-Ramón, 2023).

Forecasting event (bright green), very short-term probabilistic functional forecast (green), partially observed solar radiation (black), ground-truth unobserved radiation (red), persistence forecast (gray), and deterministic functional forecast orange.

Together, these modules form a unified computational framework for accurate, real-time solar forecasting to support reliable power system operations under uncertainty.

This work is part of my Ph.D. dissertation at the University of New Mexico (Terrén-Serrano, 2022).

References

2023

JOURNAL
Processing of global solar irradiance and ground-based infrared sky images for solar nowcasting and intra-hour forecasting applications

Guillermo Terrén-Serrano and Manel Martínez-Ramón

Solar Energy, Jan 2023

Abs DOI Bib HTML

Shadows from moving clouds in the troposphere impact the energy generated by photovoltaic systems. intra-hour solar forecast can be used to regulate solar energy dispatch. This investigation develops a data processing method for cloud dynamic feature extraction from raw sky images and Global Solar Irradiance (GSI) measurements that can be integrated into solar forecasting algorithms to reduce the operational supervision of hardware. Sky images and GSI measurements are acquired from a low-cost long-wave infrared radiometric camera and a pyranometer. This sky imager is mounted on a solar tracker that maintains the Sun in the center of sky images throughout the day. Multiple processing methods are proposed here that take advantage of a hybrid approach to approximate the optimal parameters of physical models using computationally inexpensive machine learning models. A signal processing method removes cyclostationary biases in high-resolution clear sky index values found when detrending GSI measurements using the clear sky GSI. Image processing methods are then used to remove the effects of atmospheric radiation and the Sun’s direct radiation from infrared sky images, plus the radiation effect emitted by debris on the sky imager’s germanium outdoor lens. The result is an adaptive solar forecasting algorithm that can reduce the operational cost of power grids with the high participation of solar energy in the generation mix.
@article{TERRENSERRANO2023111968, title = {Processing of global solar irradiance and ground-based infrared sky images for solar nowcasting and intra-hour forecasting applications}, journal = {Solar Energy}, volume = {264}, pages = {111968}, year = {2023}, issn = {0038-092X}, doi = {https://doi.org/10.1016/j.solener.2023.111968}, author = {Terrén-Serrano, Guillermo and Martínez-Ramón, Manel}, keywords = {Image processing, Long-wave infrared, Machine learning, Sky imaging, Sun tracking}, }
JOURNAL
Detection of clouds in multiple wind velocity fields using ground-based infrared sky images

Guillermo Terrén-Serrano and Manel Martínez-Ramón

Knowledge-Based Systems, Jan 2023

Abs DOI Bib HTML

To improve the performance of an intra-hour global solar irradiance forecasting algorithm, it is important to detect multiple layers of clouds. Horizontal atmospheric wind shear causes wind velocity fields to have different directions and speeds. In images of clouds acquired using ground-based sky imagers, clouds may be moving in different wind layers. The information provided by a solar forecasting algorithm is necessary to optimize and schedule the solar generation resources and storage devices in a smart grid. This investigation studies the performance of unsupervised learning techniques when detecting the number of cloud layers in infrared sky images. The images are acquired using an innovative infrared sky imager mounted on a solar tracker. Different mixture models are used to infer the distribution of the cloud features. Multiple Bayesian metrics and a sequential hidden Markov model are implemented to find the optimal number of clusters in the mixture models, and their performances are compared. The motion vectors are computed using a probabilistic implementation of the Lucas-Kanade algorithm. The correlations between the cloud motion vectors and temperatures are analyzed to discover the method that leads to the most accurate results. The findings point that a sequential hidden Markov model outperforms the detection accuracy of standard Bayesian model selection metrics.
@article{TERRENSERRANO2023110628, title = {Detection of clouds in multiple wind velocity fields using ground-based infrared sky images}, journal = {Knowledge-Based Systems}, volume = {274}, pages = {110628}, year = {2023}, issn = {0950-7051}, doi = {https://doi.org/10.1016/j.knosys.2023.110628}, author = {Terrén-Serrano, Guillermo and Martínez-Ramón, Manel}, keywords = {Cloud detection, Hidden Markov model, Mixture models, Sky imaging, Weighted Lucas-Kanade}, }
JOURNAL
Deep learning for intra-hour solar forecasting with fusion of features extracted from infrared sky images

Guillermo Terrén-Serrano and Manel Martínez-Ramón

Information Fusion, Jan 2023

Abs DOI Bib HTML

The increasing penetration of solar energy leaves power grids vulnerable to fluctuations in the solar radiation that reaches the surface of the Earth due to the projection of cloud shadows. Therefore, an intra-hour solar forecasting algorithm is necessary to reduce power instabilities caused by the impact of moving clouds on energy generation. The most accurate intra-hour solar forecasting methods apply convolutional neural networks to a series of visible light sky images. Instead, this investigation uses data acquired by a novel infrared sky imager on a solar tracker, which is capable of maintaining the Sun in the center of the images throughout the day and, at the same time, reducing the scattering effect produced by the Sun’s direct radiation. In addition, infrared sky images allow the derivation and extraction of physical cloud features. The cloud dynamics are analyzed in sequences of images to compute the probability of the Sun intercepting air parcels in the sky images (i.e., voxels). The method introduced in this investigation fuses sky condition information from multiple sensors (i.e., pyranometer, sky imager, solar tracker, weather station) and feature sources using a multi-task deep learning architecture based on recurrent neural networks. The proposed deterministic and Bayesian architectures reduce computation time by avoiding convolutional filters. The proposed intra-hour solar forecasting algorithm reached a forecast skill of 18.6% with a forecasting horizon of 8 min. Consequently, the proposed intra-hour solar forecasting method can potentially reduce the operational costs of power grids with high participation of solar energy.
@article{TERRENSERRANO202342, title = {Deep learning for intra-hour solar forecasting with fusion of features extracted from infrared sky images}, journal = {Information Fusion}, volume = {95}, pages = {42-61}, year = {2023}, issn = {1566-2535}, doi = {https://doi.org/10.1016/j.inffus.2023.02.006}, author = {Terrén-Serrano, Guillermo and Martínez-Ramón, Manel}, keywords = {Bayesian networks, Bayesian optimization, Deep learning, Girasol dataset, Solar forecasting, Sky imaging}, }

2022

JOURNAL
Geospatial Perspective Reprojections for Ground-Based Sky Imaging Systems

Guillermo Terrén-Serrano and Manel Martínez-Ramón

IEEE Transactions on Geoscience and Remote Sensing, 2022

Abs DOI Bib HTML

The intermittency of solar energy produces instabilities in power grids. These instabilities are reduced with an intrahour solar forecast that uses ground-based sky imaging systems. Sky imaging systems use lenses to acquire images concentrating light beams in a sensor. The light beams received by the sky imager have an elevation angle with respect to the device’s normal. Thus, the pixels in the image contain information from different areas of the sky within the imaging system field of view (FOV). The area of the FOV contained in the pixels increases as the elevation angle of the incident light beams decreases. When the sky imager is mounted on a solar tracker, the light beam’s angle of incidence in a pixel varies over time. This investigation formulates and compares two geospatial reprojections that transform the original Euclidean frame of the imager’s plane to the geospatial atmosphere cross section where the sky imager’s FOV intersects the cloud layer. One assumes that an object (i.e., cloud) moving in the troposphere is sufficiently far so the Earth’s surface is approximated flat. The other transformation takes into account the curvature of the Earth in the portion of the atmosphere (i.e., voxel) that is recorded. The results show that the differences between the dimensions calculated by both geospatial transformations are in the order of magnitude of kilometers when the Sun’s elevation angle is below 30°.
@article{TERRENSERRANO2022, author = {Terrén-Serrano, Guillermo and Martínez-Ramón, Manel}, journal = {IEEE Transactions on Geoscience and Remote Sensing}, title = {Geospatial Perspective Reprojections for Ground-Based Sky Imaging Systems}, year = {2022}, volume = {60}, number = {}, pages = {1-7}, keywords = {Cloud computing;Lenses;Cameras;Geospatial analysis;Earth;Clouds;Sun;Perspective reprojection;sky imaging;solar forecasting;solar tracking}, doi = {10.1109/TGRS.2022.3154710}, issn = {1558-0644}, month = {}, }
DISSERTATION
Intra-hour solar forecasting using cloud dynamics features extracted from ground-based infrared sky images

Guillermo Terrén-Serrano

The University of New Mexico, 2022

Abs Bib HTML

Due to the increasing use of photovoltaic systems, power grids are vulnerable to the projection of shadows from moving clouds. An intra-hour solar forecast provides power grids with the capability of automatically controlling the dispatch of energy, reducing the additional cost for a guaranteed, reliable supply of energy (i.e., energy storage). This dissertation introduces a novel sky imager consisting of a long-wave radiometric infrared camera and a visible light camera with a fisheye lens. The imager is mounted on a solar tracker to maintain the Sun in the center of the images throughout the day, reducing the scattering effect produced by the Sun’s direct radiation. Features of the cloud dynamics are analyzed to compute the probability of the Sun intercepting air parcels in the sky images. Probabilistic and deterministic multi-task intra-hour solar forecasting algorithms are introduced, based on kernel and deep learning methods, to increase the penetration of photovoltaic systems in power grids.
@phdthesis{terren2022intra, title = {Intra-hour solar forecasting using cloud dynamics features extracted from ground-based infrared sky images}, author = {Terr{\'e}n-Serrano, Guillermo}, year = {2022}, school = {The University of New Mexico}, keywords = {computer vision, deep learning, kernel learning, machine learning, sky imaging, solar forecasting.}, }

2021

JOURNAL
Girasol, a sky imaging and global solar irradiance dataset

Guillermo Terrén-Serrano, Adnan Bashir, Trilce Estrada, and 1 more author

Data in Brief, 2021

Abs DOI Bib HTML

The energy available in a microgrid that is powered by solar energy is tightly related to the weather conditions at the moment of generation. A very short-term forecast of solar irradiance provides the microgrid with the capability of automatically controlling the dispatch of energy. We propose a dataset to forecast Global Solar Irradiance (GSI) using a data acquisition system (DAQ) that simultaneously records sky imaging and GSI measurements, with the objective of extracting features from clouds and use them to forecast the power produced by a Photovoltaic (PV) system. The DAQ system is nicknamed the Girasol Machine (Girasol means Sunflower in Spanish). The sky imaging system consists of a longwave infrared (IR) camera and a visible (VI) light camera with a fisheye lens attached to it. The cameras are installed inside a weatherproof enclosure that it is mounted on a solar tracker. The tracker updates its pan and tilt every second using a solar position algorithm to maintain the Sun in the center of the IR and VI images. A pyranometer is situated on a horizontal mount next to the DAQ system to measure GSI. The dataset, composed of IR images, VI images, GSI measurements, and the Sun’s positions, has been tagged with timestamps.
@article{TERRENSERRANO2021106914, title = {Girasol, a sky imaging and global solar irradiance dataset}, journal = {Data in Brief}, volume = {35}, pages = {106914}, year = {2021}, issn = {2352-3409}, doi = {https://doi.org/10.1016/j.dib.2021.106914}, author = {Terrén-Serrano, Guillermo and Bashir, Adnan and Estrada, Trilce and Martínez-Ramón, Manel}, keywords = {Sky Imaging, Global Solar Irradiance, Fisheye Lens Camera, Long-wave Infrared Camera, Data Acquisition System, Solar Forecasting, Smart Grids, Sun-Tracking}, }
JOURNAL
Multi-layer wind velocity field visualization in infrared images of clouds for solar irradiance forecasting

Guillermo Terrén-Serrano and Manel Martínez-Ramón

Applied Energy, 2021

Abs DOI Bib HTML

The energy available in a solar energy powered grid is uncertain due to the weather conditions at the time of generation. Forecasting global solar irradiance could address this problem by providing the power grid with the capability of scheduling the storage and dispatch of energy. The occlusion of the Sun by clouds is the main cause of instabilities in the generation of solar energy. This investigation proposes a method to visualize the wind velocity field in sequences of longwave infrared images of clouds when there are multiple wind velocity fields in an image. This method can be used to forecast the occlusion of the Sun by clouds, providing stability in the generation of solar energy. Unsupervised learning is implemented to infer the distribution of the clouds’ velocity vectors and heights in multiple wind velocity fields in an infrared image. A multi-output weighted support vector machine with flow constraints is used to extrapolate the wind velocity fields to the entire frame, visualizing the path of the clouds. The proposed method is capable of approximating the wind velocity field in a small air parcel using the velocity vectors and physical features of clouds extracted from infrared images. Assuming that the streamlines are pathlines, the visualization of the wind velocity field can be used for forecasting cloud occlusions of the Sun. This is of importance when considering ways of increasing the stability of solar energy generation.
@article{TERRENSERRANO2021116656, title = {Multi-layer wind velocity field visualization in infrared images of clouds for solar irradiance forecasting}, journal = {Applied Energy}, volume = {288}, pages = {116656}, year = {2021}, issn = {0306-2619}, doi = {https://doi.org/10.1016/j.apenergy.2021.116656}, author = {Terrén-Serrano, Guillermo and Martínez-Ramón, Manel}, keywords = {Cloud tracking, Machine learning, Flow visualization, Beta mixture model, Sky imaging, Solar forecasting}, }