M. Sc. Yujiang He

AI for Computationally Intelligent Systems (AI4CIS)

Telefon: +49 561 804-6184
E-Mail: Yujiang.he[at]uni-kassel[dot]de

Standort: Wilhelmshöher Allee 67
34121 Kassel

Raum: 1106

Publikationen

[ 2024 ]
[ 2023 ]
[ 2022 ]
[ 2021 ]
[ 2020 ]
[ 2017 ]

2024[ to top ]

Time-Series Representation Learning via Heterogeneous Spatial-Temporal Contrasting for Remaining Useful Life Prediction. Huang, Zhixin; He, Yujiang; Nivarthi, Chandana Priya; Gruhl, Christian; Sick, Bernhard. In International Conference on Pattern Recognition (ICPR), bll 1–21. Springer, 2024.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
Classical contrastive learning paradigms rely on manual augmentations like cropping, masking, dropping, or adding noise randomly to create divergent sample views from original data. However, the choice of which method to manipulate samples is often subjective and may destroy the latent pattern of the sample. In response, this paper introduces a novel contrastive learning paradigm without choosing sample view augmentation methods, termed Heterogeneous Spatial-Temporal Representation Contrasting (HSTRC). Instead of sample view augmentation, we employ dual branches with a heterogeneous spatial-temporal flipped structure to extract two distinct hidden feature views from the same source data, which avoids disturbing the original time series. Leveraging a combination of cross-branch spatial-temporal contrastive and projected feature contrastive loss functions, HSTRC can effectively extract robust representations from unlabeled time series data. Remarkably, by only fine-tuning the fully connected layers on top of extracted representations by HSTRC, we achieve the best performance across several Remaining Useful Life prediction datasets, showing up to 19.2% improvements over the state-of-the-art supervised learning methods and classical contrastive learning paradigms. Besides, further intensive experiments demonstrate HSTRC's effectiveness in active learning, out-of-distribution testing, and transfer learning scenarios.
@inproceedings{huang2024time, abstract = {Classical contrastive learning paradigms rely on manual augmentations like cropping, masking, dropping, or adding noise randomly to create divergent sample views from original data. However, the choice of which method to manipulate samples is often subjective and may destroy the latent pattern of the sample. In response, this paper introduces a novel contrastive learning paradigm without choosing sample view augmentation methods, termed Heterogeneous Spatial-Temporal Representation Contrasting (HSTRC). Instead of sample view augmentation, we employ dual branches with a heterogeneous spatial-temporal flipped structure to extract two distinct hidden feature views from the same source data, which avoids disturbing the original time series. Leveraging a combination of cross-branch spatial-temporal contrastive and projected feature contrastive loss functions, HSTRC can effectively extract robust representations from unlabeled time series data. Remarkably, by only fine-tuning the fully connected layers on top of extracted representations by HSTRC, we achieve the best performance across several Remaining Useful Life prediction datasets, showing up to 19.2% improvements over the state-of-the-art supervised learning methods and classical contrastive learning paradigms. Besides, further intensive experiments demonstrate HSTRC's effectiveness in active learning, out-of-distribution testing, and transfer learning scenarios.}, author = {Huang, Zhixin and He, Yujiang and Nivarthi, Chandana Priya and Gruhl, Christian and Sick, Bernhard}, booktitle = {International Conference on Pattern Recognition (ICPR)}, keywords = {itegpub}, pages = {1–21}, publisher = {Springer}, title = {Time-Series Representation Learning via Heterogeneous Spatial-Temporal Contrasting for Remaining Useful Life Prediction}, year = 2024 }
%0 Conference Paper %1 huang2024time %A Huang, Zhixin %A He, Yujiang %A Nivarthi, Chandana Priya %A Gruhl, Christian %A Sick, Bernhard %B International Conference on Pattern Recognition (ICPR) %D 2024 %I Springer %P 1--21 %R 10.1007/978-3-031-78398-2_1 %T Time-Series Representation Learning via Heterogeneous Spatial-Temporal Contrasting for Remaining Useful Life Prediction %X Classical contrastive learning paradigms rely on manual augmentations like cropping, masking, dropping, or adding noise randomly to create divergent sample views from original data. However, the choice of which method to manipulate samples is often subjective and may destroy the latent pattern of the sample. In response, this paper introduces a novel contrastive learning paradigm without choosing sample view augmentation methods, termed Heterogeneous Spatial-Temporal Representation Contrasting (HSTRC). Instead of sample view augmentation, we employ dual branches with a heterogeneous spatial-temporal flipped structure to extract two distinct hidden feature views from the same source data, which avoids disturbing the original time series. Leveraging a combination of cross-branch spatial-temporal contrastive and projected feature contrastive loss functions, HSTRC can effectively extract robust representations from unlabeled time series data. Remarkably, by only fine-tuning the fully connected layers on top of extracted representations by HSTRC, we achieve the best performance across several Remaining Useful Life prediction datasets, showing up to 19.2% improvements over the state-of-the-art supervised learning methods and classical contrastive learning paradigms. Besides, further intensive experiments demonstrate HSTRC's effectiveness in active learning, out-of-distribution testing, and transfer learning scenarios.
PrOuD: Probabilistic Outlier Detection Solution for Time Series Analysis on Real-world Photovoltaic Inverters. He, Yujiang; Huang, Zhixin; Vogt, Stephan; Sick, Bernhard. In Energies (MDPI), 17(1), bl 64. MDPI, 2024.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Anomaly detection methods applied to time series are mostly viewed as a black box, which solely provides a deterministic answer to the detected target. Without a convincing explanation, domain experts can hardly trust the detection results and conduct a further time series diagnosis in real-world applications. To overcome the challenge, we mathematically analyzed the sources of anomalies and novelties in multivariate time series as well as their relationships from the perspective of Gaussian-distributed non-stationary noise. Furthermore, we proposed mathematical methods to generate artificial time series and synthetic anomalies, with the goal of solving the problem that it is difficult to train and evaluate models in real-world applications due to the lack of sufficient data. In addition, we designed extbfProbabilistic extbfOutlier extbfDetection (PrOuD), which is a general solution to provide interpretable detection results to assist domain experts in time series analysis. PrOuD can convert a predictive uncertainty of a trained model about a time series value into an estimated uncertainty of the detected outlier through Monte Carlo Estimation. The experimental results obtained on both artificial time series and real-world photovoltaic inverter data demonstrated that the proposed solution could detect emerging anomalies accurately and quickly. The implemented PrOuD demo case shows the potential to makes the detection results of the existing detection methods more convincing so that domain experts can more efficiently complete their tasks, such as time series diagnosis and anomalous pattern clustering.
@article{he2023proud, abstract = {Anomaly detection methods applied to time series are mostly viewed as a black box, which solely provides a deterministic answer to the detected target. Without a convincing explanation, domain experts can hardly trust the detection results and conduct a further time series diagnosis in real-world applications. To overcome the challenge, we mathematically analyzed the sources of anomalies and novelties in multivariate time series as well as their relationships from the perspective of Gaussian-distributed non-stationary noise. Furthermore, we proposed mathematical methods to generate artificial time series and synthetic anomalies, with the goal of solving the problem that it is difficult to train and evaluate models in real-world applications due to the lack of sufficient data. In addition, we designed extbf{Pr}obabilistic extbf{Ou}tlier extbf{D}etection (PrOuD), which is a general solution to provide interpretable detection results to assist domain experts in time series analysis. PrOuD can convert a predictive uncertainty of a trained model about a time series value into an estimated uncertainty of the detected outlier through Monte Carlo Estimation. The experimental results obtained on both artificial time series and real-world photovoltaic inverter data demonstrated that the proposed solution could detect emerging anomalies accurately and quickly. The implemented PrOuD demo case shows the potential to makes the detection results of the existing detection methods more convincing so that domain experts can more efficiently complete their tasks, such as time series diagnosis and anomalous pattern clustering.}, author = {He, Yujiang and Huang, Zhixin and Vogt, Stephan and Sick, Bernhard}, journal = {Energies (MDPI)}, keywords = {itegpub}, number = 1, pages = 64, publisher = {MDPI}, title = {PrOuD: Probabilistic Outlier Detection Solution for Time Series Analysis on Real-world Photovoltaic Inverters}, volume = 17, year = 2024 }
%0 Journal Article %1 he2023proud %A He, Yujiang %A Huang, Zhixin %A Vogt, Stephan %A Sick, Bernhard %D 2024 %I MDPI %J Energies (MDPI) %N 1 %P 64 %R 10.3390/en17010064 %T PrOuD: Probabilistic Outlier Detection Solution for Time Series Analysis on Real-world Photovoltaic Inverters %U https://www.mdpi.com/1996-1073/17/1/64 %V 17 %X Anomaly detection methods applied to time series are mostly viewed as a black box, which solely provides a deterministic answer to the detected target. Without a convincing explanation, domain experts can hardly trust the detection results and conduct a further time series diagnosis in real-world applications. To overcome the challenge, we mathematically analyzed the sources of anomalies and novelties in multivariate time series as well as their relationships from the perspective of Gaussian-distributed non-stationary noise. Furthermore, we proposed mathematical methods to generate artificial time series and synthetic anomalies, with the goal of solving the problem that it is difficult to train and evaluate models in real-world applications due to the lack of sufficient data. In addition, we designed extbfProbabilistic extbfOutlier extbfDetection (PrOuD), which is a general solution to provide interpretable detection results to assist domain experts in time series analysis. PrOuD can convert a predictive uncertainty of a trained model about a time series value into an estimated uncertainty of the detected outlier through Monte Carlo Estimation. The experimental results obtained on both artificial time series and real-world photovoltaic inverter data demonstrated that the proposed solution could detect emerging anomalies accurately and quickly. The implemented PrOuD demo case shows the potential to makes the detection results of the existing detection methods more convincing so that domain experts can more efficiently complete their tasks, such as time series diagnosis and anomalous pattern clustering.

2023[ to top ]

Spatio-Temporal Attention Graph Neural Network for Remaining Useful Life Prediction. Huang, Zhixin; He, Yujiang; Sick, Bernhard. In Computational Science and Computational Intelligence (CSCI), bll 99–105. IEEE, 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
RUL prediction plays a crucial role in the health management of industrial systems. Given the increasing complexity of systems, data-driven predictive models have attracted significant research interest. Upon reviewing the existing literature, it appears that many studies either do not fully integrate both spatial and temporal features or employ only a single attention mechanism. Furthermore, there seems to be inconsistency in the choice of data normalization methods, particularly concerning operating conditions, which might influence predictive performance. To bridge these observations, this study presents the Spatio-Temporal Attention Graph Neural Network. Our model combines graph neural networks and temporal convolutional neural networks for spatial and temporal feature extraction, respectively. The cascade of these extractors, combined with multihead attention mechanisms for both spatio-temporal dimensions, aims to improve predictive precision and refine model explainability. Comprehensive experiments were conducted on the CMAPSS dataset to evaluate the impact of unified versus clustering normalization. The findings suggest that our model performs state-of-the-art results using only the unified normalization. Additionally, when dealing with datasets with multiple operating conditions, cluster normalization enhances the performance of our proposed model by up to 27%.
@inproceedings{huang2023spatio, abstract = {RUL prediction plays a crucial role in the health management of industrial systems. Given the increasing complexity of systems, data-driven predictive models have attracted significant research interest. Upon reviewing the existing literature, it appears that many studies either do not fully integrate both spatial and temporal features or employ only a single attention mechanism. Furthermore, there seems to be inconsistency in the choice of data normalization methods, particularly concerning operating conditions, which might influence predictive performance. To bridge these observations, this study presents the Spatio-Temporal Attention Graph Neural Network. Our model combines graph neural networks and temporal convolutional neural networks for spatial and temporal feature extraction, respectively. The cascade of these extractors, combined with multihead attention mechanisms for both spatio-temporal dimensions, aims to improve predictive precision and refine model explainability. Comprehensive experiments were conducted on the CMAPSS dataset to evaluate the impact of unified versus clustering normalization. The findings suggest that our model performs state-of-the-art results using only the unified normalization. Additionally, when dealing with datasets with multiple operating conditions, cluster normalization enhances the performance of our proposed model by up to 27%.}, author = {Huang, Zhixin and He, Yujiang and Sick, Bernhard}, booktitle = {Computational Science and Computational Intelligence (CSCI)}, keywords = {Graph_Neural_Network}, pages = {99–105}, publisher = {IEEE}, title = {Spatio-Temporal Attention Graph Neural Network for Remaining Useful Life Prediction}, year = 2023 }
%0 Conference Paper %1 huang2023spatio %A Huang, Zhixin %A He, Yujiang %A Sick, Bernhard %B Computational Science and Computational Intelligence (CSCI) %D 2023 %I IEEE %P 99--105 %R 10.1109/CSCI62032.2023.00022 %T Spatio-Temporal Attention Graph Neural Network for Remaining Useful Life Prediction %X RUL prediction plays a crucial role in the health management of industrial systems. Given the increasing complexity of systems, data-driven predictive models have attracted significant research interest. Upon reviewing the existing literature, it appears that many studies either do not fully integrate both spatial and temporal features or employ only a single attention mechanism. Furthermore, there seems to be inconsistency in the choice of data normalization methods, particularly concerning operating conditions, which might influence predictive performance. To bridge these observations, this study presents the Spatio-Temporal Attention Graph Neural Network. Our model combines graph neural networks and temporal convolutional neural networks for spatial and temporal feature extraction, respectively. The cascade of these extractors, combined with multihead attention mechanisms for both spatio-temporal dimensions, aims to improve predictive precision and refine model explainability. Comprehensive experiments were conducted on the CMAPSS dataset to evaluate the impact of unified versus clustering normalization. The findings suggest that our model performs state-of-the-art results using only the unified normalization. Additionally, when dealing with datasets with multiple operating conditions, cluster normalization enhances the performance of our proposed model by up to 27%.
Active Learning with Fast Model Updates and Class-Balanced Selection for Imbalanced Datasets. Huang, Zhixin; He, Yujiang; Herde, Marek; Huseljic, Denis; Sick, Bernhard. In Workshop on Interactive Adapative Learning (IAL), ECML PKDD, bll 28–45. 2023.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Active learning strategies aim to minimize the number of queried samples for model training. However, two challenges in pool-based deep active learning on imbalanced datasets are observed in experiments: (1) the declining performance of active learning strategies due to imbalanced class distribution; (2)~the lack of sample diversity in acquisition batches due to the absence of timely model updates. This paper proposes the AL-FaMoUS, a general solution combining fast model updates and class-balanced minibatch selection to the active learning process. Furthermore, a simplification of the AL-FaMoUS, which selects one single sample in each acquisition minibatch, is experimentally evaluated on four image and three time-series imbalanced datasets. The results demonstrate that the implemented AL-FaMoUS outperforms the other adopted AL strategies, including uncertainty sampling and BALD solely combined with either the fast model update or the class balance selection strategy, in terms of accuracy and Macro F1 score.
@inproceedings{huang2023active, abstract = {Active learning strategies aim to minimize the number of queried samples for model training. However, two challenges in pool-based deep active learning on imbalanced datasets are observed in experiments: (1) the declining performance of active learning strategies due to imbalanced class distribution; (2) the lack of sample diversity in acquisition batches due to the absence of timely model updates. This paper proposes the AL-FaMoUS, a general solution combining fast model updates and class-balanced minibatch selection to the active learning process. Furthermore, a simplification of the AL-FaMoUS, which selects one single sample in each acquisition minibatch, is experimentally evaluated on four image and three time-series imbalanced datasets. The results demonstrate that the implemented AL-FaMoUS outperforms the other adopted AL strategies, including uncertainty sampling and BALD solely combined with either the fast model update or the class balance selection strategy, in terms of accuracy and Macro F1 score.}, author = {Huang, Zhixin and He, Yujiang and Herde, Marek and Huseljic, Denis and Sick, Bernhard}, booktitle = {Workshop on Interactive Adapative Learning (IAL), ECML PKDD}, keywords = {itegpub}, pages = {28–45}, title = {Active Learning with Fast Model Updates and Class-Balanced Selection for Imbalanced Datasets}, year = 2023 }
%0 Conference Paper %1 huang2023active %A Huang, Zhixin %A He, Yujiang %A Herde, Marek %A Huseljic, Denis %A Sick, Bernhard %B Workshop on Interactive Adapative Learning (IAL), ECML PKDD %D 2023 %P 28--45 %T Active Learning with Fast Model Updates and Class-Balanced Selection for Imbalanced Datasets %U https://ceur-ws.org/Vol-3470/paper5.pdf %X Active learning strategies aim to minimize the number of queried samples for model training. However, two challenges in pool-based deep active learning on imbalanced datasets are observed in experiments: (1) the declining performance of active learning strategies due to imbalanced class distribution; (2)~the lack of sample diversity in acquisition batches due to the absence of timely model updates. This paper proposes the AL-FaMoUS, a general solution combining fast model updates and class-balanced minibatch selection to the active learning process. Furthermore, a simplification of the AL-FaMoUS, which selects one single sample in each acquisition minibatch, is experimentally evaluated on four image and three time-series imbalanced datasets. The results demonstrate that the implemented AL-FaMoUS outperforms the other adopted AL strategies, including uncertainty sampling and BALD solely combined with either the fast model update or the class balance selection strategy, in terms of accuracy and Macro F1 score.

2022[ to top ]

Design of Explainability Module with Experts in the Loop for Visualization and Dynamic Adjustment of Continual Learning. He, Yujiang; Huang, Zhixin; Sick, Bernhard. In Workshop on Interactive Machine Learning Workshop (IMLW), AAAI, bll 1–6. 2022.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Continual learning can enable neural networks to evolve by learning new tasks sequentially in task-changing scenarios. However, two general challenges should be overcome in further research before we apply this technique to real-world applications. Firstly, newly collected novelties from the data stream in applications could contain anomalies that are meaningless for continual learning. Instead of viewing them as a new task for updating, we have to filter out such anomalies to reduce the disturbance of extremely high-entropy data for the progression of convergence. Secondly, fewer efforts have been put into research regarding the explainability of continual learning, which leads to a lack of transparency and credibility of the updated neural networks. Elaborated explanations about the process and result of continual learning can help experts in judgment and making decisions. Therefore, we propose the conceptual design of an explainability module with experts in the loop based on techniques, such as dimension reduction, visualization, and evaluation strategies. This work aims to overcome the mentioned challenges by sufficiently explaining and visualizing the identified anomalies and the updated neural network. With the help of this module, experts can be more confident in decision-making regarding anomaly filtering, dynamic adjustment of hyperparameters, data backup, etc.
@inproceedings{he2022design, abstract = {Continual learning can enable neural networks to evolve by learning new tasks sequentially in task-changing scenarios. However, two general challenges should be overcome in further research before we apply this technique to real-world applications. Firstly, newly collected novelties from the data stream in applications could contain anomalies that are meaningless for continual learning. Instead of viewing them as a new task for updating, we have to filter out such anomalies to reduce the disturbance of extremely high-entropy data for the progression of convergence. Secondly, fewer efforts have been put into research regarding the explainability of continual learning, which leads to a lack of transparency and credibility of the updated neural networks. Elaborated explanations about the process and result of continual learning can help experts in judgment and making decisions. Therefore, we propose the conceptual design of an explainability module with experts in the loop based on techniques, such as dimension reduction, visualization, and evaluation strategies. This work aims to overcome the mentioned challenges by sufficiently explaining and visualizing the identified anomalies and the updated neural network. With the help of this module, experts can be more confident in decision-making regarding anomaly filtering, dynamic adjustment of hyperparameters, data backup, etc.}, author = {He, Yujiang and Huang, Zhixin and Sick, Bernhard}, booktitle = {Workshop on Interactive Machine Learning Workshop (IMLW), AAAI}, keywords = {itegpub}, pages = {1–6}, title = {Design of Explainability Module with Experts in the Loop for Visualization and Dynamic Adjustment of Continual Learning}, year = 2022 }
%0 Conference Paper %1 he2022design %A He, Yujiang %A Huang, Zhixin %A Sick, Bernhard %B Workshop on Interactive Machine Learning Workshop (IMLW), AAAI %D 2022 %P 1--6 %T Design of Explainability Module with Experts in the Loop for Visualization and Dynamic Adjustment of Continual Learning %U https://arxiv.org/abs/2202.06781 %X Continual learning can enable neural networks to evolve by learning new tasks sequentially in task-changing scenarios. However, two general challenges should be overcome in further research before we apply this technique to real-world applications. Firstly, newly collected novelties from the data stream in applications could contain anomalies that are meaningless for continual learning. Instead of viewing them as a new task for updating, we have to filter out such anomalies to reduce the disturbance of extremely high-entropy data for the progression of convergence. Secondly, fewer efforts have been put into research regarding the explainability of continual learning, which leads to a lack of transparency and credibility of the updated neural networks. Elaborated explanations about the process and result of continual learning can help experts in judgment and making decisions. Therefore, we propose the conceptual design of an explainability module with experts in the loop based on techniques, such as dimension reduction, visualization, and evaluation strategies. This work aims to overcome the mentioned challenges by sufficiently explaining and visualizing the identified anomalies and the updated neural network. With the help of this module, experts can be more confident in decision-making regarding anomaly filtering, dynamic adjustment of hyperparameters, data backup, etc.
Adaptive Explainable Continual Learning Framework for Regression Problems with Focus on Power Forecasts. He, Yujiang. In Organic Computing -- Doctoral Dissertation Colloquium 2021, S. Tomforde, C. Krupitzer (reds.), bll 125–140. kassel university press, 2022.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
Compared with traditional deep learning techniques, continual learning enables deep neural networks to learn continually and adaptively. Deep neural networks have to learn unseen tasks and overcome forgetting the knowledge obtained from previously learned tasks as the amount of data keeps increasing in applications. This article proposes two continual learning application scenarios, i.e., the target-domain incremental scenario and the data-domain incremental scenario, to describe the potential challenges in this context. Based on our previous work regarding the CLeaR framework, which is short for continual learning for regression tasks, models will be enabled to extend themselves and to learn data successively. Research topics are related, but not limited, to developing continual deep learning algorithms, strategies for non-stationarity detection in data streams, explainable and visualizable artificial intelligence, etc. Moreover, the framework- and algorithm-related hyperparameters should be dynamically updated in applications. Forecasting experiments will be conducted based on power generation and consumption data collected from real-world applications. A series of comprehensive evaluation metrics and visualization tools are applied to access the experimental results. The proposed framework is expected to be generally applied to other constantly changing scenarios.
@incollection{he2022adaptive, abstract = {Compared with traditional deep learning techniques, continual learning enables deep neural networks to learn continually and adaptively. Deep neural networks have to learn unseen tasks and overcome forgetting the knowledge obtained from previously learned tasks as the amount of data keeps increasing in applications. This article proposes two continual learning application scenarios, i.e., the target-domain incremental scenario and the data-domain incremental scenario, to describe the potential challenges in this context. Based on our previous work regarding the CLeaR framework, which is short for continual learning for regression tasks, models will be enabled to extend themselves and to learn data successively. Research topics are related, but not limited, to developing continual deep learning algorithms, strategies for non-stationarity detection in data streams, explainable and visualizable artificial intelligence, etc. Moreover, the framework- and algorithm-related hyperparameters should be dynamically updated in applications. Forecasting experiments will be conducted based on power generation and consumption data collected from real-world applications. A series of comprehensive evaluation metrics and visualization tools are applied to access the experimental results. The proposed framework is expected to be generally applied to other constantly changing scenarios.}, author = {He, Yujiang}, booktitle = {Organic Computing – Doctoral Dissertation Colloquium 2021}, editor = {Tomforde, Sven and Krupitzer, Christian}, keywords = {itegpub}, pages = {125–140}, publisher = {kassel university press}, title = {Adaptive Explainable Continual Learning Framework for Regression Problems with Focus on Power Forecasts}, year = 2022 }
%0 Book Section %1 he2022adaptive %A He, Yujiang %B Organic Computing -- Doctoral Dissertation Colloquium 2021 %D 2022 %E Tomforde, Sven %E Krupitzer, Christian %I kassel university press %P 125--140 %T Adaptive Explainable Continual Learning Framework for Regression Problems with Focus on Power Forecasts %X Compared with traditional deep learning techniques, continual learning enables deep neural networks to learn continually and adaptively. Deep neural networks have to learn unseen tasks and overcome forgetting the knowledge obtained from previously learned tasks as the amount of data keeps increasing in applications. This article proposes two continual learning application scenarios, i.e., the target-domain incremental scenario and the data-domain incremental scenario, to describe the potential challenges in this context. Based on our previous work regarding the CLeaR framework, which is short for continual learning for regression tasks, models will be enabled to extend themselves and to learn data successively. Research topics are related, but not limited, to developing continual deep learning algorithms, strategies for non-stationarity detection in data streams, explainable and visualizable artificial intelligence, etc. Moreover, the framework- and algorithm-related hyperparameters should be dynamically updated in applications. Forecasting experiments will be conducted based on power generation and consumption data collected from real-world applications. A series of comprehensive evaluation metrics and visualization tools are applied to access the experimental results. The proposed framework is expected to be generally applied to other constantly changing scenarios.

2021[ to top ]

Uncertainty and Utility Sampling with Pre-Clustering. Huang, Zhixin; He, Yujiang; Vogt, Stephan; Sick, Bernhard. In Workshop on Interactive Adaptive Learning (IAL), ECML PKDD. 2021.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{huang2021uncertainty, author = {Huang, Zhixin and He, Yujiang and Vogt, Stephan and Sick, Bernhard}, booktitle = {Workshop on Interactive Adaptive Learning (IAL), ECML PKDD}, keywords = {itegpub}, note = {(accepted)}, title = {Uncertainty and Utility Sampling with Pre-Clustering}, year = 2021 }
%0 Conference Paper %1 huang2021uncertainty %A Huang, Zhixin %A He, Yujiang %A Vogt, Stephan %A Sick, Bernhard %B Workshop on Interactive Adaptive Learning (IAL), ECML PKDD %D 2021 %T Uncertainty and Utility Sampling with Pre-Clustering
Toward Application of Continuous Power Forecasts in a Regional Flexibility Market. He, Yujiang; Huang, Zhixin; Sick, Bernhard. In International Joint Conference on Neural Networks (IJCNN), bll 1–8. IEEE, 2021.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{he2021application, author = {He, Yujiang and Huang, Zhixin and Sick, Bernhard}, booktitle = {International Joint Conference on Neural Networks (IJCNN)}, keywords = {itegpub}, pages = {1–8}, publisher = {IEEE}, title = {Toward Application of Continuous Power Forecasts in a Regional Flexibility Market}, year = 2021 }
%0 Conference Paper %1 he2021application %A He, Yujiang %A Huang, Zhixin %A Sick, Bernhard %B International Joint Conference on Neural Networks (IJCNN) %D 2021 %I IEEE %P 1--8 %R 10.1109/ijcnn52387.2021.9533626 %T Toward Application of Continuous Power Forecasts in a Regional Flexibility Market
CLeaR: An adaptive continual learning framework for regression tasks. He, Yujiang; Sick, Bernhard. In AI Perspectives, 3(1), bl 2. Springer, 2020.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Catastrophic forgetting means that a trained neural network model gradually forgets the previously learned tasks when being retrained on new tasks. Overcoming the forgetting problem is a major problem in machine learning. Numerous continual learning algorithms are very successful in incremental learning of classification tasks, where new samples with their labels appear frequently. However, there is currently no research that addresses the catastrophic forgetting problem in regression tasks as far as we know. This problem has emerged as one of the primary constraints in some applications, such as renewable energy forecasts. This article clarifies problem-related definitions and proposes a new methodological framework that can forecast targets and update itself by means of continual learning. The framework consists of forecasting neural networks and buffers, which store newly collected data from a non-stationary data stream in an application. The changed probability distribution of the data stream, which the framework has identified, will be learned sequentially. The framework is called CLeaR (Continual Learning for Regression Tasks), where components can be flexibly customized for a specific application scenario. We design two sets of experiments to evaluate the CLeaR framework concerning fitting error (training), prediction error (test), and forgetting ratio. The first one is based on an artificial time series to explore how hyperparameters affect the CLeaR framework. The second one is designed with data collected from European wind farms to evaluate the CLeaR framework's performance in a real-world application. The experimental results demonstrate that the CLeaR framework can continually acquire knowledge in the data stream and improve the prediction accuracy. The article concludes with further research issues arising from requirements to extend the framework.
@article{he2021clear, abstract = {Catastrophic forgetting means that a trained neural network model gradually forgets the previously learned tasks when being retrained on new tasks. Overcoming the forgetting problem is a major problem in machine learning. Numerous continual learning algorithms are very successful in incremental learning of classification tasks, where new samples with their labels appear frequently. However, there is currently no research that addresses the catastrophic forgetting problem in regression tasks as far as we know. This problem has emerged as one of the primary constraints in some applications, such as renewable energy forecasts. This article clarifies problem-related definitions and proposes a new methodological framework that can forecast targets and update itself by means of continual learning. The framework consists of forecasting neural networks and buffers, which store newly collected data from a non-stationary data stream in an application. The changed probability distribution of the data stream, which the framework has identified, will be learned sequentially. The framework is called CLeaR (Continual Learning for Regression Tasks), where components can be flexibly customized for a specific application scenario. We design two sets of experiments to evaluate the CLeaR framework concerning fitting error (training), prediction error (test), and forgetting ratio. The first one is based on an artificial time series to explore how hyperparameters affect the CLeaR framework. The second one is designed with data collected from European wind farms to evaluate the CLeaR framework's performance in a real-world application. The experimental results demonstrate that the CLeaR framework can continually acquire knowledge in the data stream and improve the prediction accuracy. The article concludes with further research issues arising from requirements to extend the framework.}, author = {He, Yujiang and Sick, Bernhard}, journal = {AI Perspectives}, keywords = {itegpub}, number = 1, pages = 2, publisher = {Springer}, title = {CLeaR: An adaptive continual learning framework for regression tasks}, volume = 3, year = 2021 }
%0 Journal Article %1 he2021clear %A He, Yujiang %A Sick, Bernhard %D 2021 %I Springer %J AI Perspectives %N 1 %P 2 %R 10.1186/s42467-021-00009-8 %T CLeaR: An adaptive continual learning framework for regression tasks %U https://doi.org/10.1186/s42467-021-00009-8 %V 3 %X Catastrophic forgetting means that a trained neural network model gradually forgets the previously learned tasks when being retrained on new tasks. Overcoming the forgetting problem is a major problem in machine learning. Numerous continual learning algorithms are very successful in incremental learning of classification tasks, where new samples with their labels appear frequently. However, there is currently no research that addresses the catastrophic forgetting problem in regression tasks as far as we know. This problem has emerged as one of the primary constraints in some applications, such as renewable energy forecasts. This article clarifies problem-related definitions and proposes a new methodological framework that can forecast targets and update itself by means of continual learning. The framework consists of forecasting neural networks and buffers, which store newly collected data from a non-stationary data stream in an application. The changed probability distribution of the data stream, which the framework has identified, will be learned sequentially. The framework is called CLeaR (Continual Learning for Regression Tasks), where components can be flexibly customized for a specific application scenario. We design two sets of experiments to evaluate the CLeaR framework concerning fitting error (training), prediction error (test), and forgetting ratio. The first one is based on an artificial time series to explore how hyperparameters affect the CLeaR framework. The second one is designed with data collected from European wind farms to evaluate the CLeaR framework's performance in a real-world application. The experimental results demonstrate that the CLeaR framework can continually acquire knowledge in the data stream and improve the prediction accuracy. The article concludes with further research issues arising from requirements to extend the framework.

2020[ to top ]

Forecasting Power Grid States for Regional Energy Markets with Deep Neural Networks. He, Y.; Henze, J.; Sick, B. In International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{he2020forecasting, author = {{He}, Y. and {Henze}, J. and {Sick}, B.}, booktitle = {International Joint Conference on Neural Networks (IJCNN)}, keywords = {itegpub}, publisher = {IEEE}, title = {Forecasting Power Grid States for Regional Energy Markets with Deep Neural Networks}, year = 2020 }
%0 Conference Paper %1 he2020forecasting %A He, Y. %A Henze, J. %A Sick, B. %B International Joint Conference on Neural Networks (IJCNN) %D 2020 %I IEEE %R 10.1109/ijcnn48605.2020.9207536 %T Forecasting Power Grid States for Regional Energy Markets with Deep Neural Networks
Continuous Learning of Deep Neural Networks to Improve Forecasts for Regional Energy Markets. He, Yujiang; Henze, Janosch; Sick, Bernhard. In International Federation of Automatic Control (IFAC) World Congress, bll 12175–12182. Elsevier, 2020.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{he2020continuous, author = {He, Yujiang and Henze, Janosch and Sick, Bernhard}, booktitle = {{International Federation of Automatic Control (IFAC) World Congress}}, keywords = {itegpub}, pages = {12175–12182}, publisher = {Elsevier}, title = {Continuous Learning of Deep Neural Networks to Improve Forecasts for Regional Energy Markets}, year = 2020 }
%0 Conference Paper %1 he2020continuous %A He, Yujiang %A Henze, Janosch %A Sick, Bernhard %B International Federation of Automatic Control (IFAC) World Congress %D 2020 %I Elsevier %P 12175--12182 %R 10.1016/j.ifacol.2020.12.1017 %T Continuous Learning of Deep Neural Networks to Improve Forecasts for Regional Energy Markets

2017[ to top ]

TPM Framework: a Comprehensive Kit for Exploring Applications with Textile Pressure Mapping Matrix. Zhou, Bo; Cheng, Jingyuan; Mawandia, Ankur; He, Yujiang; Huang, Zhixin; Sundholm, Mathias; Yildrim, Muhammet; Cruz, Heber; Lukowicz, Paul. In International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies (UBICOMM). IARIA, 2017.
- [ BibTeX ]
- [ EndNote ]
@inproceedings{zhou2017tpm, author = {Zhou, Bo and Cheng, Jingyuan and Mawandia, Ankur and He, Yujiang and Huang, Zhixin and Sundholm, Mathias and Yildrim, Muhammet and Cruz, Heber and Lukowicz, Paul}, booktitle = {International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies (UBICOMM)}, keywords = {not_ies}, publisher = {IARIA}, title = {TPM Framework: a Comprehensive Kit for Exploring Applications with Textile Pressure Mapping Matrix}, year = 2017 }
%0 Conference Paper %1 zhou2017tpm %A Zhou, Bo %A Cheng, Jingyuan %A Mawandia, Ankur %A He, Yujiang %A Huang, Zhixin %A Sundholm, Mathias %A Yildrim, Muhammet %A Cruz, Heber %A Lukowicz, Paul %B International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies (UBICOMM) %D 2017 %I IARIA %T TPM Framework: a Comprehensive Kit for Exploring Applications with Textile Pressure Mapping Matrix