A Study on the Improvement of Thermal Energy Efficiency for District Thermal Energy Consumer Facility based on Reinforcement Learning

This paper presents a study on the thermal efficiency improvement of the user equipment room in the district heating system based on reinforcement learning [1], and suggests a general method of constructing a learning network(DQN)[2] using deep Q learning[3], which is a reinforcement learning algorithm that does not specify a model. In addition, we introduce the big data platform system and the integrated heat management system for the energy field in the massive data processing from the IoT sensor installed in large number of thermal energy control facilities.


Introduction
Increasing productivity in the energy industry is a common problem that has been treated for a long time.To solve this problem, researches on energy efficiency improvement, production and maintenance cost reduction are continuing.In addition, the size and type of big data that can be digitized and collected recently can be further increased by the advancement of computing technology due to the rapid development of semiconductor technology.Accordingly, the collection, storage, processing, and analysis of big data in energy, medical, and bio fields as well as multimedia big data, which are easy to access and utilize in the past, such as audio, image, and video, are actively performed.Big data on the heat-related thermal energy generated by thousands of control facilities supplied by the Korea District Heating Corporation are not exceptions.In the case of district heating, the problem of improving the thermal efficiency of the facilities in the heat user control equipment is the same and it is becoming an important research goal as well.
In this paper, we first use Hadoop [4] based Big Data Platform, which is an open source framework used to reflect the characteristics of thermal energy big data which is a classification of energy big data.The Big Data System, which is optimized to store, manage, and process the big data collected in real time from individual household, heat control equipment room, heat production facility and surrounding environment for district heating of the building complex.And how to utilize big data collected from the demonstration site, which is a form of thermal user machine room.The second is a thermal demand management system which is a web service based user interface that is statistically analyzed on the Big Data Platform and has the ability to control the equipment by changing the operating parameters according to the situation so that the equipment condition of each machine room is effectively monitored and the thermal efficiency is increased, which can be visualized and intuitively presented, is introduced.These functions are used by Web service-based user-interface (UI) to predict the outcome of a diagnosis and the longer the equipment should also introduce about visualization and integrated heat demand which could indicate an intuitive control system.
Finally, based on the reinforcement learning, the thermal efficiency of the user thermal control equipment in the district heating is one example of the study that attempts to improve the thermal efficiency.We introduce the configuration method based on the learning network(DQN) using the deep Q learning algorithm, which does not specify the model and applied to the heat user equipment.

Motivation and background
Improving the service quality of district thermal users and improving the energy efficiency of users and operators is an important research goal that can contribute to energy saving at national level.The district heating system contributes to the energy saving and the improvement of the environment in comparison with the other heating methods.To this end, the domestic district heating companies have been conducting research and development and improvement of the district heat production equipment and heat transportation equipment.While the operations are considered to be at the level of advanced countries, the heating user area is managed by the user management facility, which is a non-specialist, and the efficiency of the district heating system is reduced due to inefficient facility management and operation.In more detail, large number of households belong to the individual complexes using the district heating, and the central control room of the complex receives the heating energy from the local heat production facility through the pipeline first.then secondary supply to each household.In this process, heat loss can be reduced and heat transfer can be efficiently performed when various variables such as heat demand and outdoor environment of the individual generation, which is the subject of heating, are taken into consideration.However, since the existing control room facilities are manually controlled arbitrarily by the non-specialist personnel of the majorities of the individual complexes, problems such as a decrease in thermal efficiency and waste of energy and costs have been caused chronically.
To maximize the efficiency of heat energy among these problems, we exploit the Big Data System [5], which is prior results of study, how to provide useful information to users, and the processing of useful information for users to acquire operating information and user facilities from large amount of data flowing in real-time from 1.3 million district heating users and 5,800 heat user machine rooms supplied by Korea District Heating Corporation is processed to increase energy efficiency of user facilities and the heat demand integrated management system [6] based on the big data platform for district heating, which monitors the use of the heat.Using the above two studies, reinforcement learning method leads to a more active study of thermal energy operational control, when performing the application and operation is expected to contribute more actively to improve thermal efficiency.

Improvement of Thermal Efficiency of User Control Facility Based on Reinforcement Learning
In this chapter, we first consider the big data system that the thermal energy big data and the integrated heat management system for the district heating, and then the learning network (DQN) using deep Q learning.The introduction of the thermal efficiency improvement method is introduced.

Introduction of big data system to accommodate energy big data
The server cluster of Energy Big Data System utilizes ten nodes as shown in Figure 1 (a), and has two manage nodes, six data nodes, two edge nodes, (Edge Node).In a server cluster that is virtually bound to a plurality of server computers to operate as one and the same system, the nodes refer to each unit computer.In general, a distributed server system is composed of a master node that issues a command to process a given task and controls another node, and a slave node that processes the received control signal and commands.As shown in Figure 1 (b), the server cluster for the big data system, which is designed based on Hadoop, which is an open source framework, has a manager node that manages the computing resources and status of each node, In addition to the data nodes storing and processing data, edge nodes were included.The edge node acts as an interface between the big data system and the external network.In the energy big data system, it is necessary to execute precise calculation of various data continuously collected on a large scale and instantaneous generation of a control signal based on the result.Therefore, in the in-memory computing environment, Spark [7], which is supported as a state, has been replaced with Maple deuce.As shown in Figure 2 (a), the field data collected using the IoT module are loaded into the big data system through the data collection process and the real-time distributed streaming step and loaded into the database [8] according to the designed schema.External data can also be accepted through the data processing in Figure 2 (b).Also, regardless of the source of the data, it is possible to handle both the fixed and irregular data.We first looked at a Hadoop-based Big Data System that accepts thermal energy Big Data, collected in real time.This enables big data processing and analysis that matches the characteristics of thermal energy big data that has real-time and repeatability, thus securing vast amount of field data in the field and stable high-speed computing.As a result, Monitoring, fault diagnosis of equipment, prediction, efficient operation recommendation, and the like can be implemented at high speed and stably.

Introduction of Integrated Demand Management System for District Heating
The heat demand integrated management system is based on the Big Data platform utilizing Hadoop, an open source framework, and utilizes big data collected in real time from individual households, machinery rooms, heat production facilities and surrounding environment for district heating of the building complex.do.The Big Data is statistically analyzed on the Big Data Platform [9] [10], allowing the facility status of each thermal control facility to be effectively monitored and controlling the plant by changing the operating parameters according to the situation to increase thermal efficiency.This function is visualized intuitively with the fault diagnosis and the anomaly prediction result of the equipment in the integrated thermal management system which provides the user interface (UI) based on the web service of Figure 3.As a result, the local heat production facility manager can check the status of the control equipments in the facilities, the thermal efficiency analysis result, the failure and the abnormality of the area in whole or in detail, and the facility operation manager of the individual complex buildings can check the situation of the equipments in real time .Using statistical analysis service, the operation recommendation simulation as shown in Figure 4 (a), in which the manager checks the current thermal efficiency and applies the improvement algorithm that utilizes the big data analysis, in each control facility.API management, membership management, system management, and social service functions are also provided in accordance with the user classification.API management provides selected standard APIs such as 3'rd party energy companies to develop another service by utilizing big data platform and heat demand integrated management system.The social service in Figure 4

Conclusions
In this paper, firstly, it is introduced the thermal energy management system, which is a Web service-based user interface (UI) that allows the user to change the operating parameters to suit the situation and to control the equipment to increase the thermal efficiency.The statistical analysis is also performed statistically on Big Data System and Big Data Platform, which provide fast and reliable in the collection, storage and processing of large amounts of thermal energy related big data collected at the demonstration site.
A large amount of big data information consisting of atypical heat energy related data sets is presented by using the above Big Data System and a reinforcement learning based machine learning example made up of learning network (DQN).A method for improving the thermal efficiency of a thermal user control equipment facility based on the approximation of a Q function for improving the thermal efficiency of a thermal user control equipment facility and the reinforcement learning for enabling an efficient optimum operation by using the controller as a real input.
Finally, considering the realistic difficulties of the research purpose direct control on the thermal equipment control facilities of the building or apartments and the characteristics of the heating heat energy with the slow response of the home temperature response, the efficiency applied by the learning network (DQN).It is also needed to continue to study the improvement effect using simulator or actual testbed, available to modify control parameter without any user complain.

Figure 1 .
Figure 1.(a) Thermal Energy Big Data Server Cluster Configuration (b) Big data Server Framework Diagram

Figure 2 .
Figure 2. (a) Thermal Energy Big Data Collection Structure (b) Local data processing diagram for big data systems

Figure 3 .
Figure 3. Districted Thermal Demand Integrated Management System UI Main Page.

Figure 4 .Figure 5 .
Figure 4. (a) Operation advice and Fault diagnosis (b) Social Service Board