An IoT realization in an interdepartmental real time simulation lab for distribution system control and management studies

Modern electric distribution systems with emerging operation methods and advanced metering systems bring new challenges to the system analysis, control and management. Interdependency of cyber and physical layers and interoperability of various control and management strategies require wide and accurate test and analysis before field implementation. Real-time simulation is known as a precise and reliable method to support new system/device development from initial design to implementation. However, for the study of different application algorithms, considering the various expertise requirements, the interconnection of multiple development laboratories to a real-time simulation lab, which constitutes the core of an interdepartmental real-time simulation platform, is needed. This paper presents the implemented architecture of such an integrated lab, which serves real-time simulations to different application fields within electric distribution system domain. The architecture is an implementation of an Internet-of-Things to facilitate software in-the-loop (SIL) and hardware in-the-loop (HIL) tests. A demo of the proposed architecture is presented, applied to the testing of a fault location algorithm in a portion of a realistic distribution system model. The implemented platform is flexible to integrate different algorithms in a plug-and-play fashion through a designed communication interface.


INTRODUCTION
Nowadays, renewable energy sources as distributed generations are increasingly penetrating in power systems, more smart equipment and systems, like smart metering infrastructures, are being utilized.The infrastructures are also growing in terms of complexity and interdependency, and the communication system used for information flow is also being extensively developed as well as operational methods.Besides the challenges the new technologies bring to distribution systems, they provide opportunities for better control and management.The development of advanced control and management systems, either equipment or strategies, is not only beneficial, but also vital for power systems operation [1], [2].
There are a lot of different types of simulation tools for power systems modelling and planning, from large-scale high voltage transmission systems to low voltage distribution grids, and for a variety of applications, from transient analysis to long-term planning.Among different types of power system simulation tools, real-time simulation is becoming a highly reliable method, based on electromagnetic transient simulation, which provides a virtual environment of the systems, where new control strategies or technologies can be tested ex-ante before implementing them in the real world system, providing trustable real-like information on impacts and benefits [3].
Prior to test or simulate any, so-called, targets, the environment (e.g. a distribution system) behavior should be emulated through executing its virtual real-time model.This can support research and studies on power distribution systems with real-world situations, with flexibility in modelling different components, possibility of simulating the real-time behavior of prosumers, performing HIL laboratory validation of the new equipment such as smart metering devices, and test algorithms for smart grids management (e.g.fault detection) [4].
There are a lot of examples on the applications of real-time simulation to electrical systems including grids, power electronics and control systems.There are two main types of simulations: electromagnetic transient and phasor simulations; however, there are some examples on the combination of them.The most common commercial application of real-time simulation is in the prototyping stage of manufacturing a device or developing a system.This capability of real-time simulation provides the possibility of testing even when there are no physical prototypes (i.e.proceeding to test the device model before implementation through SIL) [5], [6], [7].Realtime simulation is also widely used before and after prototyping as design and test phases respectively.For example, developing an accurate model of an inductive superconducting fault current limiter [8], modelling of an industrial 200-hp dc drive [9], and in photovoltaic generation [10] and wind conversion systems [11], [12].
For protection system development and testing, closed-loop protection system tests are performed when a real-time simulator is combined with a suitable voltage and current amplification system [8].In control system development and testing, controllers for Flexible AC Transmission Systems (FACTS) devices being commonly tested, and generator excitation and stabilization controls are being studied.There are also examples in micro-grid control systems [13].The variety of field-specific applications requires different expertise and tools.In most of the current real-time simulation projects, the simulation architecture, including real-time simulator, workstation as virtual environment development node, the management or control algorithm development node, and the physical devices for HIL, is set up for a specific application in terms of design, prototyping, testing, etc.In other words, whenever an algorithm or new equipment requires a SIL or HIL test, a new appropriate dedicated set up should be made, and algorithm developers should closely involved in the set up process as well as carrying out real-time simulation.
The solution we propose is based on the concept of "simulation as a service": we create an interdepartmental laboratory platform to serve real-time simulation to plenty of application fields within electric energy domain.In this platform, the real-time simulation is the test and analysis core of the entire lab to which plenty of development nodes and simulation tools have access through UDP/IP or TCP/IP (namely Transmission Control Protocol and User Datagram Protocol) communication interfaces, mostly in a plug-and-play fashion.In addition, we propose an IoT (namely Internet-of-Things) oriented approach to retrieve relevant information from the real-time simulator and provide them over the Internet to other algorithms, control policies (not running on the realtime simulator) or more in general to cloud platforms [14], [15].Hence, every simulated component on the micro-grid can be considered as an IoT device.
In this paper, we present the proposed platform to facilitate testing and development of new control and management systems (both algorithms and devices) for distribution systems.Making such architecture for an interdepartmental real-time simulation lab also facilitates the combination of different methods for different objectives and applications (e.g.fault location, grid restoration and demand side management) and the study of the conflicts or interoperability of different control and management schemes simultaneously.Co-simulation of different algorithms provides the possibility of studying the different aspects of practical applications on the developed algorithms or devices.As an example, it would be possible to study the effect of communication failure, packet loss or latency on distribution network control and management algorithms which rely on communication systems.
The other advantage of such integrated platform is the softsharing of expertise from different labs or fields supporting interdepartmental cooperation in a research and development center.The proposed architecture provides a plug-and-play service system as if different algorithms or devices were interconnected in the real world.Therefore, in this framework, not all the teams developing control strategies, management algorithms, or other simulation modules need to be expert in real-time simulation procedure, or even with each other's algorithms and devices.They do not need to even share their developed products (e.g.control algorithms) while they can test and validate them on a real-time model of realistic distribution networks as if they were carrying out pilot trials.
The rest of the paper is organized as follows.Section II introduces the proposed architecture with the key components.Section III presents an implemented platform at Politecnico di Torino as a demo system to test fault location algorithms on a portion of the Turin distribution system.Finally, Section IV presents some concluding remarks.

II. FLEXIBLE INTEGRATED PLATFORM
The proposed flexible integrated platform is depicted in Fig. 1.The main components of this architecture include: i) real-time simulator, ii) communication interface, iii) application based nodes, iv) HIL facilities, and v) a supervisory and monitoring center.
The virtual model of the distribution system is built in a workstation connected to the real-time simulator.This connection can be either direct or through a local network.Control and management algorithms may be developed in different laboratories or departments.Measurement data obtained from the real-time simulation and command signals controlling the distribution system from the application nodes are exchanged through the communication interface.The components are shortly described in the following.

A. Real-time simulator
The real-time simulator, in this architecture, aims to reproduce the behavior of an electric distribution system modeled as a test bed for developed algorithms or devices, performing SIL or HIL.It is used for high-speed simulations, closed-loop testing of protection and control equipment, and generally all "What-if" analyses.Real-time simulation is actually simulating a system, which could realistically respond to the external stimuli, when the inputs/outputs of the simulation are synchronous with the real world.The requested measurements from the real-time model can be sent out through UDP-TCP/IP to be used by the control and management algorithm development nodes (SIL).In addition, the real-time simulator can communicate with real physical devices and system components (e.g. a micro-grid controller, protection automation systems, PV panels, wind turbines, etc.) through analog and digital I/O signals.
In the proposed architecture, the real time simulation lab is in charge of acquiring real/realistic network data, models the distribution system, and provides required measurements or signal connections with external applications.It provides simulation as a service.

B. Communication architecture
The communication architecture depicted in Fig. 2 exploits the publish/subscribe communication model [16], which allows the development of loosely-coupled event-based systems.Each component or application in the platform can: i) publish information and ii) subscribe to specific event notifications.In particular, it exploits the Message Queuing Telemetry (MQTT) protocol [17].In its core, the communication software adapter exploits TCP/UDP to enable the communication with the simulator in order to retrieve data from the real-time simulation engine.Such data are electrical measurements for a number of grid components, houses or appliances (e.g.aggregate power reading, voltage, current and frequency).It is worth noting that we chose UDP because of its light-weight nature which gives real-time application a good choice succeeding in real-time execution.Then, the communication software adapter parses the data into sequence of events before publishing them to an MQTT broker.The broker keeps track of all publications and subscriptions and takes care of sending new data to all subscribers.Hence, client application subscribes to the broker for receiving an information flow according to its needs.The communication software adapter works also as subscriber.it is able to receive commands from remote applications and push them to the real-time simulation engine.Exploiting this approach, each simulated grid component is seen by other software components (e.g.algorithms, control strategies and cloud platforms) as an IoT device able to send information and to receive, if possible, commands.

C. Application nodes
A variety of different applications in distribution systems can be considered within this framework.The applications could be: i) models and tools for network management, ii) control and management systems for individual users and micro-grids, iii) outage management strategies including fault and outage location algorithms, iv) demand side management, etc.
As an example, a load forecast application generates load profiles representing consumer behavior.A fault is simulated in real-time on the network.The fault locator, in a different application node (lab), uses the forecasted load data from the first application node, system topology and line data from its database, and measured voltage and currents from the real-time simulation as inputs to calculate the fault location.Thus, it would be possible to investigate the interoperability of the load forecasting and fault location algorithms and study the effect of load forecast uncertainties on fault location results.The application developers need neither to share their algorithms nor to model the grid in real time simulation.

D. Supervisory and monitoring systems
A monitoring center supervises the co-simulation performance and supports the analysis of interoperation of multiple applications and algorithms.In this real-time simulation framework, data and models from the real world can be fed into a real-time simulation by which a variety of tests and studies are performed to finally analyze the benefits gained from new technologies or control strategies.From the results of such analysis, the supervisor would detect the tentative conflicts and propose improvements of the schemes or devices under test in terms of a better interoperability.

E. Hardware in-the-loop
When the control and management algorithm is embedded in a controller device, or in any physical systems, its performance should be tested and validated in a real network.But the real network may not be available or the validation in the real network may not be feasible.Moreover, any preliminary validation in real field may not be safe.The solution is to test the real device/system in a simulated network which realistically responds to its environment.This substitution is called hardware in-the-loop.The network is modeled in real-time simulator, and the physical controllers or any kinds of real system elements are connected to the simulator through its I/O ports.Any scenario, even faulty ones, can then be safely implemented and tested.In a scenario where a simulation environment virtually exchanges power, in addition to signals, with the physical device (e.g. a PV panel), an amplifier is used to generate/absorb power and provides secondary level voltage and current (Power HIL).

III. IMPLEMENTATION -DEMO CASE
To realize the proposed architecture, the real-time simulation laboratory at Politecnico di Torino (Italy) is centered as the core lab to model an electricity distribution system.The test bed for the demo applications is a portion of the simplified medium voltage distribution grid in Turin, Italy.This portion (Fig. 3) consists of a primary substation with three MV-22 kV busbars, each of which is fed by a transformer characterized by voltage ratio of 220/22 kV.There are 5 MV lines starting from the HV/MV substation supplying 40 MV/LV transformers, and totally 49 branches.The real-time platform RT-LAB® developed by OPAL-RT® is used for simulating the distribution system.The grid model is developed in MatLab Simulink® (SimPowerSystem toolbox) with some Artemis library blocks from OPAL-RT.To execute the model on the real-time simulator, a fixed step time of 50 µs is selected for electromagnetic transient studies.The sampling time step to get snapshots of system status and measurements (reporting rate) is set as 250 µs.As an application, a fault location algorithm development node/lab is interconnected to the real-time simulation lab.The research team working on the algorithms needs neither to be real-time simulation expert nor to devote time to model the distribution grid.Different fault location methods are already proposed for electrical power distribution networks [18].Exploiting such architecture supports the researchers to validate their algorithm under test on a virtual model of a realistic network.
Almost 90% of all power system outages have their roots in distribution network [19], [20].Therefore, identification of the fault location has always been an interesting subject for utility engineers and researchers.Information on accurate fault location reduces the search space and required patrolling time.It can help utility personnel to perform fast service restoration and reconfigure the network to reduce outage time.The proposed methods for identification of the affected area of the network can be classified into outage detection and precise fault location algorithms.The first class, also known as outage mapping, employ various available data sources like customer outage calls and fault indicator signals to estimate the most likely affected area [21], [22].The second class uses the available measurements, in addition to network data (i.e.topology, line data and load data) to provide a more accurate estimation of the location of the fault that caused the resulting outage situation [23], [24], [25], [26], [27].
In this paper, the fault location method proposed in [26] is selected as an example to demonstrate the benefits of the proposed architecture for the development of a fault location algorithm.The algorithm uses only one measurement device in secondary substations in addition to the voltage and current provided by digital fault recorder installed at the head of the network main feeder.It starts from the first line section, and iteratively solves the equations describing fault steady state condition for all network line sections, one by one, to estimate all possible fault locations and the related fault resistance values.Then, for each possible fault location, the method applies the fault with the estimated fault resistance and selects the fault location with the minimum difference between the calculated and measured voltage sag at a secondary substation as the correct solution.
The algorithm is first developed and tested locally in the development lab.Then it is connected to the IoT interdepartmental architecture to validate its applicability before field tests.The proposed architecture provides measurements that are supposed to be retrieved from the real word to the fault location application.

A. Considerations about data transmission through the communication architecture
In this section we discuss our experience on the end-to-end message transmission process in our communication architecture (described in Section II.B).To conduct the experiment, we built an ICT network to communicate between real-time simulator, MQTT Message Broker, and Publish/Subscribe clients.
As mentioned before, the model was run with a step size of 50 µs for an electromagnetic transient simulation; however the reporting rate of the emulated smart meters was set as 250 µs.We choose one meter at the primary substation and one at a secondary to measure three-phase instantaneous values of voltages and currents.Therefore, there are 2*2*3 measured values (2 meters, 2 parameters, and 3 phases respectively) in each snapshot (sample).Considering a value indicating the time stamp of each snapshot, there are totally 13 values per sample.
Based on the fault location algorithm we apply, 32 samples per one cycle are required before fault occurrence and 32 samples of one cycle after fault occurrence and before protection reaction.The selected cycles should not be more than a few cycles far from the fault, however if a wide observation window (e.g. 1 second) around fault is retrieved, the abnormal/faulty values can be easily recognized.Therefore, we set the communication rate of the meter 1 second.
As the reporting rate is 250 µs, considering the communication rate as 1 second, there will be a 13*4000 matrix generated and reported every second.Hence, the MQTT software adapter receives data frames of about 154 bytes * 4000 each second and parses them into a JSON format before publishing to the Message Broker.Then the Message Broker is in charge of routing such messages to the subscribers.
For testing the fault location algorithm, we used a python script to subscribe to our model events and receive simulation data in (near-) real-time.We tested the setup with more than one subscriber.From these experiments, we proved that the proposed communication architecture can provide data to several departmental applications simultaneously.The subscribers get identical data from the time they are subscribed in.The JSON format we chose is simple to be parsed to client specific applications and can be utilized without domain knowledge about detailed internal working of the real-time simulator hardware/software.The architecture does not impose programming limitations on client's applications, thus can be implemented in various types of applications.

B. Fault location application test
Firstly, the fault location simulations have been performed using the MATLAB-Simulink software to model the network.The voltage and current phasors are calculated by Discrete Fourier Transform (DFT) using a data window of 64 samples per cycle.The simulated faults include single-line to ground (AG) faults with different fault impedances applied at the middle of all line sections of the longest feeder, providing 54 fault scenarios.Table I presents the results of the fault location algorithm for 16 of 54 performed simulation tests.The reported estimation errors are the distance between the reported fault location and the actual fault location in meters.It can be seen that as the fault resistance and the distance to the substation increases, the fault location error increases.However, in all cases the proposed method provides accurate results.After developing the algorithm, it is connected to the IoT architecture for the validation on the real time model.Following a fault occurrence on the distribution system running on the real-time simulator, the main feeder circuit breaker is opened due to the trip command of the over-current relay.
After breaker opening, the 1s window of the measured voltage and currents is sent through the communication architecture (described in Section II.B) to the fault location algorithm, The algorithm checks all line section and finds the faulted line section after some seconds.Fig. 4 shows the received voltage and current waveforms for an AG fault in the middle of 8 th line section in the third feeder.The algorithm is not aware of the fault location.It first calculates the voltage and current phasors and then analyses them to find the fault location.The fault is located at 8 th line section 454.52 m from the beginning of the line, having an estimation error of 50.98 m.
From fault locator viewpoint, real-time simulation acts with the same behaviour of the real network providing the measurements that are supposed to be provided in real word.Thanks to the developed architecture, the fault location algorithm can exploit the capabilities of RTS without any physical connections.Different sides of the architecture do not need to share their models or software's to validate them.Moreover, it is possible to perform different applications on the same model at the same time to study their co-operation and possible conflicts.For example, it would be possible to study the effect of the estimation errors in a load demand estimation algorithm on fault location results.As discussed, the fault location application receives the required measurements through the communication architecture and finds the location of distribution primary network fault.The visualization application is aimed to constantly analyze and prepare the demanded measurements for the supervisor.The supervisor monitors the network and the interoperability of multiple applications and algorithms (Fig. 5).While just two applications are performed in this case, any other application, on a different geographical location, can be added to this architecture exploiting the MQTT protocol in a plug-and-play fashion.

IV. CONCLUSION
The specific features of different application fields such as power systems, power electronics, and control systems, or different domains like EMT or Phasor simulations, bring feasibility challenges to build a universal generic simulation platform.Nevertheless, we designed, realized and tested a flexible integrated simulation architecture aiming at providing real-time simulation as a service.The purpose is to at least meet the main requirements of distribution systems control and management studies, carried out in different department laboratories.
Flexibility and scalability in terms of enabling integration of new research and development laboratory assets is considered in the framework development.This simulation platform would also support researchers and developers to propose new schemes or devices to the real system operators and regulators for the network design, improvement or reinforcement.
In this case, the new control strategies/algorithms and device prototypes can be validated in a virtual model of the real grid without devoting time to implement the network real-time model.
Labs from different departments can keep their developed algorithms confidential due to intellectual property rights or any other concerns, while testing them on realistic models for validation.
In this paper the proposed framework has been tested for the validation of a fault location algorithm with good results.

Fig. 1
Fig. 1 Architecture of the integrated interdepartmental real-time simulation laboratory

Fig. 3
Fig.3The portion of MV distribution grid for the demo simulation

Fig. 4
Fig. 4 Retrieved voltage and current waveforms for an AG fault

Fig. 5
Fig. 5 The implemented demo of the proposed architecture

Table I ESTIMATION
ERROR OF THE FAULT LOCATION ALGORITHM FOR AG FAULTS AT THE MIDDLE OF LINE SECTIONS IN METERS