IoT-Enabled Real-Time Management of Smart Grids With Demand Response Aggregators

Integration of widely distributed small-scale renewable energy sources like rooftop photovoltaic panels and emerging loads like plug-in electric vehicles would cause more volatility in total net demand of distribution networks. Utility-owned storage units and control devices like tap changers and capacitors may not be sufficient to manage the system in real-time. Exploitation of available flexibility in demand side through aggregators is a new measure that distribution system operators are interested in. In this article, we present a developed real-time management schema based on Internet of things solutions which facilitate interactions between system operators and aggregators for ancillary services like power balance at primary substation or voltage regulation at secondary substations. Two algorithms for power balance and voltage regulation are developed based on modified optimal power flow and voltage sensitivity matrix, respectively. To demonstrate the applicability of the schema, we set-up a real-time simulation-based test bed and realized the performance of this approach in a real-like environment using real data of a network with residential buildings.


I. INTRODUCTION
M ODERN power systems are less predictable compared to traditional networks. This is mainly due to growing integration of intermittent renewable energy sources such as wind and photovoltaic (PV) as well as emerging uncertain loads like electric vehicles.
The load centers are not anymore distant from the generation locations. Distributed generation brings the electricity production close to the consumption points. Therefore, distribution networks are not passive anymore. This is actually a transition from vertically integrated and centralized systems to decentralized schemes.
This transition needs to be planned with specific tools that are able: 1) to estimate production of renewable energy sources Manuscript  in time; 2) to exploit flexibility of demand; 3) to assess the capabilities and requirements of distribution networks; 4) to estimate the impact on energy markets [1], [2]. Distributed energy is the utilization of smaller power generation and storage systems used to residential, commercial and industrial buildings. Most distributed energy generation systems take advantage of renewable energy sources such as solar and wind. In this new scheme, efficiency, flexibility, reliability, and cost savings become fundamental keywords and a new architecture that manages this system is necessary. The smart grid is a reinvention of how energy is transmitted, distributed, and measured. It is becoming the new standard for utilities and consumers and represents the merging of multiple technologies into a system that provides reliable and cost-effective energy. In such future smart grids, new actors will participate in the fast-evolving and distributed energy marketplaces, such as prosumers and aggregators. Besides the economic motivations of introducing these new actors, there are technical demands for involving such players in system control and management. For example, as shown in Fig. 1, the new algorithms offered by the aggregator need to constantly exchange information with the energy management system at prosumer premises to guarantee efficient services, such as load scheduling, based on user's preferences.
There are different approaches reported in the literature or applied in power industry that are used for planning and operation of smart grids in presence of renewable energy sources [3]. They apply different optimization techniques and demand side management methods to secure the electricity supply with acceptable reliability and power quality [4]- [6]. The operation performance of these approaches technically depends on three main factors as follows: 1) accurate and efficient forecast of prosumers' behavior; 2) analysis tools and algorithms; 3) well established and deployed distributed systems. High penetration of intermittent renewable energy sources on one side, and growing utilization of new types of loads like electric vehicles and storage systems on the other side, would make challenges for the three above mentioned factors because of the following: 1) prediction of such uncertain behaviors of prosumers is more difficult; 2) more advanced algorithms are required for system analysis; 3) conventional centralized or decentralized systems lack sufficient hardware and software to manage plenty of emerging actors. A solution to tackle these challenges is to compensate any inadequacy or inefficiency of the existing planning and operation systems with real-time management of smart grids thanks to widely distributed smart meters [7]. This solution is attracting attention to leveraging the off-the-shelf devices and systems, including intelligent electronic devices, smart meters, advanced communication networks, etc.
In this article, we introduce our proposed system for real-time management of smart grids aiming at covering any inadequacy or inefficiency of conventional operation methods. Our scheme is a low-cost practical add-on to the existing power networks which minimizes the operational costs, making it ready to be applied in real-world smart grids. The scheme uses Internet-of-things (IoT) approaches for data communication. The off-the-shelf components used in our system includes energy aggregators, distributed smart meters, and day-ahead or intraday forecasting tools. Using IoT technologies to provide smart grid support and services, such as aggregating distributed energy resources, demand response, distribution network management, and participation in the electricity marketplace, is a well established topic in literature [8]- [13]. Hence, the novelty of the proposed solution is its application in novel smart-grids. We developed a communication adapter to interconnect these components with a substation level calculation and management center owned by distribution system operators. We also developed this calculation center as a Python-based flexible container to accommodate new algorithms which intend to support real-time management of smart grids. Two algorithms, as examples of applications, are also developed: 1) an optimal power flow (OPF) algorithm to mitigate the unexpected power disturbances at the primary substation; and 2) a centralized voltage regulation system to correct real-time voltage deviations at secondary substations.
Reviewing literature, there are a wide range of solutions to manage smart distribution grids with renewable energy integration. In some works, like [14]- [16], distribution system operator (DSO) uses storage units to inject or absorb power in response to variations. However, installation and operation of storage units are quite costly for grid operators. In many cases, accurate load forecasting methods are proposed to capture the dynamic behavior of demand in real-time and tune the prosumers accordingly [17]- [20]. The new challenge in these cases is the controllability of renewable energy sources (RES). In most cases, no better options than curtailment could be found [21]- [23]. In some cases, the load is shifted or the peak is shaved by load shedding [24]- [26]. These measures would reduce customer comfort, and result in extra costs due to violation of customer contract.
The main source of providing additional services in our scheme is the demand side flexibility. As an advanced and efficient solution, demand side management is proposed to support accommodation of more RES in distribution networks [27]- [29]. Different demand response (DR) methods are in place to provide flexibility. However, before sending DR signals to individual loads, the amount of change should be defined as well as the location of required adjustments. This is challenging when fluctuations of the demand curve are high due to volatility of loads, especially residential buildings, and the unpredictable variation of RES power injection.
In order to make a sound interaction between prosumers and the system operator, intermediate actors are needed to coordinate the contributions of prosumers in a smart fashion. The smartness comes from well deployment of smart devices such as smart meters and bidirectional actuators, as well as advanced models and methods aiming at minimizing energy costs. In this context, aggregators are great sources to manage small scale prosumers and develop DR schemes. In most of literature, the proposed DR methods are designed in such a way that the DSO and the aggregators can be comanaged by one entity [30]- [33]. In practice, each party has its own objectives and management strategies; for instance, DSO tries to ensure the continuity of supply with high reliability and power quality, while aggregators aim to minimize the customer costs (e.g., reducing the bills) withstanding some level of comfort. Our approach decouples the aggregator and the DSO as two different cooperating actors, enabling a smooth integration of different models with independent objectives and applications. For demonstration purposes, the focus of our developed application algorithms is on DSO side. We discuss the methodology of the application implementation and the information to be exchanged between aggregators and DSO. It should be noted that with DSO we do not refer to the distribution management system located distant from substations, rather we refer to distributed agents hosted in substations.
This article extends our previous work [34] addressing the following new considerations: 1) demonstrating the capability of the platform to handle multiple independent aggregators; 2) developing and demonstrating more applications including voltage regulation; 3) stressing the proposed algorithms with scenarios where the local flexibility is not sufficient by considering multiple building units or generators connected to the same medium voltage (MV) substation while being managed by different aggregators; and iv) discussing the capability of this platform to manage dispatchable generators as well as flexible loads.
In this article, we first introduce our proposed real-time management schema and its components and features. Then, the real-time management algorithms embedded in this schema are discussed. To better represent the functionalities of this schema, an experimental setup is introduced, where we explain the elements of the proposed management system in detail. To demonstrate the performance of this setup, some lab experimental results are presented and analyzed. The article is concluded with some remarks and future extension suggestions.

II. PROPOSED REAL-TIME MANAGEMENT SCHEMA
In this section, we present the proposed real-time management schema for real-time deviation identification and grid regulation with the presence of demand flexibility offered by aggregators. Fig. 2 represents the players and the high level interaction of the proposed scheme aiming at supporting real-time management of distribution systems by appropriate demand side management. Orange connection arrows represent data communication flows among the various actors, while green communication arrows represent power flows. Aggregators are emerging players in the new smart grid paradigm. Their role is to aggregate a set of prosumers, consumers and/or distributed generation sources in order to exploit and take advantage of demand flexibility in the energy marketplace. By estimating the power demand and generation, aggregators try to buy energy at the lowest prices in day-ahead market. Moreover, by exploiting prosumers and consumers demand flexibility, aggregators can offer to DSO ancillary services such as power and voltage regulation. DSO can request such services to aggregators if they see that in their controlled grids a voltage or a power mismatch occurs. If so, they compute the amount of flexibility needed in each node by exploiting algorithms for OPF, power balance, or voltage variation control.
Such schema can take advantage of a distributed intelligence approach where "intelligence systems are based on the use of cooperative agents, organized in hardware or software components, that independently handle specialized tasks and cooperate together to achieve system-level goals and a high degree of flexibility" [35]. The achievement of such distributed systems is obtained by exploiting communication paradigms peculiar to IoT systems, such as publish/subscribe [36] and request/response. Thanks to these paradigms, the intelligence of novel services can be distributed across different software entities that can be executed on various servers or on dedicated hardware (e.g., digital real-time simulators).
Most of the algorithms run in aggregator premises, where data of loads including their coordinates, contractual power, details of devices, information of smart meters, etc., may exist. Therefore, as shown in Fig. 2, flexibility of prosumers can be defined by the aggregators. This includes information about which node of the network has dispatchable loads and the amount of demand flexibility to be increased or decreased. This is computed by the flexibility estimation which needs as input the information provided by both load forecast and distributed generation forecast modules. In particular, load forecast component bases its computation on either previously forecast values or real-time measurements from smart meters (i.e., consumer and prosumers modules in Fig. 2). While, distributed generation forecast module of RES, for instance, PV panel production, can take advantage from machine learning models (e.g., [37], [38]) or simulators (e.g., [39]) to perform realistic estimation on energy production in short and/or mid-term, even considering historical real-world trends as input (i.e., distributed generation component). Finally, the output of these three modules together with the deviation value, such as the imbalance power at primary substation or a set of over/under voltage measurements in some secondary substations (i.e., power grid module in Fig. 2), are given as input to the algorithms in DSO premises.
The DSO would run the algorithms to minimize the deviation from the scheduled or planned values. The performance of the method against the prediction errors in load demand or renewable generation depends on the amount of flexibility offered by the aggregators. If the flexibility is sufficient, all the deviations could be recovered and the parameters, including power and voltage, would fall within the acceptable ranges. If they are not sufficient, the DSO still covers any power imbalance or voltage violation, but it uses other resources which implies higher costs, including upstream grid, or utility-owned bulk storage.
In Section III, we discuss two algorithms developed for realtime power flow control at primary substation and real-time voltage regulation at secondary substations, respectively. The algorithms are developed to be immediately embedded in the real-world systems. This is a crucial issue as real-world systems cannot be used to test new algorithms under different scenarios and configurations. For this purpose, our solution integrates digital real-time simulators (DRTS) to reproduce the behavior of a real-world system in interaction with other components and algorithms, and ensure sufficient ex-ante analysis and verification. Fig. 3 details information and data flows among the different actors in our framework. DR reference signals for ancillary services are defined by the DSO and implemented by both aggregators and prosumers. The focus of this article is on defining such reference signals. Aggregators provide, in advance or periodically, a list of reserve options and they receive in real-time the required adjustments. For the two applications presented in this work, DSO requires power flow measurements of the primary substations, voltage measurements of the secondary substations, an estimation of both load and generation forecast in short-term, and the scheduled power exchange with upstream network based on day-ahead or intraday market data.
It is worth noting that, IoT technologies are recognized in literature to be key players to provide smart grid support and services [8]- [13]. Such IoT technologies are part of a more complex distributed system, so-called advanced metering infrastructure (AMI). Among its main tasks, AMI enables a secure and reliable bidirectional communication with its individual components, either hardware or software (i.e., IoT devices and software services, respectively), adopting the required security level, protocols, and standards that can vary depending on different smart grid requirements. Often AMIs do not provide advanced services for demand response nor real-time simulation features to evaluate new algorithms and services. In this view, our solution is an add-on to existing power networks and their AMIs. Hence, it builds upon already existing protocols and redundancies, also in terms of cybersecurity, using IoT approaches for data communication and off-the-shelf components. One of core features of our solution is the "calculation center," which can be seen as a flexible container where different algorithms for real-time management of smart grids can be deployed in a plug-and-play fashion. Thus, our solution is agnostic to the specific communication technology and to a specific intelligence of a certain algorithm. This makes our solution complementary to AMI making its deployment flexible and easy in different smart grid contexts with different characteristics, requirements, and needs. Moreover, our solution provides specific components to allow real-time simulation by integrating digital real-time simulators to realistically assess the new algorithms and services before their deployment in the real-world.

III. REAL-TIME MANAGEMENT ALGORITHMS
In this section, the two algorithms developed for our demonstration are introduced: primary substation power balance and secondary substation voltage regulation. The additional implementations to manage interactions and data exchanges between DSO and aggregators are also discussed. The algorithms are developed in Python, and the grid model to test them is implemented using Opal-RT digital real-time simulator.

A. Primary Substation Power Balance Algorithm
The primary substation power balance algorithm aims to minimize the unscheduled power exchange between local distribution network and upstream transmission grid. It periodically receives the real-time power measurements ("real-time active power") from primary substation ["point of connection (PoC)"], compares it with the scheduled/forecast power exchange at PoC ("forecast active power"), to identify the power imbalance in case the difference is higher than some thresholds (e.g., 10%). Once the power imbalance is identified, an OPF algorithm is called to define the amount of power adjustments in flexible nodes.
In the OPF, some parasitic generators are modeled to represent flexibility of prosumers as dispatchable loads. The "difference" or power imbalance is dispatched among all these parasitic generators by minimizing the objective function. The capacity of these parasitic generators is set equal to the flexibility offered by the aggregators. If aggregators introduce different costs for the resources at different nodes, OPF would minimize the cost; otherwise, the power loss in the grid will be minimized. There are two other generators modeled in this OPF: one represents the scheduled/forecast power exchange at PoC, the other one represents the upstream transmission or subtransmission grid. Both of these generators are modeled on PoC node. The former has minimum and maximum power set equal to the scheduled power, while the latter has maximum capacity equal to the transformer's rated power. The cost function of the latter generator is defined based on the high cost of supplying imbalance power (the "difference") in real-time. This would make the algorithm choose the flexible loads as much as possible first as they are less costly.
The set points and parameters are periodically configured in the OPF algorithm. The pseudo-code of the high level algorithms for imbalance power management are shown in Fig. 4. Algorithm 1.1 defines whether thresholds are violated or not. Algorithm 1.2 and Algorithm 1.3 are triggered when such thresholds are exceeded. In particular, Algorithm 1.2 receives a set of messages from the aggregators reporting the amount of flexibility for both active and reactive power providing minimum and maximum power deviations in a specific node at a certain instant in time. It also reports the costs for this flexibility. Following this approach, aggregators are free to model and manage their generators and loads as they prefer, exposing only relevant information. It is worth noting that an aggregator can manage a variable number of nodes in the grid making Algorithm 1.2 flexible and scalable in managing different power grid configurations. Once all the routines in Algorithm 1.2 are completed, Algorithm 1.3 runs, performing the optimization according to the power grid topology. The main objectives of the OPF is twofold: 1) minimizing the overall costs; and 2) reducing the gap between scheduled and actual power injected at PoC. Finally, the resulting adjustments are sent back to the aggregators. In detail, the optimization problem is formulated as follows: where N s is the set of flexible or dispatchable loads to be adjusted; P gi and Q gi are real and reactive power of generators including those modeled on primary substation and distributed generators; ΔP di and ΔQ di are real and reactive power adjustments of flexible loads. The equality constraints are While, the inequality constraints are where i ∈ N , which is the set of all nodes. P di and Q di are real and reactive fixed loads, V i and θ i are node voltage magnitude and angle. G i and B i are elements of bus admittance matrix. C i is our defined fixed power factor for adjusting reactive power along with real power. At this level, information exchanged among aggregators and DSO excludes reactive power for residential prosumers.

B. Secondary Substation Voltage Regulation Algorithm
As previously discussed, the day-ahead schedule includes estimation errors of distributed generation (DG) and new loads like electric vehicles. Therefore, scheduled voltage regulation using on-load tap changer transformers (OLTC), distributed capacitors, etc., may not be sufficient to avoid voltage violations. This is also challenging when OLTC supplies several feeders with different DG penetration levels. The increase of voltage level by OLTC to improve voltage profile in a feeder with more loads may result in over-voltage in the other feeder with high penetration of DGs. Therefore, in addition to the existing operational voltage regulation measures, real-time voltage regulation is needed through demand side management.
In our schema, periodical voltage measurements at secondary substations are retrieved by the DSO from one side, and the available flexibility/reserve offered by the aggregators are received from the other side. For a real-time voltage regulation, the voltage sensitivity matrix (S) is formed to determine the amount of power and the location (i.e., the node of the grid) to make adjustments to improve the voltage profile. The sensitivity matrix is the inverse of the Jacobian matrix (J ) where using Newton-Raphson method in nonlinear power flow. The linearized steady state system power voltage equations are given by the following: where ΔP is the incremental change of real power; ΔQ is the incremental change of reactive power injection; Δθ is the incremental change of voltage angle; and ΔV is the incremental change of voltage magnitude ΔU (t) is the transpose of a vector matrix formed by a set of voltage deviations. This vector includes all the nodes, regardless to the amount of voltage deviations. The amount of voltage deviation, ΔU i (t), for node i, is obtained based on the following logic: where V min and V max define the acceptable thresholds of voltage regulation.
It should be noted that the algorithm output is a set of real and reactive power adjustments in all nodes, if necessary. However, command messages from DSO back to the aggregators only include the real power adjustment as the reactive power would be consequently changed based on the power factor. For prosumers with substantial generation (including DGs), the reactive power flexibility of DG's inverters may be offered by the corresponding aggregators. In this case, the reactive power adjustment would be communicated. The other limitation of voltage regulation through demand response is defined by the level of offered flexibility and willingness to contribute. Thus, only the adjustments on the nodes listed by the aggregators will be requested and they should fall within the capacity as well. DSO would seek other measures if this is not sufficient.

IV. EXPERIMENTAL SETUP
To demonstrate the implementation and performance of our schema, a real-time co-simulation platform is configured. We believe this platform is the closest one to the real-world system, and all components of this setup can easily be replaced by the real physical systems in plug-and-play fashion. Fig. 5 shows the high level infrastructure of our laboratory setup. It consists of the following: 1) a digital real-time simulator representing the real network; 2) the aggregator; 3) the distribution management system (DMS) of DSO; 4) real-time data of load profiles collected either by smart meters or load forecast tools; 5) distributed generation real-time estimation; and finally 6) a communication adapter to glue all aforementioned components. In this section, we briefly introduce these components and the way they are coupled and communicating with each other.

A. Digital Real-Time Simulator
In order to reproduce the behavior of the grid, digital real-time simulators are integrated into the platform. They are capable of capturing the fast dynamic behavior of system in electromagnetic analysis (EMT) as well as slow transients of the system in phasor domain. They are also capable of being used in cosimulation setups so-called software-or hardware-in-the-loop very efficiently. The communication capability of these simulators is very wide and different real-world configurations could be made using them. For our specific platform, which is based on IoT paradigms, DRTS can bidirectionally communicate with other entities (either hardware or software) over the Internet through TCP/IP or UDP/IP.
In our laboratory setup, we used an OPAL-RT DRTS to run the model of the grid which is implemented in MATLAB Simulink. The generation and load trajectories are based on some scenarios which are unknown to the DMS OPF algorithm. They are fed into the grid model running on real-time hardware. The data retrieved from the grid includes the power exchange measurement at PoC between transmission and distribution networks for power balance function, and the voltage measurements from secondary substations for the voltage regulation function.

B. Aggregator
To represent the aggregator, a separate PC with a Python script running is set up, which is in charge of controlling the end-users including prosumers and storage units from one side, and communicating with market entities as well as distribution system operators from the other side.
For our particular practice, the aggregator is communicating data and signals related to demand response. This means the aggregator periodically (i.e., in real-time), intraday, or day-ahead sends information about prosumers' flexibility to the DMS algorithms. The data include node ID (i.e., location) of the flexible loads, the amount of flexibility in power unit of measurement, and the price or cost of the reserves. DMS, instead, returns the exact amount of power adjustment along with the corresponding node IDs to the aggregator. It is out of scope of this article to discuss how the aggregator, then, control the loads and appliances to perform demand response. The aggregators seek to achieve this demand set-point defined by DMS by providing monetary incentives to household users to modify their demand pattern, or through direct load control. The objective of each aggregator is to maximize its own net profit, namely the income received from the operator minus the compensation it provides to end-users.

C. Agent of Distribution Management System
The DMS we refer to in our setup is actually a part of DSO DMS that includes the proposed solution as a set of new functions. It is designed in a way to accommodate more DMS functions and algorithms as the platform is scalable and flexible to integrate more modules.
The two algorithms detailed in Section III are executed in this machine. It runs Python script of the modified OPF, power flow calculations, voltage regulation algorithm, and other wrapping and processing functions. DMS retrieves flexibility data from the aggregators, the measurement of active power flow at primary substations and voltage at secondary substations, and the forecast values of loads and generation in the network. It runs the OPF to minimize the overall cost of load dispatching and network losses subject to assuring the power exchange setpoint from the scheduled profile. It also periodically monitors voltage profile of the feeders and in case the voltage regulation through DR is requested, it provides reference signals to the aggregators for adjustments. It should be noted that although several functions and algorithms are embedded in DMS, but not necessarily all of them are simultaneously in operation. These are additional ancillary services subject to agreements between aggregators and system operators. Therefore, the algorithms are developed to be functional independent from each other.

D. Load Profile
DMS needs to be updated with the system status including load and generation data. In distribution systems, it is difficult to update DMS with the real-time load and generation data as there are plenty of customers across the network. In order to provide data for load flow, OPF or other analysis and calculations in DMS, we integrated a module that creates load profiles. This could be generated either from widely distributed smart meters in the network, which periodically report consumption data, or from load forecasting tools. In both cases, historic data of the loads are used. The algorithms are always running based on the data at time t 1 to achieve the set-point at time t 1 + dt, where dt is the time-step. In our DR practice, 15 min intervals are considered as time-step.

E. Distributed Power Generator
Similar to load data, power flow calculation requires the generation data. There is a separate server which runs a Pythonbased code to estimate the generation values and responds to any requests from DMS to provide the power values. In our demonstration test case, we used our "PV Simulator" which was developed to estimate the power output of roof-top PV panels in urban areas [39] as our case study focuses on residential districts.

F. Communication Adapter
The communication adapter is in charge of enabling data exchange among the DRTS (i.e., OPAL-RT in our setup) and other modules in the proposed infrastructure. It implements two communication paradigms: 1) publish/subscribe [36], based on message queuing telemetry transport (MQTT) protocol [40]; and 2) request/response based on HTTP-REST [41] (HTTP stands for hypertext transfer protocol and REST for representational state transfer).
In its core, the communication adapter exploits the TCP/UDP module to allow a bidirectional communication with the DRTS. It is worth noting that the communication adapter also allows the integration of the DRTS with other smart metering infrastructure, where each smart meter is an IoT device. Hence, data coming from such devices can be used to simulate and test innovative control strategies with (near) real-time data from the grid. Conversely, each simulated grid component is also seen by other modules as an IoT device able to send information and to receive commands. Our simulated devices could be also replaced by real smart meters integrated into the infrastructure.

G. Consideration on Cybersecurity
Publish/subscribe and request/response paradigms, and their implementation MQTT and HTTP-REST, respectively, are widely used in developing IoT-based distributed platforms, as an AMI is. Thanks to these paradigms, the intelligence of novel services can be distributed across different software entities that can be executed on various servers or on dedicated hardware (i.e., digital real-time simulators in our solution). On the one hand, the publish/subscribe communication model, and consequently MQTT, allows the development of loosely coupled event-based (or data-driven) systems. Each module can publish data and this data can be independently received by a number of subscribers. Intrinsically, this feature guarantees redundancy of components. On the other hand, the most common protocol implementing the request/response paradigms is HTTP. It is a software architectural style to design and develop distributed software systems exploiting the Internet and the World Wide Web to exchange information and cooperate. Among all, REST emphasizes the scalability of interactions between components and the creation of distributed layered software platforms to enforce cybersecurity, increase reliability, and provide redundancy of components (allowing load balancing techniques) [42]. REST and HTTP are two milestones of the World Wide Web, thus, they are typically coupled together. Finally, both MQTT and HTTP can optionally exploit transport layer security (TLS) [43], which is a protocol to enable secure communication channels with cryptography over computer networks by providing. TLS guarantees authentication, confidentiality, and data integrity among communicating counterparts. When TLS is enabled, we refer to these protocols as MQTT-secure and HTTPS.
In conclusion, our solution implements both communication paradigms and protocols and, consequently, inherits these security features. As previously discussed, our solution aims at being a low-cost add-on for AMI, delegating to AMI the different security levels needed by a specific smart grid, or a portion of it. This makes our solution flexible and eases its deployment. It is also worth noting that dealing with more advanced cybersecurity aspects is out of the scope for this work.

V. EXPERIMENTAL RESULTS
Our case study is based on an urban district area located in Northern Italy with about 2200 residential buildings. The MV grid in the district (see Fig. 6) consists of a primary substation with three 22-kV bus-bars, each of which is fed by a transformer characterized by voltage ratio of 220 kV/22 kV. Forty-three substations are supplying loads (mainly residential buildings). These substations are equipped with MV/LV transformers characterized by voltage ratio of 22 kV/400 V, and a nominal power of 400 kVA or 250 kVA. The total length of MV lines, mostly constituted by underground cables, is around 39 km. The loads are more residential prosumers with rooftop PV panels as DGs.
In this study, we demonstrate the applicability and performance of our IoT-based real-time management schema to support distribution system operation with DR services. There are two applications: primary substation power balance and secondary substations voltage regulation. These applications are tested independently with different scenarios.

A. Primary Substation Power Balance Test
A cloudy day scenario is considered when PV production is less predictable and also lower than a clear sky day. Fig. 7 shows power flow data at the primary substation for three different cases: 1) "without DR," which refers to the net demand of the distribution network without DR adjustments; 2) "scheduled," which is the expected power demand of the distribution network based on day-ahead or intraday markets; 3) "with DR" profile, which is the result of running our power balance algorithm. Fig. 7 reports also two dotted curves defining the"thresholds" of power flow deviation at this primary substation (10% of the scheduled one).
The net demand of the distribution network from upstream transmission system is the power exchange at the primary substation, which is the sum of total load and total loss of the network minus total generation. As it is shown in Fig. 7, in the middle of the day (e.g., from 12:00 to 15:00) when PV production rises, a portion of the demand is supplied locally so that less power is absorbed from the primary substation. The drop of power in all the curves is due to this fact. The amount of power, which could be locally generated, would not be purchased from the main upstream network. This amount should be estimated by running some PV production forecast tools or simulators in advance. Nevertheless, there is always some error in predictions especially on cloudy days, which may result in fluctuations beyond estimated curve. In our test, an underestimation results in a real-time "Without DR" power profile beyond the thresholds. Fig. 8 provides a closer view of the power flow at this primary substation from 10 A.M. to 5 P.M. The "Without DR" curve shows the power flow if no adjustment is performed by the aggregator based on DSO's reference signals. The power mainly goes beyond the threshold which implies power deficit in the local distribution system. When no DR is applied, this power imbalance is covered by upstream network in a higher spot cost. The "with DR" curve, instead, is the result of running the OPF-based algorithm and implementing DR based on the OPF decision signals. Tables I and II are examples of data exchanged and power adjustments by an aggregator and the DSO for one of the nodes (205351 or 39). Table I reports 1) the power demand (P, Q); 2) the flexibility boundaries (min P, min Q, max P, max Q); 3) the price of energy produced when RES are present (cost gen); 4) the cost of energy when energy loads increase (cost up) and decrease (cost down), respectively. It is worth noting that we have supposed a virtual and futuristic marketplace to promote energy flexibility keeping a realistic proportion between prices from the utility w.r.t prices of load curtailment and generation at the prosumer premises. This is needed to evaluate the OPF working in an hypothetical real-world scenario. However, these prices are some setting parameters that can be easily replaced by end-users without affecting the rest of the proposed system.
The load data in our demo are stored in a module running on a separate machine. It periodically sends load data. This load    The scheduled power profile at the PoC is based on the day-ahead or intraday forecast of prosumer behavior anticipated by distribution system operator. In practice, the operator is committed to respect the scheduled profile. This means any violations beyond some sort of thresholds must be resolved locally in the distribution grid, otherwise there may be more expensive reserve power from the high voltage system to buy or even some penalties for local grid operator. In practice, specially in a cloudy day, at some points, there is deviation of total demand with respect to the scheduled or forecast profile. Fig. 7 depicts the forecast profile and the real measurements of power exchange at primary substation. The real measurements come from the grid simulator. Our OPAL-RT runs the model of network in real-time. The real cloudy-day scenario is injected as generation data to the network model, while the system operator is blind to this data. System operator would run PV simulators to forecast the generation.
Assuming this scenario, the OPF algorithm (see Section III) is invoked by DMS to calculate the total power adjustment needed, and associate it to all network nodes which are candidates to provide flexibility. The high volatility of the residential prosumers due to the presence of rooftop PV panels is controlled in real-time, and the net demand is reduced about 30% in this scenario.

B. Secondary Substation Voltage Regulation Test
In distribution systems, there are several methods to regulate voltage of the nodes across the network depending on the availability of devices and resources. Tap changer transformers are common devices, mainly at primary substations, that regulate the voltage profile of the feeders. To demonstrate the added value of DR-enabled real-time voltage control, we created a scenario in which the penetration of PVs is very different in two feeders starting from node 1 551 412 (i.e., the two middle feeders in Fig. 6). The first feeder from the left is assumed to have a higher demand so that the PV penetration is lower. The second feeder, instead, has secondary substations with lower capacity (e.g., 160 and 250 kVA) and loads with lower demand. A summer sunny day is selected to generate PV production data. With no DR-based real-time voltage control, the voltage profile of some selected nodes would be the plots shown in Fig. 9. Considering a 5% threshold for voltage regulation (an assumption for demonstration purpose), the voltage at substation 9 falls bellow the threshold (< 0.95 p.u. after 8 A.M.). Fig. 10 shows how using a tap changer would increase the voltage profile for substation 9, but simultaneously make the voltage at substation 24 violate the over-voltage threshold. This is due to higher penetration of PV in the feeder where node 24 is, and higher demand in the neighboring feeder where node 9 is. Tap changer is triggered by the voltage drop in one feeder to increase the voltage level at its secondary side. However, this creates over-voltage in the other feeder. Fig. 11 shows the result of applying our DR-based real-time voltage regulation. All voltage deviations across the network can be identified by the algorithms once the measurements are retrieved by the DMS. The voltage sensitivity matrix is obtained and the amount of power adjustments are defined. In this residential area, we assume the PV inverters are not sources of reactive power adjustments, therefore the aggregators receive real power adjustments which could be implemented by DR-based load shedding, battery charge/discharge, etc.

VI. CONCLUSION
Integration of more RES, storage units, electric vehicles, and other new devices into distribution systems would make operation of the system more challenging. The intermittent behavior of variable RES such as wind and PV would make the energy balancing and voltage regulation more difficult, as the existing storage units may be insufficient and forecasting tools are not always very accurate. A real-time management schema is presented in this article, which uses IoT and realtime functions to provide reference signals for the aggregators running demand response algorithms. Two real-time functions are developed and embedded into the schema, one for power balance at primary substations, and one for voltage regulation at secondary substations.
A laboratory test-bed is also developed based on IoT paradigms in which the grid simulator can be replaced by the real grid with minimum effort. The experimental setup proves the applicability of the proposed solution detailing the communication procedure. The results of running the two algorithms for different scenarios demonstrated the added value of real-time control of the network in cost minimization and power quality. The power balance algorithm keeps the power exchange at PoC within the acceptable range around the scheduled profile, which prevents extra high costs. The example of the test case shows more than 5% mitigation of the power deviation (0.3 for 6 MW scheduled power). The voltage regulation algorithm solves the problem of over/under voltage situations caused by tap changer effects in asymmetrically distributed PVs in feeders supplied by the same transformer. Around 2% of the violation with respect to the maximum voltage threshold is recovered thanks to power adjustment through aggregators.
The outcome of the algorithms are sets of load adjustments sent to the aggregators. As a future work, the way this outcome is used by the aggregator to perform demand response can be investigated.