Optimizing Carbon Emissions in Electricity Markets: A System Engineering and Machine Learning Approach

1. Introduction

With the continued growth of global economic activity, greenhouse gas emissions have risen to unprecedented levels, posing significant environmental challenges. To address the escalating climate crisis, countries worldwide have engaged in ongoing global efforts. Building upon the foundation laid by the Paris Agreement, the IPCC's Sixth Assessment Report, released in 2023, has emphasized the urgent need for more ambitious action and stresses the importance of achieving net-zero emissions by mid-century. Countries and regions have successfully formulated carbon emission reduction targets and measures within this framework [1,2,3,4]. The electric power grid significantly contributes to total carbon emissions, and its transformation is crucial to achieving zero-carbon goals [5,6]. According to a report by the International Energy Agency (IEA), carbon emissions from the power sector account for nearly a quarter of global energy-related carbon emissions, and this proportion is likely to increase in the future [7]. In the United States, individual states have taken leading roles in implementing clean energy policies [4]. New York State, in particular, is aggressively revamping its grid to meet its ambitious clean energy goals. The state aims to generate 70% of its electricity from renewable sources by 2030 and achieve 100% carbon-free electricity by 2040[8]. Thus, New York State offers an excellent case study for examining the challenges and opportunities in transitioning to clean energy due to its diverse energy landscape, dense urban areas, and commitment to innovative policy solutions.

Figure 1. Countries and regions that have proposed “zero carbon” or “carbon neutral” climate goals [1].

Figure 1. Countries and regions that have proposed “zero carbon” or “carbon neutral” climate goals [1].

Studying carbon emissions from power systems, market structures, and policies is crucial in controlling and reducing carbon emissions. Masoumzadeh, etc., point out that the design of the electricity market, through incentives such as subsidies and tax incentives, can effectively promote renewable energy development [9]. At the same time, by implementing a carbon emissions cap and trading system, the market can effectively limit high-carbon-emission electricity production activities [10]. These policies drive the deployment of low-carbon technologies and help phase out carbon-intensive power sources. For example, Fridstrøm points out that a reasonable carbon pricing mechanism can significantly enhance the market competitiveness of renewable energy and promote the optimization of the energy structure [11]. In addition, Wang et al. [12] analyze the impact of the European Carbon Trading System (ETS) on the power industry. They find that the carbon trading mechanism effectively reduced overall carbon emissions and accelerated the innovation and application of clean technologies. Demand-side management is also an important strategy that regulates carbon emissions by directly affecting electricity consumption. Research by Melgar-Dominguez et al shows how demand response programs can help the grid balance supply and demand and reduce reliance on carbon-intensive backup power, thereby improving energy efficiency and reducing overall carbon emissions [13]. Cross-regional power trading also has significant benefits in reducing carbon emissions. Acworth et al. emphasize that through cross-regional cooperation and power trading, renewable resources in each region can be used more efficiently, such as using the abundant wind energy resources in one area and the solar energy resources in another area, reducing reliance on local high-carbon power generation [14]. It is particularly noteworthy that the regional marginal pricing (LMP) mechanism has become a popular and effective power market pricing strategy globally. As stated by Chen et al., LMP provides accurate price signals by reflecting each grid node's real-time supply and demand conditions and transmission constraints, thereby optimizing the allocation and use of power resources [15]. Researchers further show that the LMP mechanism effectively promotes renewable energy utilization and reduces carbon emissions by incentivizing companies to adjust power generation strategies according to market demand and price changes. For example, when the LMP value is high, electricity demand increases, or renewable energy is in short supply. This will encourage companies to invest more low-carbon or zero-carbon power generation resources to meet market demand [16]. Therefore, through in-depth research and optimization of the application of LMP, not only can the operating efficiency of the power market be improved, but also the low-carbon transformation of the power system can be promoted, making an essential contribution to achieving the global carbon neutrality goal.

With the rapid development of data technology, machine learning has become a powerful tool for processing power system big data and discovering deep relationships and patterns. In power system research, machine learning is widely used in many fields, such as load forecasting, market analysis, and carbon emission estimation. For example, Ungureanu et al. showed that power demand and market prices can be accurately predicted through machine learning models, optimizing power generation and grid operation strategies, and reducing unnecessary carbon emissions [17]. Especially in carbon emission analysis, machine learning methods can effectively process and analyze large-scale environmental and operational data and identify critical factors affecting carbon emissions. In the study of Nguyen, machine learning technology was applied to investigate the emission data of multiple power stations and successfully identified the operating parameters and environmental factors that most significantly affect carbon emissions [18]. In addition, by integrating market data and meteorological information, machine learning models can predict the carbon emissions of the power system under different weather conditions to support power grid management and dispatch [19]. Machine learning also shows great potential in improving the efficiency of renewable energy utilization. Research shows that machine learning models can optimize the dispatch of wind and solar power, maximize the utilization of renewable energy, and reduce dependence on fossil fuels by accurately predicting energy production and market demand [20]. Applying this technology not only improves the efficiency of energy production but also helps reduce the carbon footprint of the entire power system. In addition, machine learning is also used to optimize the electricity market's design, such as optimizing LMP pricing strategies. By analyzing historical transaction data with machine learning models, researchers can better understand the relationship between price fluctuations and carbon emissions and then design more effective market mechanisms to promote the deployment of low-carbon power generation [21].

Systems engineering, as an approach, holistically considers various factors and interactions in complex systems and provides us with a reasonable framework to analyze and study power system problems involving multiple variables and high complexity. The unique advantage of this approach is its ability to simultaneously and systematically address various levels of power system design, operation, policy development, and environmental impacts. In studying carbon emission issues, the application of system engineering is critical for framing the situation while considering the scope of the problem. It helps us identify and analyze key factors affecting grid carbon emissions from a broader perspective. It also integrates knowledge from environmental science, economics, and power engineering to form a more comprehensive solution. For example, researchers can use system engineering approaches to assess different electricity generation technologies' environmental impacts and economic benefits, considering technology maturity, policy support, and market demand [22]. Although system engineering is widely used in many fields, such as manufacturing and aerospace, this approach still needs to be defined in studying carbon emissions from power systems. Traditionally, many studies focus on specific technology or policy interventions, but fewer take a whole-system perspective to comprehensively consider various interactions and impacts. For instance, existing research may focus on optimizing single renewable energy technologies or applying carbon capture and storage technologies while ignoring the interactions between these technologies and existing power system structures [23]. We can solve these problems more effectively by introducing a system engineering approach. This approach allows researchers to conduct more comprehensive models, simulations and when appropriate, optimizations, considering the entire process from electricity production to consumption and related policy and market factors [24]. Research by Chu et al. demonstrates how solar and wind energy can be integrated through a system engineering approach to maximize these resources' utilization efficiency while reducing the system's overall carbon emissions [25]. Therefore, applying system engineering to study carbon emissions from power systems provides an innovative perspective that can help us better understand and optimize complex energy systems. By comprehensively considering the interaction of various technology, policy, and market factors, system engineering can provide strong support for developing effective carbon emission reduction strategies and realizing a sustainable energy future [26].

This study uses system engineering methods and machine learning techniques to develop a novel analytical model to explore the complex relationship between regional marginal prices (LMP) and carbon emissions in electricity markets. LMP is the core economic indicator of the power market and directly affects the operational decisions of power generation companies and the energy allocation strategy of the entire market. By accurately analyzing the dynamic relationship between LMP and carbon emissions, this study deepens the understanding of the interaction between electricity market operations and environmental impacts. It provides objective data support for formulating carbon emission reduction strategies based on market mechanisms. More importantly, the research results can guide power market policymakers, helping them optimize market design and achieve sustainable power industry development. By integrating system engineering and machine learning technologies, the model built in this study can handle complex and changing data and predict and evaluate the impact of different policies and market changes on carbon emissions. Applying this innovative method allows us to understand and optimize carbon emission issues in the electricity market and contribute to promoting the low-carbon transformation of the electricity industry more accurately. Through this research, policymakers and market operators can more effectively use LMP as a regulatory tool to guide market participants to adopt more environmentally friendly power generation methods and optimize energy allocation, thereby reducing the carbon footprint of the entire power system. This research improves our understanding of the electricity market structure and environmental policy's interactive effects. It opens new perspectives and methodologies in electricity market design and environmental sustainability research.

2. Data Source

This study's dataset contains two main parts: regional marginal price (LMP) data and carbon emissions data, covering different regions of New York State, recorded in hourly units every day. LMP data covers eight regions, including Capital District (CAPITL), Central (CENTRL), Guinness (GENESE), Hudson Valley (HUD VL), Long Island (LONGIL), New York City (N.Y.C.), Northern (NORTH) and Western (WEST). This data includes LBMP, Marginal Cost of Losses, and Marginal Cost of Congestion. These data come from research published by M. V. Liu, B. Yuan, Z. Wang, J. A. Sward, K. M. Zhang, and C. L. Anderson in the journal "IEEE Transactions on Power System" in July 2023[27]. The carbon emission data details the carbon emissions in each region every hour. The data comes from the power industry monitoring program the US Environmental Protection Agency (EPA) provided [28].

Data cleaning is performed first in the processing stage to exclude incomplete or abnormal data records. Subsequently, all data were normalized to a unified hourly time format to facilitate analysis. LMP and carbon emissions data were integrated to ensure time and regional label consistency. This process includes checking data consistency and completeness to ensure the combined data set is ready for cross-analysis. In addition, depending on the analysis needs, data may need to be transformed, for example, converting time series data to panel data and preparing data formats suitable for statistical analysis and machine learning models. These steps ensure the high quality of data and consistency of analysis, providing a solid foundation for exploring the relationship between LMP and carbon emissions and assessing the impact of different policies and market changes on carbon emissions [29].

3. System Engineering Analysis

In the system engineering analysis stage, applying a system engineering approach to analyze and solve complex issues involving LMP and carbon emissions in the electricity market. This stage mainly includes several key steps: Problem Definition and System Boundary Identification, System Architecture and Functional Decomposition, Model Development and Integration, and Scenario Analysis and Trade-off Evaluation.

3.1. Problem Definition and System Boundary Identification

To define the problem scope and identify system boundaries, a correlation analysis was conducted to explore the interrelationships between key variables in the electricity market dataset. Through the generated correlation matrix and heat map, the relationships between regional marginal price (LBMP), the marginal cost of losses (Marginal Cost of Losses), the marginal cost of congestion (Marginal Cost of Congestion), and carbon emissions were examined. These findings provide data support for defining the main research questions [30].

As shown in Figure 2, the correlation analysis reveals several significant relationships that define the system boundaries. A moderate positive correlation (0.24) exists between LBMP and carbon emissions (Total CO2 Mass), suggesting that increases in LBMP may be associated with higher carbon emissions. More notably, a strong positive correlation (0.54) is observed between the marginal cost of losses and carbon emissions, indicating that system inefficiencies significantly impact environmental outcomes. The marginal cost of congestion shows a robust negative correlation with LBMP (-0.74), implying that in the electricity market, increased congestion costs may lead to decreased LBMPs.

These correlations are particularly useful as they highlight the complex interplay between economic factors (LBMP), system inefficiencies (losses), grid management (congestion), and environmental impact (carbon emissions). These findings guided the focus on developing a systems engineering approach that integrates pricing mechanisms, grid efficiency, and emission reduction strategies. By defining these relationships, the boundaries of the system are established, encompassing not just the physical power grid, but also the economic and environmental aspects of electricity markets. This comprehensive view enables addressing the multifaceted challenges of reducing carbon emissions while maintaining economic efficiency in power systems.

3.2. System Architecture and Functional Decomposition

To analyze the direct and indirect relationships between these variables further, a transdisciplinary engineering tool called Interpretive Structural Modeling (ISM), proposed by Warfield in 1973[31], will be used through Platform for Integrated Transdisciplinary Design Toolkit [32,33]. ISM is a method for identifying and summarizing relationships between a defined problem and specific variables, enables us to structure complex relationships in a system, establish hierarchies, and visualize direct and indirect dependencies between elements [34].

The ISM process began with the construction of a Structural Self-Interaction Matrix (SSIM) to understand the hierarchy of influence and causal relationships between each variable. This facilitates a more explicit and systematic understanding of the dimensions of the problem [35]. As shown in Figure 3, O indicates no direct relationship between the two variables. X indicates a two-way relationship between variable i and variable j, which means they influence each other. A indicates a one-way relationship from j to i.

There is no direct relationship (O) because the hour does not directly affect carbon emissions or costs but is used to locate data points. Total CO2 Mass and LBMP: is marked with an "A," indicating that LBMP may affect total carbon emissions, but total carbon emissions do not affect LBMP. LBMP and Marginal Cost of Losses are marked with an "X," meaning there is a two-way relationship between the two. LBMP may change due to changes in loss costs, and loss costs may also be affected by LBMP. LBMP and Marginal Cost of Congestion are marked with an "X," indicating that there is also a two-way relationship between the two. Increases or decreases in congestion costs may affect LBMP and vice versa. Marginal Cost of Losses and Total CO2 Mass: are marked with an "A," indicating that loss costs may affect carbon emissions, but carbon emissions will not affect loss costs.

The SSIM was then transformed into a digraph, providing a visual representation of the flow and relations to the key attributes of learning programs, as illustrated in Figure 4[36]. As shown in Figure 4, the Hour variable lacks structural relationships with other variables and appears as an isolated node, which is consistent with previous correlation analysis. Carbon emissions are linearly connected to LBMP, which is also influenced by the cost of losses and congestion. LBMP has a direct linear relation with the cost of losses, and the cost of losses reciprocally influences LBMP. Furthermore, the cost of congestion has bidirectional relationships with both LBMP and the cost of losses.

The last activity of the Integrated Transdisciplinary Design Toolkit ISM implementation is the Matrix Impact Cross-Reference Multiplication Applied to a Classification (MICMAC) analysis, which will identify the driving power and dependencies of the identified key attributes [37,38]. The MICMAC analysis arranges the factors of system performance with respect to driving power and dependence into four clusters, as illustrated in Figure 5. Cluster I includes one factor (Hour (1)) that is autonomous. Autonomous factors have low driving power and low dependence, which means they have minimal influence on and are minimally influenced by other factors in the system, making them relatively disconnected from the system's overall behavior. Cluster II includes no factor that is dependent. Dependent factors have low driving power and high dependence, which means they are strongly influenced by other factors but have little ability to influence the system themselves, making them outcome variables of the system. Cluster III includes four factors (Carbon Emission (2), LBMP (3), Cost of Losses (4), and Cost of Congestion (5) that are linked. Linkage factors have high driving power and high dependence, which means they are both influential and vulnerable to changes in other factors, making them critical variables that require careful monitoring as any change in these factors can affect the entire system and create feedback loops. Cluster IV includes no factor that is independent. Independent factors have high driving power and low dependence, which means they have a strong influence over other factors while being relatively unaffected by them, making them key strategic variables that can be used to influence the system's behavior.

The results of the ISM and MICMAC analyses identify the most influential variables in the system, namely LBMP, Cost of Losses, and Cost of Congestion. These analyses also reveal the complex interdependencies between these variables, which must be considered in any solution approach. Furthermore, they highlight the central role of LBMP in influencing carbon emissions, suggesting that pricing mechanisms could be a key lever for emission reduction. Additionally, the analyses demonstrate the linked nature of the key variables, indicating that a holistic approach addressing multiple factors simultaneously may be most effective.

These insights from the ISM process directly inform the next step of inputting data into a machine-learning model. The identified key variables and their relationships provide a framework for feature selection and model design, ensuring that the most relevant and influential factors are considered. Moreover, the understanding of variable interdependencies can guide the interpretation of machine learning results and the development of targeted strategies for reducing carbon emissions while maintaining economic efficiency in the electricity market.

3.3. Model Development and Integration

Based on the comprehensive system analysis provided by the ISM and MICMAC techniques, the identified high interdependencies among key factors such as LBMP, Cost of Losses, and Cost of Congestion, along with their significant influence on carbon emissions, necessitate a modeling approach capable of capturing non-linear relationships and intricate feedback loops, makes neural networks emerge as an ideal solution tool for addressing the complex interplay of variables in the electricity market and carbon emission system. Neural networks excel in this regard, offering the ability to learn and represent complex, multidimensional interactions without requiring explicit programming of these relationships [39]. Furthermore, the linked nature of the key variables, as revealed by the MICMAC analysis, suggests that a holistic approach is needed. Neural networks can effectively integrate multiple inputs and consider their combined effects, aligning well with this requirement [40]. The adaptability of neural networks also allows for the incorporation of new data and insights as they become available, making them well-suited for the dynamic nature of electricity markets and environmental policies [41]. By leveraging neural networks, it becomes possible to develop a predictive model that not only accounts for the identified system complexities but also provides a flexible framework for exploring various scenarios and strategies aimed at reducing carbon emissions while maintaining economic efficiency in the power system

The neural network constructed contains multiple layers, each with its specific role. First is the input layer, whose dimensions correspond to the number of features in the training data. Next, the network consists of three hidden layers with 128, 64, and 32 neurons, respectively, using ReLU (rectified linear unit) as the activation function. These hidden layers are designed to increase the expressive power of the model, allowing the network to capture complex patterns and nonlinear relationships in the data. The output layer contains a neuron that predicts carbon emissions. The model is compiled using the Adam optimizer, an efficient stochastic gradient descent method suitable for processing large-scale data sets. The loss function is the mean square error (MSE), which measures the difference between predicted and actual values. The entire training process is executed silently within the specified 100 cycles, and 20% of the data is used as a validation set to monitor the performance during the training process.

The evaluation results of the model show that the root mean square error (RMSE) is 191.42. These indicators reflect the accuracy and error of the model in predicting carbon emissions. As can be seen from the figure, the data predicted by the model (red dashed line) and the actual data (blue solid line) have consistent trends at most data points, although there are some deviations. This shows that the neural network is effective at capturing the main trends in the data, although there is room for improvement in capturing specific peaks and fluctuations. This visualization helps us intuitively understand the model's performance and points out the direction for future optimization, such as adjusting the network structure or further refining feature engineering to improve the model's prediction accuracy for electricity market dynamics.

4. Result and Discussion

In this study, we take the HUD Valley region (HUD VL) as an example to evaluate the impact of different market policy changes on carbon emissions. We set up four different market scenarios to simulate these policy changes: Baseline Scenario: This scenario is set under normal operating conditions with a regional marginal price (LBMP) of $25.0/MWh and a marginal cost of losses of $0.15/MWh, the marginal cost of congestion is -$20.0/MWh. Increase LBMP: In this scenario, LBMP is increased to $30.0/MWh to simulate an increase in market prices while other cost parameters remain unchanged. Reducing loss costs: This scenario reduces the marginal cost of losses to US$0.10/MWh to explore the potential impact of lowering operating losses on carbon emissions. Reduce the cost of congestion: In this scenario, the marginal cost of congestion is reduced to -$10.0/MWh, and the positive effect of reducing congestion on the environment is examined. By simulating these different scenarios, we can better understand the direct impact of market dynamics on carbon emissions, thereby providing data support for market policy formulation.

Figure 7. Impact of LMP on Carbon Emission in HUD VL.

Figure 7. Impact of LMP on Carbon Emission in HUD VL.

Further analysis of different regions, according to our model predictions, different market scenarios significantly impact carbon emissions in other regions. In the Capital Region (CAPITL), carbon emissions increase from a baseline of 300.54 tons to 333.95 tons under increased LBMP, suggesting that higher electricity prices may inadvertently promote emission growth. Conversely, reducing loss costs decreases emissions to 296.88 tons, demonstrating the positive effect of operational efficiency on emission reduction. The congestion cost reduction scenario results in emissions of 316.41 tons, highlighting the importance of grid congestion management in environmental protection policies. Similar trends are observed in the Central Region (CENTRL), where baseline emissions of 307.26 tons rise to 336.05 tons with increased LBMP. Loss cost and congestion cost reductions show positive effects, with emissions decreasing to 309.54 tons and 324.12 tons respectively. The GENESE region exhibits notably lower emissions overall, ranging from 4.67 to 5.97 tons across scenarios, indicating significant regional variations in emission profiles and sensitivities to market changes.

The Hudson Valley (HUD VL) region presents a particularly striking case. Baseline emissions of 257.78 tons increase to 263.35 tons with higher LBMP but show significant reductions to 248.88 tons with lower loss costs and a dramatic decrease to 219.88 tons with reduced congestion costs. This underscores the potential for targeted congestion management strategies in emission reduction. Other regions, including Long Island (LONGIL), New York City (N.Y.C.), North (NORTH), and Western (WEST), exhibit similar patterns. Generally, increased LBMP scenarios lead to higher emissions, while reduced loss cost and congestion cost scenarios result in emission reductions.

These comprehensive results provide strong evidence that carbon emissions from the power system can be effectively controlled and reduced through nuanced market price adjustments and targeted cost management strategies. However, they also reveal the complexity of these relationships, as the magnitude and direction of effects vary across regions and scenarios. The findings offer robust data support for formulating targeted environmental policies tailored to various regions. They highlight the potential of market mechanisms in regulating carbon emissions but also underscore the need for a multifaceted approach that considers pricing, efficiency improvements, and congestion management simultaneously. The consistent trends across diverse regions reinforce the model's reliability and the universal applicability of certain market-based strategies for emission control. However, the regional variations also emphasize the importance of tailored approaches for each area.

5. Conclusions

This study uses a system engineering approach including a transdisciplinary method, combined with advanced machine learning technology to analyze the relationship between regional marginal prices (LMP) and carbon emissions in the electricity market. Applying system engineering ensures that the research can comprehensively consider and integrate market design, policy adjustments, economic factors, and environmental impacts, providing an overall analysis framework. This comprehensive approach enables this study to identify various influencing factors and systematically assess the combined impact of these factors on carbon emissions.

Through simulations of different market scenarios, this study reveals the specific impact of market policies, such as changes in LBMP, adjustments to loss costs, and congestion costs on carbon emissions. A system engineering approach enables these complex relationships to emerge clearly. It highlights the importance of considering the system's integrity and the interactions between its parts when formulating environmental policies and optimizing electricity market structures. This interdisciplinary comprehensive analysis demonstrates the central role of system engineering in solving environmental and energy problems, providing solid theoretical and methodological support for formulating effective carbon emission reduction strategies.

In addition, the study recommends optimizing the power market design based on system engineering principles, emphasizing the reduction of carbon emissions through comprehensive regulation of market mechanisms and operational strategies. Future research should explore applying system engineering methods in power system management, especially in data-driven decision support systems, to promote sustainable development and environmental protection in the power industry.

In summary, the applied system engineering approach provides a valuable perspective and tool that enables policymakers and market operators to fully understand and manage the complex interactions between electricity markets and environmental policy. This approach allows strategies to be more accurately assessed and implemented to optimize energy efficiency, reduce carbon emissions, and drive the power industry toward the goal of low-carbon transformation.