A Qualitative Evaluation Approach for Energy System Modelling Software — Case Study Results for the Open Energy Modelling Framework ( Oemof )

The research field of energy system analysis is dealing with increasingly complex energy systems and their respective challenges. Moreover, the requirement for open science has become a focal point of public interest. Both drivers have triggered the development of a broad range of (open) energy models and frameworks in recent years. However, there are hardly any approaches on how to evaluate these tools in terms of their capabilities to tackle energy system modelling challenges. This paper describes a first step towards a flexible evaluation of software to model energy systems. We propose a qualitative approach as an useful supplementary to existing model fact sheets and transparency checklists. We demonstrate the applicability by evaluating the newly developed “Open Energy Modelling Framework” with respect to existing challenges in energy system modelling. The case study results highlight that challenges related to complexity and scientific standards can be tackled to a large extent while the challenges of model utilization and interdisciplinary modelling are only tackled partially. However, the challenge of uncertainty remains for the most part unaddressed at present. Advantages of the evaluation approach lie in its simplicity, flexibility and transferability to other tools. Disadvantages mostly stem from its qualitative nature. Our analysis reveals that some challenges in the field of energy system modelling cannot be addressed by a software as they are on meta level like model result communication and interdisciplinary modelling.


Introduction
Energy systems are subjected to substantial structural changes, mainly due to environmental reasons and concerns of security of supply, One central driver for these changes is the increasing share of decentralized and intermittent renewable energy based generation units.As energy constitutes one foundation of modern societies, modification within the generation, consumption and distribution of energy affects a broad range of stakeholders.Ministries and likewise municipalities as well as economic and social interest groups are confronted with a constantly changing environment and high uncertainty regarding the future composition and design of increasingly complex energy systems.
Within the sketched transformation process, model based analyses have become indispensable to give guidance in a diverse set of questions.Amongst others, this includes grid control and planning, dispatch and unit commitment, expansion planning, energy market design as well as environmental and social analysis of highly integrated energy systems.Energy system modelling software has been heavily discussed and in recent years, model based results haven been criticized for the black box character of internal model logics and underlying assumptions [1,2].As a result, more scientists have opened their software and data [3] which improves transparency, enables reproducibility and allows other people to re-use or build upon developed tools.Thus, a rough division into a group of closed (1st generation) and a group of open (2nd generation) energy system modelling tools can be derived [4].
The diverse research questions associated with the transformation of energy systems can obviously not be addressed by one single model or approach.This is underpinned by the large amount of existing models and their differentiation along social, technological and economic lines.Therefore multi-purpose model generators or frameworks [4][5][6][7] are of growing importance in the energy modelling community.In this context it is crucial for model users and developers to identify software that is fit for the intended purpose.Due to the nature of model generators and frameworks with their multi-purpose design and versatility this task is not trivial.Hence, methodologies for quantitative as well as qualitative evaluation are important in terms of software selection.For this task scientific model comparisons for specific models and model fact sheets as well as transparency checklists have been proposed (s.section 2).However, there is a lack of comprehensive evaluation of energy modelling software with regard to their contribution in tackling described modelling challenges.
In this paper we propose a qualitative evaluation approach as a step towards model generator and framework evaluation.Moreover, we apply the approach to a 2nd generation energy modelling tool.Within a case study, The Open Energy Modelling Framework (oemof) is evaluated regarding its capability to address present and future challenges in energy system modelling.
In section 2 we give a short overview on existing evaluation approaches and describe the newly developed evaluation approach.Subsequently, we apply the approach in a case study (section 3), which is based on a literature review of existing energy system modelling challenges and oemof.Finally, we discuss the proposed approach and its application in section 4. The last section 5 summarizes the main findings in a short conclusion.

Methodology
An evaluation of energy modelling software can be undertaken by quantitative, qualitative or mixed methods.Quantitative approaches may be used to evaluate performance in terms of run-time or computational traceability as well as accuracy of results.An example is the ongoing project RegMex comparing simulation pathways of renewable energy systems [8].In that context, standard test cases serve as a common basis for model comparison.A mixed quantitative and qualitative approach is used in [9] where the evolution of a model is characterised by comparing different model versions.In that paper specific input and output related metrics are defined to allow for quantitative comparison.
For analysing aspects that cannot be expressed in numbers, qualitative methods can be applied.Systematic reviews of models and presentations of classification schemes [10,11] fall into this category and are important for modellers, model users and decision makers to identify the potential scope of application of a model.Similarly, qualitative model comparison helps to understand the details of and differences between models that are designed to answer similar research questions.The US Energy Modelling Forum has carried out model comparisons since the 1980s by looking at the foci of models, their internal logic and representation and their results (see e.g.[12,13]).Another qualitative approach is presented in [14].In order to increase transparency of energy scenario based studies a transparency checklist is proposed.In addition to enhanced transparency, this list may also provide a basis for model comparison.Besides reviews and comparisons, a presentation of structured arguments provides possibilities for analysing modelling software with respect to e.g.practicability or the degree of openness.To our knowledge this kind of analysis has not been carried out for energy system modelling software.In particular, an approach specifically designed for a qualitative assessment of model generators or frameworks does not exist.
In the following we propose a qualitative approach to evaluate energy system modelling software regarding its capability to face present energy system modelling challenges.The approach is designed to allow for a structured and extensive evaluation of one specific model generator or framework 1 .It is not meant as a substitute but rather as a supplement for the aforementioned approaches focussing on model generators and frameworks.Figure 1 shows the conceptual idea of the developed methodology.The process of evaluation is divided into three phases: 1) Setup 2) Evaluation and 3) Summary of results.In the first phase (Setup), domain specific challenges m ∈ {1 . . .M} including their associated aspects a m,n are identified.This may be undertaken based on a literature review or on an expert based survey.Additionally, central properties j ∈ {1 . . .J} of the analysed modelling software are identified.In analogy to the aspects of each challenge, central characteristics of each property are determined.The Evaluation (2nd phase) is based on the identified challenges and properties.For each combination of challenge and property the relation of characteristics and aspects is investigated qualitatively.Phase 3 summarizes the relationship of each challenge-property pair by aggregating relationships of associated aspect-characteristic pairs.

Case study
In this section we apply the proposed approach to a case study in the field of energy system modelling.According to the approach we first define the Setup of our analysis by identifying domain specific challenges as well as the properties and characteristics of a modelling framework.Subsequently, we perform a qualitative Evaluation of the framework with respect to the challenges.The Summary of results is presented at the end of this section.

Setup
It has already been mentioned by other authors that the field of energy system modelling and its models may seem opaque to outsiders [9,11].One reason for this may be the broad definition of the term energy system model.Depending on the research question, energy system models may range from detailed, highly technical models of small sub-systems to large macro-economic models covering whole economies.Typical criteria for categorizing models are top-down (macro-economic relationship of components) vs. bottom-up approaches (technology specific), simulation vs. optimization of the system being studied and partial equilibrium (e.g.considering only the power sector) vs. equilibrium models (considering the whole economy) [15].For a comprehensive description of the model landscape as well as of model topologies we refer to existing reviews [10,11,16,17].We restrict our analysis to general challenges and their respective aspects in the field of energy system modelling.These challenges relate to steps in the modelling process as described by [18], ranging from the development of a mental model of an energy system to the application of the model including the communication of results.
Coming up with a classification scheme for energy modelling challenges can be compared to proposing a scheme for energy model classification with respect to the generality of the categories.In the case of energy system models, various options of classification exist, though there are 'few models -if any -that fit into one distinct category' [19, p.7].This is similarly true for categorization of energy system modelling challenges.For our analysis we propose the five major challenge categories complexity, uncertainty, interdisciplinary modelling, scientific standards, and model utilization, which are characterized by different relevant aspects, summarized in Table 1.Generally, the relevance of a challenge for specific software may vary as it is determined by the focus of the underlying research question.The subsequent sections provide a detailed description for each of the identified challenges.Diversification, distributed generation and stronger integration of energy sectors with versatile interdependencies are growing challenges for energy system modellers.
Considering the power-heat-transport nexus, integrated models nowadays play a decisive role in providing insight into different flexibility options [33], using excess electricity in an economic way [34] and for meeting climate targets [35].While a high spatial and temporal resolution is required to consider varying weather conditions and cover different flexibility options, spatial and temporal coverage is also necessary for the analysis of long-term development of an increasingly interconnected power system.For instance, Després et al. [15] conclude that long-term energy models would benefit from an improved representation of variable renewable energy sources in the power sector.The growing requirement of flexibility particularly on the demand side (e.g.storages or demand side management) additionally increases modelling complexity in systems with high penetration of renewables.
Although computing resources have been improved significantly in recent years, decisions between model complexity and computational cost need to be taken with regard to optimization algorithms.Attempts in tackling complexity have been made by applying models in a consecutive or interactive manner.For instance, Deane et al. [36,37] soft-link an energy system model with a dedicated power system model.Similarly, Trepper et al. [38] present an integrated approach to assess potential benefits of market splitting in Germany by connecting a European dispatch model to a detailed unit commitment scheduling model.With the liberalization of markets, heterogeneous participants with different objectives and new business cases are difficult to model with classical optimization models that traditionally have been applied in oligopolistic markets.Here, agent-based models have been proposed as a useful tool to test market design [39].In general, agent-based modelling is applicable to complex systems across different domains and disciplines [40].With the objective to represent and cover market behaviour, agent-based models like EMACS (Electricity Market Complex Adaptive System) [41] are applied to electricity markets.In this context, such models can for example be applied to assess cross-border effects of CO 2 -policies [42] or the complex interactions of the huge variety of actors caused by policy measures to integrate renewable energies into the market [43].Although agent-based models can be applied as a classical economic approach capturing the complexity of market actor interactions, it is increasingly utilized to also capture the human dimension which is referred to in the context of interdisciplinary modelling (see section 3.1.3).
The increasing complexity of models is accompanied by a rising amount and complexity of input and result data.Data are crucial because absence of necessary data may hamper the development of new modelling techniques, as Krysiak and Weigt [44] argue in the case of demand side management modelling.Keirstead et al. [17] state that data availability is one challenge for (urban) energy system modelling.Acquiring or generating input data is not a trivial task as it requires versatile software skills (e.g.GIS, databases, reverse engineering) and may be linked to other sophisticated research areas (e.g.meteorology in the case of power production from wind turbines).Therefore, data processing to generate model input data can be one of the most time consuming tasks of the whole modelling process.
Regarding the output, models usually produce large amounts of data that have to be analysed.Therefore, result processing may be a difficult and time-consuming task in itself.Among other aspects, this affects the appropriate visualization of multi-dimensional data (temporal, regional, unit-wise).The challenge complexity with its main challenges sector coupling, temporal or regional resolution, input data and result processing is linked to the challenges of model utilization.There exists a continuous trade-off between modelling complex interactions with the required level of detail and keeping the model clear and comprehensible for the recipients of the results and for the modellers themselves.

Uncertainty
Uncertainty has already been identified as a challenge for energy system modelling decades ago [45].Craig et al. [46] state that uncertainities in long-range energy forecasts are systematically underestimated.Uncertainty in terms of energy system modelling can be classified into a number of aspects.Generally, literature has different scopes, approaches and scientific backgrounds to classify uncertainty which results in different classification schemes [16,18,20].Mirakyan and Guio miss a 'common agreement on typology of uncertainty' [18].They propose a new framework that has a broader scope and a more detailed classification compared to uncertainties described by Pfenninger et al. [16] and Hunter et al. [20].This framework for categorization of uncertainty incorporates energy system modelling, decision making and subsequent planning processes: (1) linguistic uncertainty, (2) knowledge or epistemic uncertainty, (3) variability or aleatory uncertainty, (4) decision uncertainty, (5) planning procedural uncertainty and ( 6) level of uncertainty.
Even though not very often discussed in the context of energy modelling, the aspect of linguistic uncertainty (1) affects energy system planning and decision making based on model results.Linguistic uncertainty arises from natural language being vague, ambiguous, and because meaning of words may change over time [47].An illustrative example is the ambiguous usage of the term model.
Knowledge or epistemic uncertainty (2) covers various levels of uncertainty related to context or framing, data, structure of a model, technical and accumulated uncertainty that includes all other.Various examples for this type of uncertainty exist in literature as this category covers a wide range.Assumed learning rates and consequently future costs (e.g. for renewable energy technologies) are decisive parameters for energy system models [48] as those often aim for minimal system costs.If not carefully chosen, biased results may lead to incorrect policy recommendations if they do not reflect the sensitivity of these assumptions [49].Methodology and key pitfalls of assessing future costs of energy technologies based on learning rates is an important topic among the research community [50][51][52] that illustrates the importance of dealing with uncertainty related to input data.Another problem related to uncertainty is associated with scenario development.Laugs and Moll show that most scenarios only represent a small bandwidth of possible pathways.This under-representation of extreme scenarios hampers the scientific discourses and 'skews the overall outlook on possible energy futures' [53].
Uncertainty arising from the structure of a model is the second major aspect and lies in the nature of models.Due to the complexity of the real world, a model is a simplified representation.A linearised description of an energy system is one prominent example for simplification applied in models that may have strong impact on results [54].
Structural uncertainty has special importance for long-term planning models as they cannot be fully validated [55].Although tackling structural uncertainty is tricky, one attempt is made by DeCarolis et al. [56], who explore the near-optimal decision space with the technique modelling to generate alternatives (MGA) [21].
Variability or aleatory uncertainty (3) refers to 'inherent variability manifested in natural and human systems' [18].It can also be referred to as random or stochastic.The aspect variability can be addressed with established mathematical methodologies.For example, the open source model TEMOA applies stochastic programming [20] to deal with variability uncertainty.For deterministic models other options are scenario and sensitivity analysis or Monte Carlo simulations.
Decision uncertainty (4) stems from decision makers with a different understanding and judgement of objectives and appropriate solutions and strategies [18].For example risk perception or the way of representing model results to decision makers may affect their decision [47].Availability of resources in terms of information and time to process it affects decision making as well [18].According to Wardekker et al. [57], uncertainty perception varies depending on how information is provided.This relates planning procedural uncertainty to the aforementioned aspect decision uncertainty.

Interdisciplinary modelling
The development of energy system models is typically undertaken from an engineering or economic perspective.Jefferson [58] argues that emphasizing equations and economic theories is hindering to focus on complicated factors and their future implications.Furthermore, Wiese [22] states that twenty-first century challenges need to include other perspectives than least-cost optimization.As stated above, differences are inevitable between ideal results of optimization models with one single rational decision maker and real world developments with a multitude of heterogeneous actors [59].In addition to an increased complexity, this is also a challenge from an interdisciplinary point of view since modellers need to integrate perspectives that are not captured by standard economic or engineering approaches.However, if energy research is not undertaken in an interdisciplinary way, researchers 'are not likely to grasp the problems, and thus the solutions to this challenging (energy) research space' [24, p.247].
Social and behavioural factors are important to assess the adoption of renewable technologies [60,61] or the representation of consumers' real behaviour in energy models [62].For example, social acceptance has a relevant impact on grid and wind power expansion [63,64].Thus, attempts exist to capture the human dimension in energy system modelling by applying social science methods.But considering the strong interconnectedness of energy systems and society, social sciences are rather under-represented in contemporary energy research [65].
Another requirement in interdisciplinary modelling results from the strong interdependencies between the energy, water and food sector.Granit et al. [23] argue that increased understanding of the water-energy-food nexus is necessary to achieve sustainable development goals.They present first attempts for integrated tools and state that further cooperation between the modelling disciplines is required.
To comprehend the dimension of challenges in interdisciplinary energy modelling, one has to consider that finding a coherent terminology and taxonomy within one field is already complex.This is referred to as linguistic uncertainty in 3.1.2(subsection Uncertainty).Between different disciplines, a lack of understanding due to different terms impedes a common understanding of energy systems.

Scientific standards
Complying with scientific standards includes the aspects transparency, repeatability, reproducibility and scrutiny.These principles ensure that science moves forward and can perform course corrections through independent verifications [66].Beyond that, these are also fundamental for the societal process, which depends on return of knowledge that has been publicly funded.Repeatability or the sometimes used term replicability describes the ability to repeat an experiment and come to the same results.In contrast, reproducibility means that results can be repeated by a different researcher in a different computer environment [67].Although definitions exist, these two terms are not always utilized with this clear distinction in literature.
Transparency of methods, code and data lays the foundation for the other three aspects, as it is a precondition for building up on existing scientific work in the field of energy system modelling.However, Ince et al. [29] state that for computational science, transparency at all stages constitutes a basic condition for reproducibility.Even if this is fulfilled, reproducibility remains a challenging task due to hardware, software and natural language related uncertainty.The common situation of constantly changing versions of energy system models and the lack of precisely describing these when presenting results adds another dimension to the challenge of reproducibility [9].As Pfenninger et al. [1] argue, full -meaning effective -transparency of energy system models is still hampered by different barriers.Specifically, the lack of open licenses on the original sources of data is an obstacle for making model data publicly available.Moreover, a sparse or lacking documentation of data makes it inconvenient for others to use these data.
To facilitate repeatable analysis, DeCarolis et al. [28] recommend five steps of best practices in energy economic optimization model development.These steps, we argue, can and should be applied to every energy system model: 1. Make source code publicly accessible, 2. Make model data publicly accessible, 3. Make transparency a design goal, 4. Develop test systems for verification exercises and 5. Work towards interoperability among models.In fact, with today's information technology it has never been easier to comply with these recommendations.And indeed, source code of different models has been made publicly accessible in recent years (e.g.Balmorel [68], OSeMOSYS [7], TEMOA [20], calliope [69], PyPSA [6]).Meanwhile, up to now 25 open energy models and frameworks are registered on the website of the open energy modelling initiative [3].Contrary to increasing model transparency, publishing solely aggregated results of energy system models is still a common procedure.For instance, a list of models used in the UK shows that input data and code of the majority of models are not open [11].As almost any result can be generated by modifying decisive input data, variables or code, the common practice makes repeatable results impossible.Attempts exist to overcome these problems.
Regarding data, the Dataverse project is one example of technical support in linking associated data with the published article [70].
While point three on the list (transparency as a design principle) has already been discussed above as the foundation, the fourth point (verification exercises) refers to the aspect of scrutiny.The importance of scrutiny for energy system modelling is (in this paper) mainly discussed in a technical sense but also on the societal level.Point five on DeCarolis best practice checklist addresses applicability and re-usability which is discussed related to model utilization.
On the technical level, scrutiny refers to identifying inconsistencies or faults (so called bugs).Every computer model is prone to bugs, whereas the probability of theses errors increases along with the complexity and size of the model.Detecting bugs is particularly vital in energy system modelling, as small errors may have great impact on the results.Johnson [71] highlights that peer-reviewed open source software has significant advantages related to bug findings.Besides this, Ndenga et al. [72] point out that the size of a community, i.e. users and developers, is one metric for bug reports.
On a societal level, scrutiny refers to the detection of bias in model code and data.The possibility to scrutinize model results is essential for credibility [26] and the development of public trust in the modelling results, particularly as participation of society in the design of energy pathways becomes increasingly important [27].Being widely utilized for policy advice, the trade-off between being policy-relevant without being policy-prescriptive is of vital importance for model-based research [25].Though, Mai et al. [26, p.9] conclude that, accidentally or purposefully, all models incorporate biases.Going one step further, Biewald et al. [73] argue that value-laden and ethical issues cannot and should not be avoided in model-based studies, but assumptions based on ethical opinions should be communicated transparently which can increase policy relevance of these studies.Similarly Edenhofer and Kowarsch [74] state that value-neutral scientific recommendations for public policy means are not possible.As model based research has to deal with normative-ethical aspects, they suggests a new culture in academia that defines the role of modellers as cartographers of solution spaces.Detecting value-laden assumptions is even more difficult than detecting bugs, as software tests fail at this.Hence, again transparency of source code and data is pivotal for energy model usage in policy advice and essential for complying with quality standards [75].
Although all discussed aspects also refer to all computational heavy sciences, Pfenninger et al. [1] argue, that energy policy research is lacking behind other fields in terms of complying with scientific standards.

Model utilization
The aspects of the challenge model utilization are linked to growing model complexity.In the modelling process, three main groups of persons are involved: (1) developers (2) users and (3) decision makers.It is noteworthy that in some cases these groups may not be completely distinct, as developers and users might be identical.In terms of model utilization we identify two main aspects.Firstly, usability that is associated with the user-decision maker interface.Secondly, applicability that can be understood as a problem of 'ease of use' at the developer-user interface or 'ease of adaptation' at the developer-developer interface.
As models only produce useful information if the recipients understand the causal relations, there remains a trade-off between the level of complexity and the general usability.Bale argues that '[m]odellers need to engage with their beneficiaries from the outset so that models are properly scoped and fit for purpose.'[31, p.157].Most notably, this is important as models are made for obtaining insight not for generating numbers [76].
The difficulty to find a common language between developers, users and recipients of their results exists almost as long as the models themselves.In 1976, the Energy Modelling Forum was formed to 'foster better communication between the builders and users of energy models in energy planning and policy making' [30, p. 449].Energy research is generally application-oriented but stands out among other policy fields with externalities.Due to its vertical and horizontal complexity, entailed costs and strong path dependency, energy models are indispensable for policy support [77].However, the decision maker's idea of useful information may significantly disagree with those of the users' point of view [26, p.9].This is a crucial point, as '[a] model is not fit for purpose if it is developed without sufficient critique of the motives for producing the model.' [31, p.155].Therefore, the aspect communication of results is a crucial aspect of the modelling process.In particular, valuable information may not only be lost at the user-decision maker interface.To tackle this problem in operation research, the concept of model assessors, analysing and evaluating models for decision makers has come up long time ago [78].Additionally, Strachan et al. [32] propose further improvements like platform-based expert user groups for coordination and interdisciplinary external stakeholder review for energy system models.
Between developers and users, an easier and better understanding of model mechanisms than at the developer-decision maker interface could be assumed.Unfortunately, this seems not always to be the case.One example for differences in understanding models and results is the discussion about results from the NEMS model (see [79] and [80] for details).The usefulness of a model rises if it can be applied to a diverse set of problems and by different researchers.Ideally, the expense of a developer for building up on an existing model should be lower than the expense for building a new one from scratch.In the context of energy system modelling it has been argued that '[s]ociety as a whole saves time and money if researchers avoid unnecessary duplication' [1, p.212].Considering the rising amount of open energy models for similar purposes [3], it yet seems that developers tend to rather develop a new model instead of using existing ones.A reason for this may be the rising complexity and different software skills required to adapt models.Consequently, being open does not seem to be sufficient in terms of usability, even if a deep modelling understanding exists.Thus, the aspect applicability is also connected to scientific standards as it is vital for the repeatability and more importantly to the reproducibility of results.

Properties of the Open Energy Modelling Framework
The framework itself and the characteristics we refer to in this section are described in existing publications [4,81] and the online documentation of the framework [82].In the following, additional literature is referenced where necessary.The framework has been developed for the analysis of energy supply systems considering power and heat as well as prospectively mobility.It consists of different libraries with defined interfaces for their combination.Applications depict concrete energy system models constructed from oemof libraries.Inside comprehensive models, specific parts of such an application can be developed flexibly by combining oemof-libraries with external libraries.The core concept of oemof is based on a network structure which describes the general topology of an energy system.
Available applications built within oemof (e.g.renpassG!S [83], reegis [84], HESYSOPT [85]) illustrate that the modular approach of the framework allows applications with very different objectives.The general description, the toolbox character, the flexibility concerning temporal and spatial resolution

Evaluation
As described in section 2, within the evaluation process, each characteristic of a model property is evaluated regarding its contribution to the aspects of each energy system modelling challenge.Figure 2 shows the conceptual idea of the applied evaluation process which is now applied on the selected case study.Based on the outlined energy system modelling challenges and their respective aspects, the contribution of the Open Energy Modelling Framework (oemof) to address those is evaluated.

Complexity
Due to its structural properties, oemof allows to create flexible energy system models which can easily be adapted and linked.For example, modelling strongly integrated energy systems is straightforward due to its network structure.Energy systems can even be developed separately by different persons and be connected afterwards.If, for instance, a specific sub-system should only appear in certain calculations, it can be connected and disconnected flexibly to a graph based energy system representation with all its components depending on the requirements.Additionally, generic classes can be used to easily integrate other models.This has for instance been tested with the PyPSA library [6].Applications built on oemof have shown the integration of electricity, heat and mobility as well as energy market simulation models [86] and power flow analyses [87].Moreover, the four aspects of oemof's structural properties enable complete work flows which facilitate input data handling and result processing.The temporal and regional resolution are not fully addressed as no specific methodologies are implemented within the framework.Nevertheless, the problem of resolution is partly addressed.For example the optimization library solph provides timestep-flexible modelling with timestep-dependent weighting.
By sharing essential structural properties, hard-and soft-linking of oemof-based applications can be performed even with different modelling approaches or with models of different regional and temporal resolutions.As Trutnevyte et al. [88] argue, this can be a key for energy system analysis.Furthermore, applications built on oemof's generic structure may be extended by other developers.The object-oriented approach generally provides a flexible interface for extensions.Based on the underlying concept, incorporating new modelling methodologies is possible although not done yet (e.g agent-based models based on core components).Moreover, the framework provides a complete toolkit for modelling highly integrated, renewable energy based systems.Thus, not only optimization models can be build, but also input data such as feed-in or demand time series may be generated on the basis of oemof functionalities.Especially the feed-in libraries allow for a high spatial and temporal resolution.Overall, the underlying generic basis in combination with a flexible programming language facilitates the modelling process for complex and changing systems.

Uncertainty
With its collaborative concept, oemof may already contribute to a process of addressing linguistic uncertainty.Identifying common elements in energy system modelling can help in determining coherent terminologies.Here, experiences from collaborative modelling can be a key for the necessary interface definitions of different existing models.Therefore, at least ambiguity is tackled as developers have to agree on specific terms during the development process.Multiple perspectives of developers with different backgrounds in the process of collaborative model development decrease the risk to overlook or omit decisive features of energy systems.Hence, with this property, epistemic uncertainty related to model structure uncertainty is partly addressed by oemof as well.At the moment the framework does not provide any functionalities tackling problems of variability uncertainty.Nevertheless, above mentioned techniques to tackle uncertainty such as stochastic programming could be added with reasonable effort since they are already integrated in the underlying Python optimization packages.Furthermore, methods to explore a large space of parameter variations (i.e.scenario or sensitivity analysis) could be easily built on top of an oemof based model.
However, sensitivity analysis or stochastic programming counteract the challenge of complexity, as these measures are computationally expensive.Even if such approaches are applied under the reasonable assumption of increasing computational resources, missing regulatory certainty in combination with disruptive events can hardly be tackled by existing technical methodologies.Hence, policy makers need to be aware that reliable policies and regulatory schemes are crucial for the degree of reliable advice energy modelling can provide as for policy making.Instead of handling these uncertainties as practical constraints they have to be analysed additionally (e.g.influence of temporal and regional resolution on results).This is important as growing complexity of the modelled systems necessitates reducing model complexity.In turn, structural uncertainties of these simplified models increase.Connected to this issue are open questions that directly link to model utilization (e.g.'Is a model with unquantified structural uncertainties fit for a specific purpose?').Despite mentioned problems, we recommend putting future effort in incorporating existing approaches for tackling variability uncertainty.

Interdisciplinary modelling
The provided framework does not directly address the aspect of taking down disciplinary walls between energy system modelling and other research disciplines.Even though oemof is not restricted to a specific mathematical approach, capturing the human dimension is not at the core focus of the framework.Nevertheless, oemof's core package may serve as a base for implementing other methods (e.g.agent-based models).In general, the concept allows to integrate other modelling techniques, i.e. approaches that suit interdisciplinary modelling.
Another important characteristic originates from the collaborative development.This kind of development is not common in energy system modelling yet.In the case of oemof, it triggered the process of finding common definitions and understanding of energy research related problems.As a consequence, two important arguments in terms of interdisciplinary modelling for oemof's contribution are: (1) experiences from the collaborative development process can be translated to interdisciplinary model development, (2) the resulting generic basis allows for an easy coupling of energy system model components with new model components of other research areas (e.g.components in water resource modelling).

Scientific standards
With a strict free software, open source and open data philosophy, oemof satisfies a basic condition in terms of transparency and allows for repeatability, reproducibility and scrutiny.However, publishing undocumented source code of complex models still represents serious obstacles for others.Therefore, the four-level-documentation, code-review, and version-control of oemof are important elements for effective transparency.Moreover, oemof provides functionalities to increase transparency on application level.With a standardized input/output data format, simultaneous publishing of model source code and the corresponding documentation (including data) is possible.Human readable, spreadsheet based and thus cross-platform data structure provides a simple user interface and lowers the entry barrier for new users significantly.Supported by a clear version-control workflow, this allows for the release of monolithic model versions including all data and documentation.Thus, scientific model results are not only transparent but also reproducible.Additionally, this principle allows a stand-alone run of applications without the need of supplementary software infrastructure (e.g.databases).Hence, applications can also be archived and exchanged easily.
From the beginning of the project, the collaborative framework development with different backgrounds of its developers required a high quality documentation.As a result, transparency is improved for new developers and external users through the mentioned four-level-documentation.With the high level of transparency and the collaborative development the framework supports scientific progress.Additionally, the collaborative framework development of oemof increases the probability of finding bugs.Moreover, controversial approaches can be discussed and the results can directly lead to improved code.With these properties, oemof fulfils all five best practice recommendations for repeatable analysis worked out by DeCarolis et al. [28].decision makers and other stakeholders.Furthermore, the aforementioned common terminology enables the different groups to communicate effectively.Beyond challenges outlined above, oemof supports the energy system modelling community by providing a basis for model development that is highly reusable and adaptable.This could save other resources (e.g.time) in terms of parallel work especially when it comes to long-term projects with a high extent of interfaces between groups and work packages.Therefore, oemof is in line with the argumentation of increased productivity through collaborative burden sharing [1, p.212].Also, the applicability of oemof models is improved by the underlying structure.Once this structure has been internalized by users and model developers, its usage and development is straightforward.Through the modular and layer based design -where each layer has a certain degree of interdependency from the remaining part of the framework -new users can create applications based on the desired module without knowledge of the complete framework.Moreover, the overall concept is consistent and the graph-based structure is in line with the code, data and documentation.Thus, even complex cross-sectoral models or applications developed with another scientific scope can be understood quickly.Generally, a well-defined modelling workflow increases overall transparency.
However, the general problem of result communication is not addressed by the framework as it can only be addressed on a meta-level.The problem of how results are communicated is a recurring point in literature.Communication of energy system modelling results fails when recipients only see concrete numbers (e.g. total energy system cost) as an outcome, though models should primarily be seen as a tool for understanding mechanisms and getting insights [74,76].This simple idea seems to be ignored on a regular basis but the problem cannot be tackled by a model or framework itself.Extended usage of pre-prints and discussions about results and methodologies within the community before the actual publishing process can be one step into the right direction.Proposals to reinvent the modeller-policy interface made by Strachan et al. [32] also seems promising for overcoming this problem.

Summary
While conducting the case study with the Open Energy Modelling Framework, we categorise the diverse challenges by complexity, uncertainty, interdisciplinary modelling, scientific standards, and model utilization.As argued within the evaluation process, oemof contributes by addressing these challenges based on its particular characteristics.Figure 3 summarizes all challenges and properties with their specific contribution levels.
Important issues related to complexity are particularly addressed adequately by oemof's structural properties.Due to the generic code basis and the object oriented implementation the modular framework allows modelling of integrated renewable energy systems, easy model linking as well as input and result processing.Most aspects of uncertainty are not tackled by oemof, but the collaborative development and structural properties may reduce linguistic and structural uncertainty.Important aspects like variability uncertainty are not addressed.Although this may be improved in future versions, we argue that uncertainty can not only be tackeld through software but should also be reduced by regualtory consistency.Additionally, the trade-off between complexity and uncertainty has to be balanced by modellers and model users.As delineated in section 3.2, oemof lays important foundations for interdisciplinary modelling as the generic basis allows for modelling components originated in other research areas.However, this has not been implemented in any oemof applications so far.Developing common understanding of interdisciplinary problems is not a trivial task but a necessary basis for appropriate modelling.Here, collaborative development may play an important role by translating into interdisciplinary model development.Being developed in an academic context, challenges related to scientific standards are addressed thoroughly with the free software and open source philosophy.Collaborative framework development requires a high quality documentation and improves transparency for new developers and external users.Moreover, potential bugs can be identified and fixed quickly due to a growing community and direct feedback between users and  developers.This level of addressing is underlined by being in compliance with the best practice recommendations of DeCarolis et al. [28].Regarding challenges in terms of model utilization, oemof's philosophy constitutes an important precondition for tackling those.Effective transparency at all stages is crucial for communication of results as well as for application building and re-usability of models.Similarly to uncertainty, we find that many aspects of the challenge model utilization can not be tackled by software but have to be addressed on a different level.
From the evaluation we conclude that challenges related to complexity and scientific standards are strongly tackled.In contrast, uncertainty is not addressed at present as major aspects of this challenge are not sufficiently considered.Regarding the challenges model utilization and interdisciplinary modelling we argue that oemof captures these partially.However, it should be mentioned that our analysis indicates that some of the aspects of these challenges can not be addressed by a software as they are rather located on a meta level.

Discussion
The evaluation of energy system modelling frameworks is realized by a 3-stage approach that can be applied to arbitrary energy system models, model generators or frameworks: (1) Setupidentification of challenges and its aspect as well as properties and characteristics, (2) Evaluationcomparison of challenges and properties, and (3) Summary -presentation of results.This approach of evaluating the capability of energy system modelling frameworks provides a structured, hands-on, and sufficiently detailed procedure that goes along with different disadvantages and advantages.
Disadvantages are mainly caused by the qualitative nature of the approach.First, the selection of important aspects and properties by the analyst may be subjective and influence results.Moreover, a missing scheme for quantification leads to a certain amount of vagueness.In particular, when results from the evaluation of two frameworks are compared the vagueness can be a severe issue.As there exist no weighting for the challenges, results cannot reflect the importance of an individual challenge.Thus, an overall positive analysis may overlook the importance of a single aspect.However, the approach may be improved by combining or supplementing it with quantitative elements to improve cross comparison of different modelling tools in the future.Advantages of the proposed approach lie in its simplicity and flexibility in terms of challenges and models.Modellers are able to reflect modelling tools in a structured manner regarding selected challenges.Thereby either the tools or challenges can be exchanged flexibly e.g. the approach can be transferred to another framework which is evaluated with regard to the challenges defined in this study or the same framework is evaluated in terms of different challenges.At the same time, research needs and possible gaps can be identified by means of the approach.As demonstrated in the case study, general recommendations for future efforts in energy system modelling can be derived when a tool of the newest model generation is selected and evaluated.
In this case, the presented evaluation identifies general issues in energy system modelling.First of all, we demonstrate that it is not possible to address challenges completely on the same level at once.Additionally, we show that some approaches for tackling challenges may counteract each other (e.g.complexity and uncertainty).Therefore, we think that a process of applying and analysing different approaches, accompanied by a discussion within the scientific community is necessary to tackle the diverse challenges sufficiently.This process is particularly relevant as we identify challenges that are recurring problems and can not be tackled solely by technical modelling approaches.Among these are problems of result communication and interdisciplinary modelling.We argue that these must be primarily addressed through changes in communication and organizational structures.Especially, the fundamental questions if models are fit for purpose and what model results may reveal -or may not reveal -seems to be disregarded too often.Here, collaborative modelling can be a valuable method supporting this process.

Conclusions
Though different approaches have been made in evaluating and comparing models, systematic evaluations of frameworks is under-represented.This paper presents an evaluation of the Open Energy Modelling Framework (oemof) regarding its capability to address present challenges in energy system modelling.For this, we present a qualitative evaluation approach that we applied to oemof in a case study.We compile and classify current challenges in energy system modelling based on a literature review and map central characteristics of oemof to the classified challenges in order to evaluate how the latter are addressed.
Within the evaluation, it has been outlined that oemof provides a solid basis for tackling present challenges in energy system modelling and analysis.We show that the challenges complexity and scientific standards are addressed well, interdisciplinary modelling and model utilization are addressed partly, while uncertainty remains untouched.Therefore we recommend to include uncertainty analysis in studies.This holds for oemof as one representation of a open source framework, but also for other energy modelling frameworks and model generators in general.To allow for rather sophisticated methods like stochastic programming, more research in the field of complexity reduction within energy system models is needed.In addition the case study reveals that certain challenges can not be addressed by improving software, as they are located on a meta level.Among those are issues of model result communication and interdisciplinary modelling.
The application of our approach in a case study demonstrates its general capability to assess energy system modelling software in terms of present and future challenges.Advantages of the evaluation approach lie in its simplicity, flexibility and transferability to other tools whereas disadvantages mostly stem from its qualitative nature which always induces some degree of subjectivity.

Figure 1 .
Figure 1.Schematic illustration of the methodology: Each characterstic of the framework properties is evaluated in terms of its contribution to each aspect of the energy system modelling challenges.

3. 2 . 5 .
Model utilization Policy measures and planning processes based on the results of energy system models cannot be affected directly by the modeller.The open source and -depending on the specific applicationopen data approach enables decision makers and planners to obtain a deeper understanding of model results considering details of model inputs.This may enhance communication between modellers, Preprints (www.preprints.org)| NOT PEER-REVIEWED | Posted: 21 August 2017 doi:10.20944/preprints201708.0069.v1

Figure 3 .
Figure 3. Visualization of evaluation results from mapping challenges and their aspects with properties and their characteristics.

Table 1 .
Categorized energy system modelling challenges.

Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 21 August 2017 doi:10.20944/preprints201708.0069.v1 makes
oemof a framework instead of a model.It is implemented in Python using several packages (e.g. for data analysis, optimization) and can optionally be combined with a PostgreSQL/PostGIS database.From literature we derive that the central properties of oemof can be divided into three major categories summarized in Table2.

Table 2 .
Properties with respective characteristics of the Open Energy Modelling Framework.