Technology Readiness Levels (TRLs) in the Era of Co-Creation

1. Introduction

Technology Readiness Levels (TRLs) is a methodological tool that emerged to assess the maturity of new technology. The approach was first developed and standardized by NASA (Sadin S.R., et al., 1989). The initial intention was to provide for a disciplined way to differentiate between technology readiness levels. The original definition of TRL was a scale from level 1, corresponding to the formulation of a new concept up to level 7 which ascertained that the system had been successfully demonstrated in a real environment (Sadin S.R., et al., 1989). Although initially used within NASA, and aimed at securing a common language between research and operational personnel, the TRL approach has now extended well beyond NASA and the aerospace industry and has been widely adopted across research and industry. In the process, it has also expanded up to nine levels (Mankins J.C., 1995); indeed, this nine level scale corresponds to the formulation currently in use worldwide. Numerous entities beyond NASA are using the TRL system, and Mankins provides evidence of a global adoption, even though it appears that some users may opt for some slight modifications or more extensive documentation on the meaning of every single readiness level. Such an example is the European Space Agency’s Handbook (ESA, 2008) which defines each level by means of numerous standards that a technology must meet to be categorized at the particular level.

Interestingly, the TRL approach is widely used as a mechanism to classify innovation. Innovation is strongly related with the development and implementation of new technologies, and TRL can be a useful tool for assessing the progress and potential of these technologies. For example, investors may use TRL to evaluate the potential risks and rewards of investing in a particular technology, while policymakers may use TRL to assess the feasibility of implementing a new technology in a particular sector.

Thus, in the EU HORIZON programme for Research and Innovation proposers and contractors are typically required to position their research and development plan against this ladder and to explain what both the starting TRL and the aspired target TRL are. The figure below illustrates the EC nine levels of the TRL ladder and provides a short description of the associated meaning of each level.

The nine level model was also implemented by the US Department of Energy (DoE) with small variations in some levels (Roman F. Bastidas Santacruz et.al., 2023). In other cases the TRL was modified to assess readiness of not only technology but the process of incremental innovation (Lee M.C, et.al., 2011) and the innovation readiness level was defined.

Additionally, two more levels, 10 and 11, have also been recommended in the literature (Straub J., 2015; Hicks B., et al., 2009), and this is something which manifests the constantly evolving nature of technology maturity classification, towards an always greater resolution. However, in the following discussion we will restrict ourselves to the nine levels as shown in Figure 1.

In the discussion below we will present an empirical case which provides evidence that the TRL approach is not always able to provide guidance as regards the next step up the innovation ladder. We will discuss why this has been the case and what the explicit limitations and eventual workarounds have been found to be. We will also attempt to generalize and delineate the particular features that render the TRL approach insufficient in specific settings. Broad guidelines for extending the TRL framework to address the observed limitations will also be discussed.

2. Related work

The current operational and technological context of many manufacturing companies often obliges them to develop innovative solutions leading to a complex asset lifecycle composed of different phases, starting from conception and planning, progressing through design and engineering, construction, validation, verification and commercialization. Roman F. Bastidas Santacruz et.al. (2023) developed and proposed a TRL template aiming to support the lifecycle management of manufacturers’ assets, fostering the exploitation of related data and knowledge since the development phase of the technologies employed. The proposed TRL model was structured in nine levels, classified in three main phases, and was considered an improvement of the EC and NASA TRL standards. Nonetheless, some weaknesses were also identified during the verification phase. Improvements were recommended with the intention to make the assessment and verification of the assets’ TRL easier.

In 2022 during the Sustainable Places 2022 conference, eleven European projects joined forces on a clustering workshop entitled ‘Low-TRL Renewable Energy Technologies’ (Perez Caballero L.M., Neira D’Angelo F., Tschentscher R. et al., 2022). These EU-Horizon 2020 research and innovation programs addressed high risk technology developments for breakthrough renewable energy and fuel technologies, and their activities focused on providing knowledge and scientific proof of the feasibility of the proposed concepts. They were presented and brought together trying to co-address similar challenges in terms of bringing their technologies to TRL9. In order to do so, a validation of their TRL was a prerequisite and their rank was TRL 3–4. Recommendations that followed proposed a roadmap, to move further up the TRL ladder, for market uptake and full integration in the energy system.

At the same time, the accelerating use of artificial intelligence (AI) and machine learning (ML) technologies led Alexander Lavin et.al. (2022) to research the definition of a principled TRL process to ensure robust, reliable and responsible systems. Their proposed Machine Learning TRL framework highlights that the existence of data-related requirements along every step would ensure that the development process considers data readiness and availability, as it will not only help define ML-specific testing considerations (levels 5 and 7) but it will also help surface ML-specific failure modes early. Its synergy with the existing, industry standard software engineering practices would allow it to handle unique challenges. Their research concludes with the assumption that the proposed framework’s distinct advantage is the nomenclature: an agreed-upon grading scheme for the maturity of an AI technology, and a framework for how/when that technology fits within a product or system, enabling everyone to communicate effectively and transparently (Lavin A. et.al., 2022). Thus, when it comes to AI technology TR models differ from other cases (Godoe P. and Johansen T., 2012; Flavian C. et.al., 2020), as AI is a multidimensional construct, hitherto not widely explored in this context.

Thereupon it should be highlighted that no matter the sector, successful innovation involves more than a great idea (Bogers M. et.al., 2019) as collaboration with others makes things happen (Gafour O.W.A, Gafour W.A.S., 2020) and even though new ideas may often be born into a reactionary environment, collaboration has the ability to overcome inertia (Matthias G.W., et.al., 2019) and unlock the chains of convergent thinking, resulting in something innovative. On the other hand co-creation means including ‘transdisciplinary actors’ (Dübner et.al., 2018) and other key stakeholders, especially people who would be affected by a specific decision, in a decision-making process. In order to do so, co-creation is a methodology that is based on iterations during a creation process (Sjödin D. et.al., 2020), which takes collaboration to the next level. When co-creating, stakeholders dive in deep (Kuenkel P., 2016; Chatty T. et.al., 2022); they bring each person’s unique perspective, skill sets and experiences to produce the best possible solution with the highest impact (Lasker R.D. et.al., 2003). However, a collaborative partnership is just not enough, especially when creating new products or strategies as new technologies encounter many implementation challenges hindering or halting introduction (Yu J.C. et.al., 2021). As co-creation produces something that did not exist before (Sanders L., Simons G., 2009), this implies that the end result of a co-creation process is to have created something or to have brought something into existence (Ind N., Coates N., 2013).

An interesting background to this treatise is the classification work on modern innovation presented in the literature by Carliss Baldwin et.al. (2023). These researchers start from a description of the three key types of innovation: single user innovation, where an individual or a firm contemplates investment in an innovative design; producer innovation, where producers undertake larger designs aiming at many end users; and open collaborative innovation, involving users and various third parties that collectively generate and share a design. The authors rigorously investigate the particular conditions affecting the costs incurred and the value generated to conclude on what effectively makes a particular setup advantageous and competitive. They also discuss hybrid types of innovation where more than one of the above three fundamental types coexist. We will argue below that co-creation is often a setup that cannot be well described by the current TRL scheme. According to the classification presented in this work, co-creation, in fact, is a hybrid innovation model, including aspects of the open collaborative innovation model as well as the producer innovation model. Apparently, the current TRL approach mostly reflects the producer innovation model that has been the dominant approach in the 20th century. Indeed, Schumpeter (1934) positions producers at the center of the innovation stage, stating ‘it is the producer who initiates economic change’. Teece (1996, 2000) and Romer (1990) echoed Schumpeter’s views by suggesting that the vast majority of innovative design results in private profit maximizing firms. The dramatic reduction of communication costs of recent times is what essentially enabled the open source innovation model to emerge and currently occupy a measurable and increasingly important role in the innovation arena. To this extent it is perhaps that the current TRL model, by being based on the key hypotheses and practices of the 20th century, may fail to fully and adequately model the maturity evolution of new era innovations that are of the open source model or are hybrids, sharing an important part thereof.

The details of open collaborative innovation have also received increasing attention in the literature. Ghosh (1998), Raymond (1999), and Baldwin (2006) showed that the key condition for collaborative innovation to be a rational and potentially preferable option when compared to the produced innovation paradigm is that communication costs incurred by each participant joining the collaboration are well offset by the value of the design contributed by the other parties. This condition essentially allows design modularity and propels open collaboration. And it is a condition more and more met in current times when the cost of communication is rapidly reducing. This comes in stark contrast to the centralized innovation model of the 20th century where the firm itself was by far the most suitable entity to innovate in mass-produced products and processes, something well and in detail established also in the literature (Chandler A.D. Jr., 1977; Hounshell D.S. 1985). The high cost of communication in those days did not allow the potential of open collaboration to be unleashed. Nowadays the facility to locate and use several communication channels allows even the public sector to approach co-creation.

Co-creation has also captured the attention of the public sector as the new public management theory relies on the concept of collaborative interaction, networks and partnerships with the private sector and industry. In the literature review, Nuno Baptista et.al. (2020) classified co-creation benefits in the public sector as innovation related and improved decision making. According to the authors, the present context of austerity and reduced public resources has led policy makers and politicians to consider the benefits of co-creation, co-production and innovation. Within this context for innovations to produce the outcomes that matter, it is important that the key stakeholders be involved in the design (Baptista N. et.al., 2020) and hence the TRL level is adequate. Once again the question emerges of whether the current TRL model can highlight the true and effective readiness of these innovation developments and thus suggest the next natural step in ascending the maturity ladder.

It is also important to distinguish between open collaboration and co-creation. Although a co-creative setup may also be an open collaborative one, this is not always the case. Co-creation may also apply in completely restricted multi stakeholder setups as it may also exist in a closed collaborative environment, where the typical producer innovation coexists with open collaborative aspects. This scheme is the one that underpinned our empirical case, to which we will turn to immediately below.

3. The case of an AI based energy forecasting technology

In the framework of a HORIZON research project (TRUST AI, www.trustai.eu) we were responsible for the development of an energy forecasting approach based on artificial intelligence. Though a variety of such AI forecasting models have been developed for the last twenty years, our forecasting approach would be quite unique in that it would be able to provide user explanations of the forecast. This would then create confidence on the side of the users and this confidence is considered as essential in the literature; an example is the case of demand response applications, which rely on forecasts and where the user perception may impede uptake of flexible pricing and participation in such demand response schemes. We will not provide any further technical detail here as our purpose in this paper is not the AI forecasting itself but the classification of the particular development along the TRL scale.

Thus, at the project onset we started from a TRL 2, having a clear formulation of the technology concept that we would seek to develop and validate in real settings. Our validation was done in a real building, one that could be considered between ‘relevant’ (TRL 6) and ‘operational’ (TRL 7). Depending on the application context it may in some cases be difficult to always draw a clear line between these two levels. Our interpretation was that a truly operational environment would present more data collection uncertainty than the one in which we tried our algorithm, and in which data issues were already streamlined and presented no true difficulty. Thus, an operational environment would perhaps present some further data challenges, which we were in this case not exposed to.

However, the question that soon arose was not so much about whether the achieved TRL was that of a 6 or 7. Instead, the question pertinent to our discussion here was how one can possibly climb further up the ladder, reaching TRL 8 and 9. What can TRL 8 mean in our case? How can you qualify a forecasting algorithm? Even more, how can you possibly generate a market model around a fully developed system, when the system is a forecasting algorithm?

The answer is simply that you cannot. It is highly unlikely that one will find in the market such a technology, per se. Such a development may indeed generate some new value but will not, by its very nature, ever amount to a self-standing market value.

We would also argue that this is hardly any rare case. On the contrary, it is quite typical that a new technology presents some innovative features but falls short of enabling, by itself, any viable business case. At that point one may of course consider traditional licensing instruments and arrangements. However, if one wishes to remain on the maturity upgrading path this is of course no option. Licensing may provide some financial benefit but will take you off the TRL ladder for good. If you wish to push your way upwards on the TRL ladder, licensing is obviously no solution. Indeed, TRL has been designed for endeavors strategically looking forward to reaching the highest level. Licensing may be a way out of this aspiration but what happens if it is not possible or not desirable?

We would argue that co-creation is then the only option left for our technology endeavor. By co-creation we effectively acknowledge that our algorithm is no carrier of a standalone market value and that some multi stakeholder environment is due, one where our specific development would be an integral part of a broader solution, to which it can potentially increase the market value.

The figure below illustrates the exact positioning of the demand forecasting (dark gray rectangles) in such a multi stakeholder environment. The term multi stakeholder denotes that many third parties can integrate applications (third party app library) and extract data (third party data). The demand forecasting in question is just one case of such a third party app.

With regard to the innovation classification discussed in the review, this environment is of a hybrid nature; it includes open source functionality (open collaborative innovation paradigm) but there are also parts of the functionality that are priced (producer innovation paradigm).

In the following, we will not be interested in the technical details of the above architecture, and we will restrict ourselves solely to how this particular co-creative setup evolved and especially how this impacts upon the TRL classification.

Figure 2 illustrates these items in joint action, in three different shades of gray.

Figure 2. Placement of a demand innovative and explainable forecasting solution in a broader multi stakeholder environment of three partner roles [1], [2] and [3] as well as a potentially generic future role [1a].

Figure 2. Placement of a demand innovative and explainable forecasting solution in a broader multi stakeholder environment of three partner roles [1], [2] and [3] as well as a potentially generic future role [1a].

3.1. TRL in co-creation

New technology that reaches the points TRL 6 and 7, like our particular case, but has no possibility to move further up the ladder, will need to branch off to an alternative, co-creative path. The following figure shows what we have practically found that this path included. Indeed, four more steps have been found necessary to accommodate the new value setup. Incidentally the 4S method (State, Structure, Solve and Sell) used in project management comes close to effectively describing these missing steps. Essentially the fourth step is just a reiteration of TRL 7 (Technology demonstrated in operational environment) which is now rephrased in TRL 7C4 Co-creative solution demonstrated in operational environment.

This new structure is illustrated in Figure 3 below.

We will now describe how we were guided empirically to define these additional steps as well as the specific responses we provide in our endeavor.

Figure 3. The adaptation of the TRLs to address co-creative solutions.

Figure 3. The adaptation of the TRLs to address co-creative solutions.

• 7C1- Concept development for a co-creative setup (STATE where the value lies)

After realizing that there was no way for the particular forecasting development to independently move up from the TRL 6/7, we embarked on the formulation of a co-creative concept. This is shown as the first step of the branch off, and called 7C1: concept development for a co-creative setup. The challenge for this 7C1 step would be threefold: first, to define a concept that would be able to reach beyond TRL 7; second, to make sure that our particular development (explainable forecasting algorithm) was a value-adding part of this concept; and third, to make sure that the new development could add value to the other parts of the co-creative solution. Indeed, as highlighted by Baldwin (2023), any collaborative solution in order to be viable needs to generate tangible value to all parties, exceeding the effort consumed, one by one.

To address these requirements we initially considered bringing together the forecasting technology with a building management system (BMS) technology that we deliver in house and has been market available for some time. Figure 2 illustrates this co-creative venture. The light gray is the pre existing functionality and the dark gray is the newly embedded one.

However, linking a BMS with novel forecasting did not seem to result in a tangible added market value. In particular, the third condition set above was not fulfilled, as the BMS did not appear to gain any tangible market value from the demand forecasting. Although some value would result for the BMS, it was rather unlikely that this value could be monetized.

Thus, this approach alone was not sufficient; although linking with the BMS was potentially interesting, it was not sufficient to secure a co-creative solution that would be beneficial for all parties. At that point we tried to sketch what this new value could possibly be. Following a number of internal brainstorming sessions we came to the conclusion that a promising direction could be that of a demand response solution. Demand response technology, in a narrow sense, assists users to plan their energy uses so that they may benefit from low prices. The more dynamic the pricing tariff the bigger the potential benefit. Demand response is also used in a more broad sense to denote any change of demand patterns that may be informed and driven not necessarily by price but by other, for instance behavioral related, aspects. An example could be a thermostat change to reduce energy costs without compromising thermal comfort. Even more, we were now confident that as demand response relies on forecasts to issue advice and recommendations to the building users, our forecasting technology was a vital element of the overall solution and it would add new value that could now be monetized.

Most importantly, this approach surpassed the limitation identified in the early phases; the demand response was potentially a tangible value adding layer to the BMS, and therefore the BMS gained in value by feeding its data to the demand response controller. All three parties appeared to benefit from this arrangement.

Thus, the 7C1 phase concluded by having defined a market oriented concept, one that would bring together three value adding components as illustrated below:

Figure 4. The co-creative value setup in our case study.

Figure 4. The co-creative value setup in our case study.

It is interesting to note at this point that the multi-stakeholder environment was not there in the first place. It only emerged empirically in 7C1 as a potential direction in pushing the forecasting application up the TRL ladder.

Additionally, integrating the demand forecasting in the co-creative framework allowed us to consider more broadly the integration of third party apps in the same framework as illustrated by the notation ‘1a’ in Figure 2. Indeed, it is only half of the truth that co-creation leveraged our demand forecasting development and allowed it to proceed up the TRL ladder. More than this, it is that co-creation was essentially conceived as a powerful concept that could besides leveraging the forecasting app also create additional benefits, by increasing the potential of any third party to link to the platform and allowing it to emerge in a collectively managed environment, something we are currently investigating how to best manage.

Key challenge for 7C1: The identification of a co-creative value adding scheme that can generate added value to all partners (three in this case) and a market opportunity for the weaker, in this perspective, component (item [1]).

• 7C2- the definition of the co-creative actors (STRUCTURE how the value is delivered)

After the co-creative concept definition we entered 7C2, i.e. the definition of the co-creative actors that would be able to provide their part while also subscribing to the co-creative value. Although this is presented as a second step, in practice it was performed in a rather iterative way together with 7C1; various possible actor setups were considered and these helped formulate the value itself. Depending on the case the step can run iteratively or sequentially with Step 7C1.

In principle there may be several approaches to investigate, depending on the links, alliances and partners that may be available. In our case we did not have to reach much out of the business group as item [1] was already available within the company and item [3] was in development by a close business partner, who had no difficulty in understanding the merit of the specific co-creative value setup proposed. Additionally we considered value adding the engagement of a consumer association for validation purposes, although it has till now proven difficult to conceptualize a clear cooperation framework.

Challenges for 7C2: Overall 7C2 is a risk management exercise; the more you have to reach out of your human network the greater the risk and the communication costs. The only means of mitigation is a clearly formulated concept in 7C1, one that makes clear what the contribution and the expectation from the co-creative setup can be.

• 7C3- the prototyping of the co-creative solution (SOLVE the particulars of the solution)

This is a currently ongoing phase, aiming at the prototyping of the co-creative environment as shown in Figure 2. Methodologically, there is nothing new in this step, as more or less it is already provided for in the classical TRL formulation. It includes a mix of activities typically characterized as TRL 4 to 6, with a main view on the integration and interoperability of the three items, each of which had already independently climbed up to 7 and beyond.

• Challenges for 7C3: Interoperability and seamless integration of the three items.

3.2. The case of an early inception of co-creation

Of course, co-creation may be deliberately set up as such from the very beginning. Should this be the case, moving up from TRL 6/7 should not be impossible; having a market goal from the very beginning makes walking up all TRLs realistic and potentially feasible. In this case the same concepts discussed above can be moved lower down the ladder, while 7C3 is not really needed as its activities are dispersed over TRL 4 to 6.

Figure 5. Early time co-creation.

Figure 5. Early time co-creation.

Indeed, this was not the case for us! With our demand forecasting algorithm our key limitation occurred because the specific development came about with no even remote market concept in mind. It started as a formal piece of research, which day after day appeared more and more interesting as it successfully climbed up to TRL 5/6, at which moment possible market models became an issue of concern.

We would also argue that this late scenario is not any rare event; it is quite typical, especially for academia-originating research and development activities which are, more often than not, pursued with no clear market horizon in mind. But these nevertheless may, on the way, generate some tangible value, albeit not of the type required to really reach, on its own, the market.

4. Results

In our empirical case, that of an innovative development in explainable energy demand forecasting, at a certain point in time we realized that the specific development was inherently limited in terms of its market perspective and could therefore not reach beyond a TRL 6/7. We understood that co-creative setups were a possible and promising way towards providing a market potential for this development, a direction we are currently working on.

We also found that the classical formulation of the TRL model of innovation has important shortcomings in describing the particulars of our approach. The classical TRL has emerged based on notions of producer innovation and not really of collaborative and co-creative setups that present challenges not to be adequately traced on the ladder. However, such setups are more and more frequently encountered in the innovation arena and foster the need for an adaptation of the TRL approach.

We presented a conceptual framework to extend the TRL ladder to account for co-creation. Three more steps were found necessary: to define the co-creative value, to define the co-creators and to make sure that the economics of the co-creation are clear and rational to all, as well as to secure the interoperability of their solution items. The challenges of each one of them was briefly discussed.

Our case was one of late co-creation, meaning by this that the co-creative idea occurred later, while the innovative result was already at TRL 6/7. However, most of the considerations are also pertinent in early co-creation when the co-creative setup is in place from the very onset.

As confirmed by extensive literature research, we also found that the communication costs are the main concern for the viability of a co-creative pathway. These have to be counterbalanced by the additional value resulting independently and for all co-creators, and not just in aggregate terms. This is a most critical condition for the viability of the scheme and for securing an effective move up the TRL ladder.

5. Conclusions (and future work)

The producer innovation model is by no means the only one in use, as it was during most of the 20th century. Both the user as well as the open collaborative model are increasingly entering the innovation scheme and shaping the course of innovation. Baldwin (2023) provides a comprehensive review of the economics of these three models, in an effort to understand conditions where each presents benefits, as well as understanding the mechanisms at play that may erode this advantage.

These models, as well as the many hybrids that may result therefrom, need to be meticulously considered in the light of the current TRL formulation. The TRL approach is a product of the 20th century and reflects the dominant producer innovation model of those times. We have empirically found that this model does not allow us to see the true potential value as it emerges in our time; a co-creative and not just a producer mindset is required for such a refined and adapted model to clearly emerge.

Indeed, similar empirical evidence needs to be collected and similar investigations will need to be carried out over all possible hybrids, bringing together producer, collaborative and user innovation, to see if and how these can be really described by the current TRL formulation, or if some amendments are due, such as those suggested in our particular hybrid case study, merging producer and open collaborative innovation. Even more dynamically, one needs to consider how such setups can unleash new value in innovative approaches which, if left trapped in the producer model and mindset, are not able to reach high on the TRL ladder.

As we approach the end of this treatise we find it pertinent to make a reference to a radically different approach to innovation, as introduced by Harvard professor Clayton Christensen and his co-authors in his seminal Prosperity Paradox (2019), and to see how this resonates with the TRL discussion. In this work, Christensen provides a compelling historical description of numerous incidents of so-called market generating innovations and how they have benefited their initiators but, far more importantly, also the public domain, having pulled in many subsequent investments for social infrastructure.

The approach suggested by Christensen is radically different in that it pays little if any attention to the ‘newness’ of any given development, a trait we typically tend to consider as defining innovation. Instead, Christensen takes a value driven view, whereby he assesses innovation in terms not of newness but of its market generation potential. In his research and in this book in particular he recounts numerous cases stripped of any ‘newness’ that, however, excelled in terms of market generation, especially in the context of emerging economies. To be fair, one has to credit Schumpeter as the first who ever so clearly traced the line between ‘invention’ fostering newness and ‘innovation’ fostering value. Here is how he puts it in his landmark Fundamentals of Economic Development (1949): ‘Although entrepreneurs of course may be inventors just as they may be capitalists, they are inventors not by nature of their function but by coincidence and vice versa. Besides, the innovations which it is the function of entrepreneurs to carry out need not necessarily be any inventions at all. It is, therefore, not advisable, and it may be downright misleading, to stress the element of invention as much as many writers do.’ Thus, he clearly draws the line between the two terms. Innovation is for the first time clearly not a synonym of invention.

Whether we consider Schumpeter, in drawing the line between invention and innovation, or Christensen, in his robust, empirical evidence-based market generating approach to innovation, it is not difficult to discern the inadequacy of our current ‘newness’ centered TRL formulation to describe the evolution of the value generating innovation idea fostered by both. Indeed, we believe this would be a fascinating area of future research, similar to the case of co-creation discussed in more detail in this paper. In the light of related empirical evidence, we would once again need to consider a possible overhaul of the current TRL formulation which, by reflecting almost exclusively the ‘producer innovation’ and the ‘newness’ concepts, appears to be inadequate for providing for an effective and useful structuring of the maturity of a multifold and multipurpose, modern era innovation.