Guideline-enabled data driven clinical knowledge model for the treatment of oral cavity cancer acquired through a reﬁned knowledge acquisition method

: Validation and veriﬁcation are the critical requirements in the knowledge acquisition method for the clinical decision support system (CDSS). After acquiring the medical knowledge from diverse sources, the rigorous validation and formal veriﬁcation process are required before creating the ﬁnal knowledge model. Previously, we have proposed a hybrid knowledge acquisition method for acquiring medical knowledge from clinical practice guidelines (CPGs) and patient data in the Smart CDSS for treatment of oral cavity cancer. The ﬁnal knowledge model was created by combining knowledge models obtained from CPGs and patient data after passing through a rigorous validation process. However, detailed analysis shows that due to lack of formal veriﬁcation process, it involves various inconsistencies in knowledge relevant to the formalism of knowledge, conformance to CPGs, quality of knowledge, and complexities of knowledge acquisition artifacts. Therefore, it is required to enhance a hybrid knowledge acquisition method that thwarts the inconsistencies using formal veriﬁcation. This paper presents the veriﬁcation process using the Z formal method and its outcome as an enhanced acquisition method – known as the reﬁned knowledge acquisition (ReKA) method. The ReKA method adopted veriﬁcation method and explored the mechanism of theorem proving using the Z notation. It enables to identify inconsistencies in the validation process used for hybrid knowledge acquisition. Additionally, it reﬁnes the hybrid knowledge acquisition method by discovering the missing steps in the current validation process at the acquisition stage. Consequently, ReKA adds a set of nine additional criteria to be used to have a ﬁnal valid reﬁned clinical knowledge model. The criteria ensure the validity of ﬁnal knowledge model concerning formalism of knowledge, conformance to GPGs, quality of the knowledge, usage of stringent conditions and treatment plans, and inconsistencies possibly resulting from the complexities. Evaluation, using four medical knowledge acquisition scenarios, shows that newly added knowledge in CDSS due to the addition of criteria by ReKA method always produces a valid knowledge model. The ﬁnal knowledge model was also evaluated with 1229 oral cavity patient cases, which outperformed with an accuracy of 72.57% compared to a similar approach with an accuracy of 69.7%. Furthermore, ReKA method identiﬁed a set of decision paths (about 47.8%) in the existing approach, which results in a ﬁnal knowledge model with low quality, non-conformed from standard CPGs. In conclusion, ReKA is formally proved method which always yields valid knowledge model having high quality, supporting local practices, and inﬂuenced from standard guidelines.


Introduction
Trust in the knowledge base is a crucial factor in the adoption of clinical decision support systems (CDSS) used for medical diagnosis and treatment plan [1].It mainly depends on the reliability of the knowledge source and the consistency of the knowledge acquisition method [2].There are diverse sources of clinical knowledge, such as patient data, clinical practice guidelines (CPGs), clinical trials, systematic reviews, and even social media.Various knowledge acquisition approaches have been proposed to acquire clinical knowledge from these sources.For example, using machine learning and ontological approaches, knowledge models from patient data are created [3][4][5], and different cognitive approaches are used to develop knowledge models from CPGs and other medical resources [6][7][8][9].Depending on the requirements, these knowledge models may need to be transformed into different model formats.For example, the knowledge model from CPGs can be converted into computer-interpretable guidelines (CIGs) so that it could be directly plugged into CDSS for inferencing.Furthermore, sometimes it is required that the knowledge acquisition methods transform two different knowledge models (sharing the same domain problem possibly with different sources) into a unified knowledge model.It is critical in knowledge engineering disciplines that each transformation, provided by the designed knowledge acquisition method, shall ensure the two basic requirements: 1.The transformed knowledge model is the valid representation of the source knowledge model(s).
2. The transformation process is consistent enough to produce always the valid knowledge model.
Figure 1 shows the knowledge transformation with a set of knowledge acquisition methods in general.
The two basic requirements, for each transformation is depicted as necessary questions to be answered, at each knowledge acquisition method of transformation.Question 1 reflects the first requirement mentioned above, and the answer is to provide a validation mechanism in the knowledge acquisition method.Question 2 represents the second requirement of the knowledge acquisition method, which necessitates verification mechanism in the knowledge acquisition method.In a nutshell, validation, and verification are the critical requirements in the CDSS development process to ensure that the knowledge model is valid, and the entire knowledge acquisition method is consistent.
In terms of verification, most of the existing approaches [8,10,11] emphasize on the principles of knowledge engineering.However, none of them have focused on the alignment of the verification process to the development processes of CDSS.On the other hand, formal methods are widely used in software engineering disciplines such as verification of program [12], formal modeling for scenario-based requirement specification [13], formal verification of secured online registration protocols [14], and formal verification of web services on cloud infrastructure [15].Additionally, some attempts were made to use the formal method (Z notation) to express the knowledge base structure and reasoning mechanism in the form of software architectural style.For example, Gamble et al. [16] applied Z notation to formally model the knowledge-base to get the clear distinction of reusability of knowledge, enhanced understandability, and flexibility of specification in comparison to traditional knowledge specification approaches.This paper introduces the formal verification process, using Z notation, for our earlier proposed hybrid knowledge acquisition method of Smart CDSS [17] -which is intended to produce guideline-enabled data-driven knowledge model.In hybrid knowledge acquisition, we equipped the method with the sophisticated validation process.Although, at that time, the knowledge model created for oral cavity cancer was validated based on the well-established validation criteria and test-based validation process.However, the knowledge acquisition method was not formally verified for internal consistencies.The adaption of the formal verification process gives an enhanced knowledge acquisition method -which is known as a refined knowledge acquisition (ReKA) method.In ReKA, we are using Z notation.The selection of Z notation was mainly based on its key features such as data rich formalism, ease in knowledge modeling, and support of tools.
It is important to mention here, that the artifacts of the proposed verification process (using z notation) align to the content of a development framework that was indigenously used for the development of Smart CDSS in the cancer domain.The development framework for Smart CDSS is based on RUP [18,19] and ISO RM-ODP processes [20,21].To the best of our knowledge, the existing approaches had neither explored the use of Z notations for the verification of knowledge acquisition nor used the formal methods as a method content in a CDSS development framework.
Before the ReKA method, the validity of the knowledge model relied on domain experts.They were free to refine the decision paths in the final knowledge model.This freedom in refinement leads toward a set of inconsistencies which were ignored by the previous method.The detailed evaluation shows that the introduction of formal verification has significantly contributed to revealing hidden inconsistencies in earlier proposed hybrid knowledge acquisition method.In the presence of these inconsistencies, the knowledge model evolution is not always guaranteed to be valid.The ReKA method, as a result of the verification, can identify the main cause of the inconsistencies and guaranteed always producing the valid final knowledge model.This paper addresses following research questions: a) Does introduction of formal verification using Z notation is able to identify the inconsistencies in the developed knowledge acquisition method with respect to standard knowledge resources such as CPG?; b) Does formal verification ensures that knowledge acquisition methods will always maintain the quality of the knowledge?; c) Does propose formal verification is able to prevent inconsistencies occurred due to complexity and freestyle usage of refinement in the knowledge?; d) Is the knowledge model created using ReKA comparable with existing hybrid knowledge models in terms of validity, quality, and integration with workflows?
The main contribution of this work is as follow: • The proofs of the theorem using Z notations provides a comprehensive explanation for checking the consistency of the knowledge acquisition method.These proofs enable detection of hidden inconsistency in the acquisition method (hybrid knowledge acquisition) and provide with an additional set of nine criteria to ensure that the enhanced method (ReKA) always produces a valid knowledge model.
• The formal verification activities are streamlined into a concrete set of processes which align to various artifacts of Z notation.
• Various aspects of Z notation exploited for the knowledge modeling and associated processes are expressed as the inferenceable mathematical models.
• The ReKA method is formally proved approach which always produces valid knowledge, reflects the CPGs as global evidence and encourage the recommendations well supported by local evidence.At the same time, it is revealed that the model created using ReKA is outperformed compared to the similar approaches available.

Overview of knowledge acquisition for Smart CDSS
In our earlier work, we proposed a novel hybrid knowledge acquisition method for Smart CDSS [17].
The acquisition method was accompanied by the proper validation process to ensure the validity of the final knowledge model.In this paper, we are introducing a formal verification for the hybrid knowledge acquisition method, which results in an enhanced method -ReKA.Before going into details of formal verification, it is worthwhile to introduce the knowledge models and validation processes briefly of the hybrid knowledge acquisition method.We encourage the readers to read [17] for detailed descriptions of the models and validation processes used in the hybrid knowledge acquisition method.

Hybrid knowledge acquisition approach for Guideline enabled data-driven knowledge model
In the clinical domain, patient data and CPGs are the most common sources of knowledge for CDSS.
Most of the existing knowledge acquisition methods use both sources of knowledge independently.From patient data, the knowledge models are created using machine learning, while from CPGs, various cognitive methods of knowledge acquisitions apply to the knowledge models.Both methods have potential pros; however, there exist some limitations for each of them.The knowledge acquisition method which combines both approaches can overcome somehow those limitations.The key limitations of data-driven knowledge acquisition methods using machine learning are as follows: • The quality of the knowledge model depends on the quality of patient dataset.So the performance of the model (such as accuracy) may vary for the same domain with different datasets.
• The model validation relies on the statistical validation process (e.g., 10-fold cross-validation).In this case, the validation purely depends on data; and the domain experts are unable to assert any additional criteria to apply constraints on the final knowledge model.
• The final knowledge model supports only local evidence as it derives from patient data.The recommendation becomes trustworthy for another organization, if standard evidence from CPGs and other published studies also associate with the knowledge model.
The use of CPGs as a knowledge source somehow resolves the inherent problems with the data-driven approach.CPGs covers population-based knowledge supported by standard clinical evidence gathered from different clinical studies.Although it covers-up some cons of the data-driven approach, however, the knowledge models derived from CPGs also come with limitations: • CPGs are generic, and the model representing CPGs may not be able to integrate into health-care work-flows directly.
• The knowledge model strictly following CPGs discourage the local practices.In most cases, it is possible that local practices may not fully conform and contradict to CPGs, but may have a huge impact on patient care at that particular jurisdiction.
Very few studies include CPGs and patient data as a combined source for hybrid knowledge modeling.
For example, Toussi et al [22] used model derived from patient data to complete the missing decisions in the CPGs.However, the primary motivation of hybrid knowledge acquistion method is to combine the data-driven knowledge acquisition method and CPGs based knowledge acquisition method to dilute their cons and take advantages of their pros in terms of the refined knowledge model.This knowledge acquisition method is adopted under the umbrella of the three-phase iteration process model of creating an executable knowledge model for Smart CDSS [17] in the cancer domain.The first two phases of the process model dedicated to knowledge acquisition, which covers knowledge model creations from CPGs and patient data, and the validation process.The third phase concentrates on the executable knowledge model and development of associated toolset [23].Figure 2 depicts the abstract representation of hybrid knowledge acquisition method, and the next section provides a brief description of the core knowledge models and validation process of this approach.

Knowledge models and validation mechanism
Hybrid knowledge acquisition method includes a set of tasks encompassing two phases of the iterative three phase model [17].In this section, we briefly describe this method of explaining the knowledge models and the process associated with the validation of the models (see Figure 2).The outcome of this method is the final knowledge model -known as a refined clinical knowledge model (R-CKM) which obtained after the rigorous validation process.It consumes the knowledge models created from CPGs -known as a clinical knowledge model (CKM) and prediction model (PM) created from patient data.
Prediction Model: A PM is a decision tree obtained from patient data using decision tree algorithms.
The decision tree algorithm used for this study was CHAID [24], which was selected based on rigorous selection criteria.PM creation involves the formal machine learning method -CHAID and reflects the local practices from patient data.As a decision tree formalism of the machine learning paradigm -it includes the root node and grows in a top-down fashion.The nodes represent conditions and leaf nodes as conclusions.The conclusion always lies at the leaf node where the branch selection at each condition uses proper statistical evaluation processes to proceed for the appropriate decision path.Finally, performance (such as accuracy) for each decision path evaluates from patient data, and its overall performance represents as mean accuracy of all the decision paths in PM.

Clinical Knowledge Model:
A CKM is a formal decision tree created from CPGs after a rigorous inspection process by a team of physicians.It follows decision tree formalism started with a root node.The tree grows in a top-down fashion from the root node by adding subsequent nodes to make a decision path.
The nodes represent a decision node and a conclusion node.Decision node represents condition(s) (such as patient symptoms) to select the next branch of the tree among decision paths.Conclusion node reflects the recommendations (such as treatment plan).In CKM, the conclusion node can also play a role of condition node for the next follow-up conclusion.For example, an initial treatment plan for cancer patient may be surgery, and after follow-up, the secondary treatment plan can be radiotherapy only if surgery already is done.In this context, unlike the decision tree formalism of PM, the conclusion node may appear as an intermediate node in the CKM decision tree.Moreover, the branch selection of the CKM decision tree does not follow any probabilistic evaluation of the condition.Because CKM is a reference model of CPGs, so its performance evaluation against local patient data is not required.

Refined Clinical Knowledge Model:
A R-CKM obtained after a rigorous validation process by combining PM and CKM.It follows the same formalism as of CKM.However, it also reflects some of the properties of PM to encourage decision making from local practices.Unlike CKM, all decision paths in R-CKM evaluated from local patient data, and it also requires evidence for decision paths which are refined but have no direct conformance to the CKM (i.e., guidelines).
Validation Process: A validation process is the core of the hybrid knowledge acquisition method which unifies two different models to a single refined knowledge model.Figure 3 depicts detailed steps of the process.It consumes PM and CKM as an input model and produces R-CKM as an output model.Each decision path in PM is selected and added to the decision path of R-CKM after passing conformance criteria based on CKM.The PM decision path may be refined by domain expert if required.The activities for the validation process briefly summarized in three steps: 1. Setting validation criteria: Domain experts define criteria based on CKM (guidelines) and other evidence to be fulfilled by decision path in PM.At the same time, each criterion is classified as primary (compulsory) or non-primary (optional with an alternate), and the order of checking specifies by priority.In the case of an oral cavity cancer treatment plan, domain experts decided two primary and two non-primary criteria.i) The minimum performance limit must be satisfied by each selected decision path in PM (e.g., accuracy of 50% in this study), ii) the selected decision path in PM must not conflict with the CKM (guidelines), iii) the decision path in PM should conform to any decision path in CKM, and iv) if criterion iii) is not fulfilled, then the decision path in PM must be associated with an evidence which proves its necessity and effectiveness of inclusion into R-CKM.
2. PM validation against criteria: During this step, each decision path is selected and evaluated against the well-established criteria.The decision path of PM becomes part of R-CKM if it fulfills the criteria.
3. Inspection and refinement of selected PM decision path: The selected decision path can become directly part of R-CKM.However, the domain expert may want to refine it further to reflect the most concrete concepts used in the healthcare workflows.Moreover, the refinement process also allows adding further choices of the treatment plan in the decision path if required.
The hybrid knowlede acquisition method was used in the creation of the knowledge model for Smart CDSS in oral cavity cancer [17,23] with proper validation mechanism.However, it was not formally verified even after using as a core method of knowledge acquisition for Smart CDSS.In the validation process at the  [17] refinement step, the process provides the freedom to domain expert for adding further treatment plans as a condition to the selected decision path.It leads toward inconsistency and can not guarantee the validity of R-CKM at all time.In order to cope with this issue, the content of this work introduces the verification process using formal Z notations.After applying the Z formalism, the outcome concludes with the detection of inconsistencies in the step of refinement of the validation process.It explicitly enlisted nine additional criteria that must be in place after refinements are made to the decision path in R-CKM.To accommodate the newly discovered criteria the existing knowledge acquisition method is enhanced as ReKA method -an outcome of the formal verification process.

Preliminaries
There are several ways to represent objects in the Z notation.Declaration, abbreviation, and axiomatic definitions are simple ways to represent objects in Z notation."Schema" and "free" types are special ways to represent complex objects in Z notation.All of these types obey mathematical laws and have rules for reasoning with the information that they contain.At this point, the introduction and use of these concepts are important; however, in this paper, we skip the detailed description of the concepts used in Z notation.
So, the important concepts introduced with brief details and all other concepts used in this paper provided in Figure 4. Readers may consult reference materials [25,26] and other research works that have used Z extensively [27][28][29].
• Declaration: This is the simplest way to define an object.When an object is a set or some basic type, brackets use to enclose the name of an object.If there are more than one objects, comma uses for separation between them.For example, type definition (1) in Figure 5  • Abbreviation: Abbreviation introduces another name to an existing object.For example, type definition (2) in Figure 5 is the abbreviation for cancer treatments.
• Free type: Free type allows a variety of data structures to be represented using sets with explicit structuring information.For example, type definition (3) in Figures 5 highlights three different object definitions.ConditionOperator is a free type that distinctly represents the set of operators used in the Condition.The Condition further expresses the complex definition of the conditions used in the clinical rules.treatmentSet is a free type that covers high-level semantics for cancer treatments that provided to a patient in a proper sequence by using the guidelines.
• Axiom: Axiom provides the ability to define objects and includes constraints upon it.In an axiomatic definition, the object definition represents in two compartments: declarations and predicates.
Declarations represent the content structure of an object and predicates introduce constraints on the contents.Figure 6 shows an example of axiomatic definition for CKM specification.
• Schema: Schema is the most powerful artifact in Z notation and describes the system behavior.
Similar to an axiom, it defines objects using declarations and predicates.However, the schema  can take different forms such as a modeling static structure, modeling operations, and modeling different states of the object after operations.An example for modeling CKM is shown as the schema "ClinicalKnowledgeModel" in Figure 6.

Motivation of Formal Methods
The ability of domain experts to trust knowledge content is a key factor that influences the success of CDSS implementation.The trust in knowledge primarily depends on how well the knowledge contents have passed through a sophisticated validation process to ensure consistencies in the refined knowledge model.According to a systematic review by Mor Peleg [2], formal verification techniques are used to validate the clinical knowledge for internal consistencies and to check for the fulfillment of the desired properties and specifications.There are two broad categories of these techniques: model checking and theorem proving [2].In model checking, the knowledge transformed into an appropriate model-checker format, and the model checker verifies the consistency of the knowledge model for the fulfillment of the desired properties.Alessio Bottrighi et al. applied the model checking approach to integrating the computerized guideline management system with a model checker [30].The guideline representation language GLARE is used and integrated with the SPIN model checker to verify the clinical guidelines.
Theorem proving uses the logical derivation of theorems in order to prove the consistency of the knowledge contents available in the formal specification.Annette T. Teije et al. [31] used KIV-based formalism to represent medical protocols and defined semantics of the desired properties.The desired properties of the protocol are verified using formal proof of the KIV theorems.Based on the substantial advantages and the need for formalism in knowledge validation and verification, we introduced the formal verification process as a formal method content into the development framework of Smart CDSS.Selection of an appropriate formal method requires formal guidelines to find the best fit for a knowledge representation scheme.In this work, we used the Z notation as the formal representation language for knowledge representation and for modeling the validation method features.We used the formal theorem proving mechanism to remove inconsistencies in the method, which ultimately ensures a consistent and valid knowledge model.
Following are fundamental features of Z notation which compels its suitability for clinical knowledge modeling and verification of the acquisition process.
1. Easy knowledge modeling: Using the Z notation, it is simple to decompose the knowledge specifications into small pieces and formally define the static and dynamic aspects of the knowledge acquisition (i.e., the knowledge representation and validation process [25]).The "Schema" represents this aspect of Z notation, where the first order predicate logic uses for the constraints on the typed knowledge contents.
Moreover, dynamic schema represents the validation process that operates within the boundaries of the knowledge representation schema.The subsequent sections will elaborate, detailed contents of the formal verification process for the knowledge acquisition method in terms of Z specifications.2. Data-rich formalism: Another aspect of Z notation is the notion of "types" [26].Z types are mathematical data types that can be used to represent any object in a system uniquely.They specifically obey a rich collection of mathematical laws, which make it possible to determine the behavior of the system [25,26].This aspect of Z leverage, towards data-rich formalism of knowledge contents and the resulting artifacts, can be easily mapped to standard viewpoints of RM-ODP [32] (e.g., the information viewpoint).H. Bowman et al. used Z notation for consistency checking of the two views in the information viewpoint [33].Similarly, artifacts of Z notation can also map to the "analysis" and "design" disciplines of the RUP framework.
3. Support of tools: The Z specification language not only enables formal specifications for a system and a language but also allows for the systematic reduction of such specifications into implementations [27].
Moreover, there is a wide range of tools available to check for syntax and type consistency in the specifications.

Refined Knowledge Acquisition (ReKA) method
ReKA method is an enhancement of our hybrid knowledge acquisition method.It follows the same three-phase model used for hybrid knowledge acquisition method.It uses all the steps of hybrid knowledge acquisition described in section 2.1.Besides, it introduces new processes that involve the formal verification artifacts at different phases of the three-phase model.Figure 7 shows the extended three-phase model used by ReKA method.The extended processes are reflected as an additional layer on the basic processes.as not valid because it creates inconsistencies in the final knowledge model.Furthermore, we have also compared ReKA approach with the most relevant hybrid model acquisition approach.The subsequent sections further discusses the technical details of the formal verification processes used in the ReKA method.

Establishing Formal modeling process
To the best of our knowledge, no substantial evidence exists in a knoweldge engineering discipline that discusses Z notation with discrete processes having proper guidance.Based on the capabilities of Z notation and the guidance available for applying different concepts of Z notation to formal modeling [25,26], we formulate a formal modeling process for knowledge acquisition method.This is comprised of four distinct processes: "modeling problem", "defining function and model states", "proving consistency", and "refine specification for concrete design".Below is a brief discussion of each of these processes.The outcomes of this process produce primitive types, free types, sets of axioms, and sets of the static schema, which represents the knowledge models.
2. Defining functions and models state: This includes tasks to define the behavioral aspects of the system under consideration.Defining operations related to the candidate models and associating the appropriate state model (as a consequence of the operation on the model) are the main activities of this process.Schemas are the central construct in Z and can represent the operations and states of the models.For the knowledge acquisition method, operations are defined for the retrieval of contents from PM, and CKM models.These operations will not affect changing the state of the corresponding models.Different operations define for the R-CKM model in order to validate the candidate decision path from PM against the CKM model and to evolve the final R-CKM model.As a result of the evolution of the R-CKM model, the corresponding state model is defined to formally represent possible changes in the contents of the R-CKM model.

Proving consistency:
Identifying inconsistencies in the specifications of the modeled problem is the ultimate goal of formal methods.The main task is to make sure that the defined models are consistent and have no contradictions with their desired requirements.Moreover, it is desirable to verify that the operations defined in various models are consistent and that their outcomes are within the intended boundaries of the domain.Z specification provides a well-established way to achieve both of these goals.The first part achieves, to prove the constraint part of the state schema of the model is satisfiable using "initialization theorem" -to indicate that an initial state, at least, exists.The second part requires to investigate "preconditions" for the candidate operations -that may be calculated from the operational schema using the one-point rule.
For the knowledge acquisition method, the "initialization theorem" proves the satisfiability of the R-CKM state schema.Moreover, "preconditions" investigates for all operations that evolve the R-CKM model.4. Refining specification for concrete design: The refinement process tends to construct and describe another model that complies with the original model of the design but is closer to implementation.The refinement process comprises large tasks that are applied in consecutive iterations at the data and function levels to ensure that the specifications are free of any uncertainty.These specifications are closer than previously modeled specification to an executable program code.In order to prove that refinements are consistent within themselves and appropriately represent the original design model, it is necessary to establish a theory for refinement that includes a set of rules for proving the correctness.

Preprints
In this research work, we exploit the first three processes to model the clinical knowledge and the validation process in order to prove that the knowledge acquisition is sufficiently consistent to produce always valid refined knowledge model.The refinement process is helpful for systems where the outcomes of the design are required to be sufficiently close for direct conversion into executable code.This process is included purposefully because our knowledge specification can be easily converted into executable code if we properly exploit the Z refinement mechanism.Furthermore, we are presenting the "Proving consistency" step in the results section to emphasize the outcome of the formal verification process.

Modeling problem
The Treatments == {CancerTreatment} In order to define the formal representation of the knowledge model, primitive types are needed to capture the basic concepts used in the knowledge representation scheme.In Smart CDSS, the knowledge models follow decision tree representations where the combination of conditions with logical relationships constitutes the decision path.The Condition includes clinical concepts as an attribute with an exact value or a range of value sets.For example, a condition in the decision tree test node TreatmentIntent = radical represents a patient categorization primarily based on the severity of cancer.Z primitive types (shown in Type Definition 1 (line 2) represents these concepts, and Type Definition 2 provides the corresponding language syntax for the condition.
Type Definition 2 BNF for some primitive types Moreover, free types in Smart CDSS reflects the semantics of the clinical concepts and provides conformance to decision tree representation formalism.For example, treatments provided to patients In Smart CDSS, the knowledge model typically uses decision tree representation; however, PM is different from CKM and R-CKM in terms of the decision path.PM does not include treatments as a condition.To distinctly represent this formalism, ConditionCKs (line 3) defines a particular condition as a free type for CKM and R-CKM.Similarly, RefinedTreatmentPlan (line 4) represents refinement in final R-CKM, which dictates the addition of a treatment to R-CKM as a type of refinement (indicating the placement of treatment plan at a particular position in the decision path).
Type Definition 3 Free types to capture semantics of knowledge artifacts ConditionKMs ::= seq Condition TreatmentPlan (3)

Knowledge models
Clinical knowledge models, such as PM, CKM, and R-CKM, are represented as axioms and schemas.
Subsequent sections explain the specifications for these models.
Prediction model specifications: Prediction model specifications cover the properties associated with PM by decision tree formalism.Figure 9 shows the PM created for oral cavity cancer treatment intervention [17] with details of corresponding attributes and their formalism semantics.The PM specifications are created using an axiom (Axiom 1) and the PredictionModel schema (Schema 1).The axiomatic definition for PM represents the basic constructs of PM using decision tree formalism.Accordingly, the decision paths are the main constituents of the decision tree skeleton where a combination of logically-related conditions makes a single decision path that has one conclusion.The conditions and conclusion are also known as nodes of the decision tree where the conclusion is always a leaf node.The decision tree obtained from the data (using machine-learning approaches) also has accuracy in terms of possessing correctly classified data cases (i.e., using 10-fold cross-validation).
In Smart CDSS, PM follows decision tree formalism, which is obtained from patient medical records where conditions are used to represent patient information (e.g., symptoms, problems (diseases), clinical observations, and other demographic information (patient history)) and the conclusion represents the treatment plan.Axiom 1 includes declarations for the decision path as a partial function from the condition to the treatment plan (line 3).Its accuracy represented by a total function from the decision path to the accuracy (line 5).The decision path conditions are represented as a finite set of the Condition (line 1), and the conclusion represented by a finite set of the TreatmentPlan (line 2).In order to reinforce the basic properties of the PM decision path, predicates are used to constrain the defined properties.For example, the PM decision path accuracy must lie between 0 and 100 (line 8).For all decision paths, there must exist one conclusion, and the conclusion must be a TreatmentPlan (line 11).
Moreover, for validation purposes, we also associate the evidence (if it exists) with the treatment plan recommendation that is provided by the decision path in PM.The evidence is a finite set (line 6), which can represent the effectiveness of the treatment plan in given patient cases in terms of the success rate (as a percentage).It may also include external evidence from other research works.Therefore, the decision path may have evidence represented by a partial function from the decision path to the set of evidence (line 7 and line 12).
Prediction model specification is further extended through the PredictionModel schema (Schema 1).PM is formally represented as a decision tree that is associated with the clinical objectives using the injective function from the decision path to the clinical objectives (lines 1, 3, and 7).The PM is associated with accuracy, which is the weighted mean accuracy of all of the decision paths in PM (lines 2, 4, and 8).For simplicity, we consider an equal number of patient cases for each decision path; this simplifies the accuracy of PM (line 8).In addition, PM is a decision tree, which means it must include one root node that must be a condition (lines 5 and 9).In CKM, the treatment plan comes as a condition in one decision path and may act as a conclusion for another decision path.In other words, the CKM conclusion may occur in an intermediate node.(5) Axiom 2 Clinical knowledge model specifications (3) R-CKM follows the formalism of CKM in that it also uses decision tree representation, which includes decision paths that have been formally validated from standard guidelines or possess sufficient evidence to prove their effectiveness.Figure 11 shows the R-CKM of a treatment plan for oral cavity cancer [17] with precise semantics and formalism.In this respect, the R-CKM decision path modeled (similar to CKM) by a partial function from free type ConditionKMs to the treatment plan; this is shown in the axiomatic definition (line 3).
As a result of refinements, the decision path of R-CKM may fully not conform to guidelines (CKM).In such cases, the evidence is required to justify the effectiveness of the refinements made to the decision path of R-CKM.To capture this context, a finite set of Evidences (line 6) is associated with the decision path of R-CKM as a partial function (line 7,15).
The predicates defined in Axiom 3 (lines 13, 14) capture the semantics of the decision path in R-CKM; a treatment plan can be a condition in the decision path, and the decision path must start with a condition (this should not be a treatment plan).
In addition to CKM formalism, decision paths in R-CKM become a part of the model after passing through formal validation process and refinements (Figure 3).In this respect, the decision path in R-CKM Moreover, R-CKM is derived from PM and validated against CKM (guidelines); thus, the total function defines from the R-CKM decision paths to the intended CKM (line 4), and R-CKM modeled by a finite set of related decision paths (line 3) associated with CKM (line 9).Furthermore, a predicate adds to the schema (line 13), which constrains all of the decision paths; these must be derived from PM and aligned to CKM.
Similarly, using schema inclusion, PredictionModel (Schema also included (lines 1 and 2) into the RefinedClinicalKnowledgeModel (Schema 3) in order to make the contents of PM and CKM available to the R-CKM model.

Validation process specifications:
Validation process specifications encompass the validation process (Figure 3) and properly represent the validation criteria defined for final knowledge model -R-CKM (See step 1: Setting validation criteria at Section 2.2).The schema PMPathValidation (Schema 4) models the basic semantics of the validation process.It includes schema RefinedClinicalKnowledgeModel (line 1), which is used to associate the validation process with R-CKM.It also provides a declaration for the two inputs that the validation process is supposed to consume: the PM decision path (line 2) and the minimal accuracy (assigned by a domain expert and acceptable for R-CKM) that requires for the PM decision path (line 3).
The validation criteria defined in the validation process of the knowledge acquisition method reflected by predicates in the schema PMPathValidation (lines 4-7).The first two primaries (compulsory) criteria defined in the schema by conjunction predicates (lines 4 and 5) and two other criteria represented by disjunction predicates (lines 6 and 7).

Defining functions and state models
The main functions of knowledge models are to evolve R-CKM based on the validation of the decision path.The only evolving model is R-CKM, so the state model for R-CKM is presented.

Operations on knowledge models
Two types of operations defined for the knowledge model.For PM and CKM, only retrieval operations are required to represent access to different components of the model.So for as R-CKM is concerned, it requires specifications for both retrieval and state change operations.
Operations for PM and CKM: PM and CKM specification provide a set of operational schema related to retrieval of various components of the PM and CKM respectively.For the brevity purpose, we concentrate on operational schema related to the evolution of the knowledge model.Retrieval schema for the PM and CKM are straight forward, and we shall not discuss it further.
Operations for R-CKM: R-CKM is the only knowledge model that evolves through proper validation processes using PM and CKM.Therefore, in addition to retrieval operations, R-CKM also requires definitions (ran(dom(dp pm ?)) ⊆ ran(dom(dp ckm )) ⇒ ran(ran(dp pm ?)) ⊆ (ran(dom(dp ckm )) ∩ ran(ConclusionCKM)) ∪ ran(ran(dp ckm ))) (7) for operations that represent the addition of new decision paths into the final model (in the presence of the validation criteria).For brevity purposes, we only concentrate on operations that are related to the evolution of R-CKM.
EvolveRCKM (Schema 5) is an operational schema that mainly represents the evolution of the R-CKM model.The evolution of R-CKM mainly describes as a two-step process after setting the validation criteria: (1) a decision path from PM is evaluated against the validation criteria and (2) the selected decision path is refined further (if needed) and added to the R-CKM.Accordingly, EvolveRCKM (Schema 5) is defined as a composite operational schema to reflect these steps.This composition is modeled as the combination of two schemas: PMPathValidation (Schema 4) and AddPathRCKM (Schema 6).To get a clear picture of this process, Figure 12 demonstrates three paths of the PM(Figure 9) in the context of the validation process(Figure 3) and produce the R-CKM(Figure 11).The two paths (path 1 and path 2) are fulfilling the first two compulsory criteria (having a minimum threshold of accuracy without any conflicts with CKM) and passing the criteria regarding conformance to CKM (Figure 10).Path 3 fulfilling the compulsory criteria; however, it goes for alternate criterion "Evidence" because of the suggested treatment plan does not conform to CKM.In the refinement step, path 2 and path 5 are refined to path 2.1, 2.2 and path 5 respectively.So for as path 1 is used without any refinements.• Declaration(Output): The final decision path of R-CKM, after refinements, is considered to be an output for the schema AddPathRCKM (line 4).
• Predicates(Pre-conditions): These include a set of predicates (lines 5-12) that must be met before any changes are made to the R-CKM model (Schema 3:RefinedClinicalKnowledgeModel).Most of these pre-conditions are not known in advance but are calculated using the one-point rule and simplification proofs.We shall describe some important pre-conditions, as evaluation results, for the formal verification process in Section 5.
• New refinements introduced to a set of the R-CKM model, which results in the addition of a PM path with the associated refinements (Schema 6: Lines 18 and 19).These states reflected in lines 9, 10, and 21 in Schema 7.
• With the new decision path, the R-CKM model evolved for a new conclusion (Schema 6: Line 20), which yields new states in the model properties of RCKMConclusion, as indicated in the state model schema at lines 4 and 17.
• For the new R-CKM path, the accuracy of the path will be associated, and the overall R-CKM accuracy is recalculated (Schema 6: Lines 21, 22, and 23).The resulting state changes reflected at lines 6, 14, 15, 22, and 26 in the state model schema.
• Evidence of the PM's decision path associated with the refined decision path in R-CKM (Schema 6: Lines 24 and 25).These changes reflected in lines 7, 8, and 20 in the state model schema.

Results and evaluation
This section explains the evaluation of the proposed work using two perspectives.First, it demonstrates the theorem proving mechanism to show inconsistencies in the hybrid knowledge acquisition method before formal verification.The outcome of the formal verification is presented as an enhanced knowledge acquisition method -as ReKA method.We evaluate the enhanced method (in the context of formal verification) against our initial approach and describes its discrepancies using real clinical scenarios.Second, we compare our enhanced approach with one of the existing relevant approaches developed by Tossie et al. [22].The Initialization theorem provides a mechanism to prove that the model (R-CKM) is consistent and fulfills the requirements.It determines the model has at least an initial state.Definition 1 defines the initialization theorem.
Definition 1: For the system state "State" and its initial state "StateInit", the initialization theorem takes the following form: For the R-CKM model represented in the schema RefinedClinicalKnowledgeModel (Schema 3), the initial state is defined using the state schema InitRCKM (Schema 8).
For the given initial state InitRCKM of the R-CKM model's schema RefinedClinicalKnowledgeModel, the initialization theorem is represented by Theorem 1; this is inspired from the basic definition provided in Definition 1.
Theorem 1 Initialization theorem for initial state of R-CKM The proof of this initialization theorem leads to consistent specifications for the R-CKM model.It is almost impossible to prove the initial state of the modeling specifications, which include contradictions.
Hence, it can conclude that the model does not fulfill the desired requirements.
In order to prove the initialization theorem, we can take advantage of the one-point rule as well as some other set theory laws and fundamental definitions.The one-point rule helps replacing the existential quantifier when the bound variable has an identity within the boundaries of the quantification expression.
For the one-point rule, Definition 2 provides the essential background related to replacing the existential quantifier.
Following the definition of the one-point rule, and other fundamental laws and definitions, the proof of initialization theorem is given in Proof 1.The proof is straightforward, and each step explaines with instructive definitions.

R-CKM evolution consistency proof using simplified pre-conditions and proving the property composition
The pre-conditions of an operational schema represent a set of states, for which the outcome of the operations is properly defined.The pre-condition of operation is another schema, obtained from a given operation, that hides components related to the state after the operation and provides an output that results from an operation.
The one-point rule gives the following equivalence for the given existential quantifier.
(∃ x : a Definition 2: The one-point rule Definition 3: For the operational schema "operation," the state of the system modeled by "state" and the "output" is the list of outputs associated with the operation.Then, the following is equation represents the pre-condition of the schema.
pre operation = ∃ state • operation Definition 3: Precondition of an operation We establish a theorem (Theorem 2), which is based on the basic definition of the pre-condition schema (Definition 3), to calculate the pre-conditions for the operational schema AddPathRCKM (Schema 6).
Calculation of pre-condition requires simplification of predicate part of the theorem (Theorem 2) which involves expansion of all schemas.After the expansion of all possible schemas, the one-point rule plays a pivotal role in simplifying and proving the primed statements in the schema.Due to space limits, the proof is provided as a supplementary appendix.The Supplementary A explains the proof with instructive definitions at each evolving step of the schema.For brevity purposes, the proof does not discuss the pre-condition calculation in detail; however, we believe that the given explanation is sufficient to determine the pre-conditions for the AddPathRCKM operational schema.
Although the simplification process seems quite complicated in terms of resolving all of the primed statements, however using set theory fundamental laws and the one-point rule, it becomes straightforward.
Additionally, it is interesting because it reveals new pre-condition predicates that were not known in advance.The next section provides details evaluation of the newly discovered pre-condition which gives birth to an enhanced ReKA method.The primed predicates in Proof 2 (at Supplementary A) are underlined (numbered 1-13).The prime predicates requires simplifications to conclude the proof.To save the space, the Supplementary B presents the simplification proofs.

Evaluation: Comparative analysis of ReKA and hybrid knowledge acquisition method
As a consequence of "Proving consistency" mechanism the main problem regarding inconsistencies identified in the step-3 (selection and refinement of the selected PM decision path) of the validation process in the hybrid knowledge acquisition method.The inconsistencies cover-up by introducing nine additional criteria (see Table 1) to place after refinement.As an outcome of the formal verification, the enhanced ReKA method introduces to accommodate the newly discovered criteria.The ReKA criteria cover the broad categories of inconsistencies defined below.Each criterion contributes to one or more categories of inconsistencies.• Root of the R-CKM remains the same for any decision path when R-CKM already has some decision paths.
• Root of the R-CKM will be the first condition for the decision path when R-CKM has no decision path.
pos ≤ (#(dom dp pm ?) + #(ran dp pm ?)) • Example: Treatment refinements in the root of the decision path are not conformed.• Conditions in the refined decision path must come from the defined condition set of the R-CKM.

Continued on next page
ran decisionPathConditionRCKM • Example: Conditions outside the condition set make R-CKM non-integrable to HIS workflows.

ran(ran rckmPath!) ⊂ ran ConclusionRCKM
• Conclusion in the refined decision path must be within the scope of the defined treatments.
• Example: Conclusions for the treatment plan must be valid cancer treatment.

5.
(ran(ran rckmPath!)∩ ran decisionPathConditionRCKM) • Conclusion of the refined path may be the condition of another decision path in R-CKM.

⊂ ran decisionPathConditionRCKM
• Example: The refined path may be an extension of an existing decision path.
• Example: The refined decision path should be tested for the set of patient data.

7.
head (dom rckmPath!)/ ∈ ran ConclusionRCKM∩ • The first condition in the refined decision path must not be a treatment plan.
ran decisionPathConditionRCKM • Example: A treatment plan is given based on some available symptoms (conditions).
• Example: The refined path is obtained from PM and refined after confirmation from CKM.
1. Category-1 (Violating formalism of the R-CKM): These inconstancies occur due to bypassing the construction norms of the R-CKM.This category of inconsistency makes the final model invalid in terms of affecting outcomes of other decision paths.Criteria 1, 2, and 7 ensures to avoid inconsistencies related to R-CKM formalism.

Category-2 (Violating conformance to guidelines (CKM)):
The refinements which produce a decision path that does not conform to clinical guidelines without associating any additional significant evidences.
These criteria were in place during the initial steps of the acquisition process (in the hybrid knowledge acquisition method); however, it was unavailable to ensure the conformance after refinements.Criteria 5 and 9 explicitly discuss that each refined path must conform to CKM.

Criteria violation:
Category: 2 Specific Criteria: 9 A rationale for violation: The guidelines (CKM) recommends follow-up after RT for palliative patients.

Criteria violation:
Category: 1, 5 Specific Criteria: 1, 2, 7 A rationale for violation: The treatment path after modification represents valid guideline-based treatment.However, it violates the formalism of the decision tree because modification in root node affects other decision paths.A rationale for violation: Although the refinement 1 and refinement 2 conform to the guidelines (CKM), they are invalid due to the following reasons: Category-4: The hospital healthcare information system (HIS) at current stage maintain data for essential adverse histopathologic risk factors such as extracapsular spread (ECS).It keeps all other related factors such as perineural invasion (PNI) and lymphovascular invasion (LVI) in a broad category of others-histopathologic risks.Hence introducing PNI as a discrete data item raises an issue of the direct integration of the decision path to HIS workflows.Furthermore, the HIS records the risk of ECS to the patient with the radical status of treatment intent.So based on the above discussion; refinement#1 is not valid knowledge within the scope of local practices, and refinement#2 is also invalid as it introduces unknown data items.Category-3: Violation of criteria 3 in category-4 leads to the undefined quality of final RCKM.The decision path acquired from refinement#1 and refinement#2 cannot evaluate against patient data because of the unavailability of the missing data for ECS and PNI risk factor respectively.

Criteria violation:
Category 4: Specific Criteria: 4 Category: 3 Specific Criteria: 6 Category: 2 Specific Criteria: 9 and 5 A rationale for violation: The Salvage Surgery refers to surgical treatment which uses after the failure of initial treatment.It is recommended explicitly having residual disease exist after RT for neck metastasis.The refinement for decision path with extended Salvage Surgery is invalid due to the following reasons: Category-4 and Category-3: It creates an issue of direct integration to HIS workflows because the existing treatment plan cannot differentiate between standard surgery and salvage surgery.This specific violation in turns raises category-3 violation which has the same interpretation mentioned in Scenario 3. Category-2: The refinements do not conform to the given guideline (CKM) even after interpretation of salvage surgery to standard surgery.The main reason is the salvage surgery is applicable for patients who are categorized as radical by TreatmentIntent.

Category-5(Introducing inconsistencies due to complexities ):
This category is related to Category-1.
However, it further covers the inconsistencies that occur due to lack of availability of descriptions for the construction of R-CKM.Criteria 8 is the detailed formal description of how to refine the path in order to avoid any inconsistencies.Criteria 1, 2, and 7 of Category-1 also comes under this category.
The ReKA method can elaborate on the ambiguous steps in the validation process related to refinements.
To better understand the impact of the ReKA method, Table 2 discusses four refinement scenarios to decision path by domain expert( see Path-2 in Figure 12 ).Each of these scenarios introduces inconsistencies which relate to one or more categories.It is important to note that these refinements are valid according to hybrid knowledge acquisition method.• Evolution criteria setting : Any refinement in decision path suggested by domain expert must be evaluated against a set of evolution criteria (specified in Table 1) to avoid inconsistencies as mentioned earlier in the R-CKM.
• Criteria checking: All evolution criteria are compulsory and refined decision path in R-CKM must fulfill each criterion.Any refinement to decision path which is not fulfilling any of the nine criteria lists must not be considered, and domain expert is prompted for the violation and indicating non-valid evolution of the R-CKM model.
• Evolution of R-CKM: After passing the evolution criteria, the refined decision path becomes part of the R-CKM, and the process terminates faithfully.

Comparison with existing approach
One way of combining the traditional data-driven approach and guideline-based approach is to use PM as a source and transforming it into the final knowledge model R-CKM after rigorous validation process which is conformed from CKM -the guidelines.However, the combination of these approaches can be done in another fashion -considering CKM as a source and evolving it from PM with all newly missing decision path in the CKM.In this section, we will discuss one of the existing most relevant approaches [22], which lies in the second category and draw a comparative analysis with our approach.In order to know the detailed description, Figure 14 shows the high-level steps in both knowledge acquisition approaches.Both knowledge acquisition approaches are given the same CKM and PM as an input.The resulting outcomewe called an R-CKM model with both approaches is different.

Proposed
Existing Investigate decision path in PM which is not in CKM Extend CKM for missing decision path Investigate each decision path in PM and apply conformance Extend PM to R-CKM after conformance and refinements  "n" is not properly validated against CKM, so in R-CKM it is removed  "v" is not added to R-CKM while only "u" is added as refinements. Presence of "v" make R-CKM non-integrable as        Figure 14.Demonstration of existing approach vs proposed approach [22] The main limitations of the existing approach are highlighted in Figure 14, and a detailed discussion is provided in Table 3.In a nutshell, the existing approach tends to use the PM as a key source to refine the decision paths in the CKM while compromising the quality of the model (missing rigorous validation) and integration to existing healthcare workflows.We also applied the existing approach on the SKMCH data set of 1229 oral cavity cancer patients and used the CKM as a reference guideline model (derived for oral cavity NCCN guidelines).In the final knowledge model, we identified that 26% of the decision paths were violating the quality criteria of lower accuracy (in our case, it should be greater than 50%), 30% of the decision paths were not conformed to guidelines, and 9% decision paths were violating multiple criteria, i.e., lower accuracy and non-conformance.Overall, 47.8% decision paths were lacking to pass the validation criteria.Figure 15 shows the details of the decision tree C4.5 algorithm (which is referred by  Comparison of proposed approach with existing approach using SKMCH oral cavity cancer data [22] As described in Table 3, the final model obtained from Toussi et al. approach is not necessarily integrable to evaluate its performance against patient data available at a local organization.However, as shown in Figure 15, the source model for Toussi et al. approach is C4.5, which has overall accuracy on SKMCH dataset is 69.7%.Considering Toussi et al. approach produces the final knowledge integrable to existing healthcare workflow, there is a great chance that overall model accuracy will fall from 69.7% because of its generalization.In the case of proposed work, we have a rigorous selection process for choosing the appropriate machine learning algorithm and as indicated the CHAID decision tree is a candidate algorithm with an accuracy of 71% on a data set of 1229 oral cavity patients (see [17] for details of part of knowledge acquisition related to this part).Moreover, the final knowledge model -R-CKM performance is improved to 72.57%, as shown in Figure .To conclude, the proposed approach also gives greater performance over

Conclusion
This paper has introduced enhanced ReKA method as a result of the formal verification using Z notation.Z notation proves the consistency of the acquisition process and hence improve hybrid knowledge acquisition method.ReKA method is established after involving three steps formal verification process to represent the knowledge models formally, and associated validation process of hybrid knowledge acquisition using various artifacts of Z notation.Subsequently, the mechanism of theorem proving in formal verification has identified inconsistencies in the previously established knowledge acquisition by introducing nine additional criteria.These criteria address the broad categories of inconsistencies related to the formalism of knowledge, conformance to CPGs, quality of knowledge, and complexities of knowledge acquisition artifacts.(2.78) Definition 4: For any two functions f and g, the dom property for the union is defined as follows; Definition 10: The union (∪) of two functions is not always a function.However, ⊕ is the same as a union but ensures that combinations of the two functions are also a function.For two functions f and g, ⊕ is defined as follows: f ⊕ g = (dom g − f ) ∪ g For functions f and g, the dom property for ⊕ is defined as follows; dom(f ⊕ g) ⇔ dom f ⊕ dom g Definition 10: dom property over ⊕ Definition 11: Modus ponens, or implication elimination, is a simple argument form and rule inference in logic.For predicates p and q, the modus ponens can be formally represented as follows; p ⇒ q, q p

Figure 2 .
Figure 2. Hybrid knowledge acquisition method represents multiple object declarations.ConditionAttribute and ConditionValue are the set of concepts and the corresponding values, respectively, in the clinical knowledge model that construct the basic Condition.

Figure 9 .
Figure 9. PM for treatment intervention in oral cavity cancer and its formalism (C CRT: Chemo Induction followed by chemotherapy; C NOS: Carcinoma NOS; CT S: clinical stage S value; CT T: clinical stage T value; HistoDisc: Histology description; RT: radiotherapy; SCC: Small cell carcinoma; SqCC: Squamous cell carcinoma; S RT: Surgery followed by RT; TP Intent: Treatment Plan Intent)

)Figure 10 .
Figure 10.CKM for treatment intervention in oral cavity cancer(CI: Chemoinduction; CRT: Chemotherapy; CT N: clinical stage N value; CT S: clinical stage S value; CT T: clinical stage T value; ECS: Extracapsular spread; FU: Follow-up; MCT: Multidisciplinary consultation; RT: radiotherapy; S: Surgery) [17] has an accuracy represented by a total function from the decision path to the accuracy (line 5).Also, the accuracy of the decision path must be a finite value bounded interval [0,100] indicated in line(4,10).The refinement in R-CKM is represented by an injective function as shown in (line 9, 16) which maps the refined treatment plan (a free type, line 4, Type Definition 3) to the PM decision path (line 8).The declarations and predicates of schema RefinedClinicalKnowledgModel (Schema 3) are mostly similar to those of CKM (Schema 2); both share the same formalism.A total function (line 7) defines the new contents to support the overall accuracy of R-CKM.The intended accuracy calculated by the weighted mean accuracy for all of the decision paths in R-CKM (line 12).

• 3 ••Figure 12 .Schema 5
Figure 12.A running example of validation and refinement of three decision paths of PM

3. Category- 3 (
Compromising quality of R-CKM): These inconsistencies are related to the quality of R-CKM which are mainly instigating from the refinements to existing PM decision path without re-evaluation on patient data.Criteria 6 defines performance (such as accuracy) associated with each refined decision path after evaluating against existing patient data.4. Category-4(Introducing out-bounded refinements): This category discusses the inconsistencies in decision path that comes intentionally or unintentionally by introducing conditions or treatment plans which do not exist in the hospital information system or out of the scope of the healthcare provider.Criteria 3 and 4 dictates that a domain expert must include only appropriate conditions and treatments that exist within the boundary of the capacity of the healthcare provider.Preprints (www.preprints.org)| NOT PEER-REVIEWED | Posted: 19 June 2019 doi:10.20944/preprints201906.0179.v1

Figure 13 Figure 13 .
Figure13depicts the enhancements made in the knowledge acquisition method in the validation process.Figure.13ashows the initially proposed hybrid knowledge acquisition method with detailed steps of the validation process.The ReKA method, with the suggested improvements due to formal verification, is depicted in Figure.13b.The ReKA method enhances the hybrid acquisition method by introducing the following specific sub-steps:


No validation to extension of path in CKM  Final Knowledge Model is not integrable

Figure 15 .
Figure 15.Comparison of proposed approach with existing approach using SKMCH oral cavity cancer data[22]

•Figure 16 .
Figure 16.R-CKM results using oral cavity cancer data [R-CKM is implemented as set a of HL7 MLMs and the evaluation results drawn are based on the structure of MLMs]

Definition 4 :Definition 5 :Definition 6 :Definition 7 :Definition 8 :Definition 8 :Definition 9 :
dom over union For any two sets a and b, the set subtraction is formally defined as follows;a \ b = {x ∈ a | x / ∈ b} Definition 5: Set subtraction For any two functions f and g, the ran property for the union is defined as follows;ran(f ∪ g) ⇔ ran f ∪ ran gDefinition 6: ran over union Preprints (www.preprints.org)| NOT PEER-REVIEWED | Posted: 19 June 2019 doi:10.20944/preprints201906.0179.v1Proof 5 simplifies the primed statement in Proof 2 (line 2.44) using the one-point rule (line 5.02), the definition of range (line 5.03 using Definition 6), and other laws and principles of set theory, which are described in the following definitions.For any two sets a and b, the following property holds; a ∪ b = a ⇔ b ⊂ a Definition 7: Union Properties Set intersection is distributive over A set union.For sets r, s, and t, the set intersection distribution over a union set can be defined as follows; r ∩ (s ∪ t) = (r ∩ s) ∪ (r ∩ t) Set intersection distribution law over union For sets a, b, and c, the following definition holds; a ∪ b ⊂ c ⇒ (a ⊂ c ∧ b ⊂ c) Definition 9: Set union and proper subset Using the one-point rule (line 6.01) and definitions of basic set theory (lines 6.02 -6.04), Proof 6 concludes the primed statement in Proof 2 (line 2.45) into the R-CKM model (Axiom 3: line 14).Proof 7 concludes the primed statement in Proof 2 (line 2.46) into the R-CKM model (Axiom 3: line 15) using the one-point rule (line 7.02) and definitions of basic set theory (lines 7.03 -7.05).Using the one-point rule (line 8.02) and definitions of basic set theory (lines 8.03 -8.05), Proof 8 concludes the primed statement in Proof 2 (line 2.47) into the R-CKM model (Axiom 3: line 16).Proof 9 concludes the primed statement in Proof 2 (line 2.48) into the R-CKM model (Schema 3: line 8).This proof is straightforward and its conclusion is reached by using the one-point rule (line 9.02, 9.09) and solving the inequalities with fundamental mathematics.The proof is logically decomposed into two parts (lines 9.03-9.07and lines 9.08-9.11).Each part is proven separately and the final statement is concluded (line 9.12).The remaining proofs (Proof 10-Proof 15) use the same pattern of logical proofs to simplify the remaining primed statements of Proof 2 (line 2.49-line 2.54).Each step in the proofs is provided with instructive definitions, and necessary definitions are included where explanation is required.Preprints (www.preprints.org)| NOT PEER-REVIEWED | Posted: 19 June 2019 doi:10.20944/preprints201906.0179.v1

Hybrid knowledge acquisition method Validation Process Figure 3. Validation Process Select next     from PMFor each   in

Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 19 June 2019 doi:10.20944/preprints201906.0179.v1 follow
a sequence according to standard guidelines and protocols; ChemoInduction follows radiotherapy treatments and surgery for radical patients (from CKM).In order to capture these semantics, Type Definition 3 defines two free types: TreatmentSet and TreatmentPlan (line 1 and line 2, respectively).

Table 3 .
Detailed description of guideline-enabled data driven formally verified approach vs existing approach