The NGDOs Efficiency: A PROMETHEE Approach

1. Introduction

In recent years, the economic crises and the publication in the media of dubious practices by Non-Governmental Development Organizations (NGDOs) have put organizations dedicated to development cooperation in trouble. It is necessary to strengthen these non-profit organizations with donors and with society in general and, in addition, provide their managers with tools that allow them to identify the areas in which they must act to improve their results, making the most of their resources. Efficiency studies cover both needs: on the one hand, they make it possible to identify which entities are the best performers, legitimizing them in relation to the society and, on the other hand, they give organizations a vision of their operation and possible areas for improvement.

There are several studies dedicated to efficiency in NGDOs. However, the limitations that third sector entities present, mainly due to the absence of a benefit that serves as an indicator and the use of peculiar resources, such as free donor financing public and private or the use of volunteers, it results in not having objective and agreed items to measure efficiency. There is not even a doctrinal consensus on the very notion of efficiency in the field of Development Cooperation, nor on what are the outputs, the objectives that these entities pursue.

This paper aims to study the efficiency in the development cooperation sector, define the objectives and resources of the NGDOs and find general indicators that allow verifying to what extent the entities are achieving their goals and using their resources to be able to apply them to an efficiency study.

The efficiency indicators are inferred from those used by public donors when they select the NGDOs to receive funding, the so-called eligibility criteria published in different norms and laws. Thus, efficiency is measured in this paper using the indicators collected in the eligibility criteria of three of the main cooperation agencies, the Spanish Agency for International Development Cooperation (AECID), the European Agency (EuropeAid) and the Agency for the United States for International Development (USAID).

It is a novel method of fixing identifiers because, up to now, in the studies analyzed, they were established either by the authors themselves, or by means of surveys conducted with NGDO personnel or agency technicians. The fact that the indicators are obtained from donor regulations allows them to be more general and more objective than those traditionally used.

One of the first facts found in this study is the opacity surrounding the eligibility criteria. The different criteria, the indicators used to corroborate compliance, and the sources from which the donors extract the data to quantify the indicator are scattered across different standards without establishing a clear relationship between them. The weights are generally available for the criteria, without disaggregating the weight of each indicator.

Given this opacity and in order to know if the indicators published by donors were sufficient to determine the award of a public grant to a NGDO, a simulation of granting public funding was applied. For this, the 74 NGDOs that in 2016 (up to 2018-2023) belonged to the Spanish Coordinator were taken. With the donor variables, a study of the efficiency of the coordinator´s NGDOs is carried out, using a technique that is an adaptation of the Promethee multi-criteria method known as PROMETHEE Productivity Analysis (PPA) developed by Ishizaka, Resce and Mareschal (2014) to study the efficiency of UK universities. Our efficiency study reveals that 44.6% of the NGDOs (33 out of the 74 studied) operate inefficiently, compared to 29.7% which are efficient.

We dedicate the following section to the literature review. The third section is oriented to inferring, from donor regulations, the efficiency indicators. For this, the eligibility criteria of three public agencies are analyzed, the Spanish Cooperation Agency (AECID), the European Agency (EuropeAid) and the American Agency (USAID). For each Agency, the criteria, indicators, verification sources and associated weights are extracted, reported in its regulations. This study shows that the criteria related to the narrative of the candidate project to be subsidized are, apparently, the most valued by donors, but a pre-selection is deduced, the existence of an internal tide of criteria and indicators, which donors do not specify.

The fourth section begins by combining the indicators established by the three donors, analyzed in the previous section with the methodology of Promethee applied to the list of organizations that have received the most public grants. The paper ends with the conclusions of this research, taking into account the results obtained in the previous section.

2. Literature Review

The first attempts to develop performance indicators for non-profit organizations were based on the adaptation of those established for for-profit organizations. Despite doctrinal efforts to reach a consensus, as early as 1998, Forbes showed how difficult it was to find valid indicators, since measures such as profitability or return per shareholder, which made it possible to evaluate the performance of for-profit organizations, were not easy to translate to the non-profit world.

Measuring the performance of third sector entities is subject to a series of limitations already known in the academic literature. The absence of a benefit that serves as an indicator, the search for rewards with qualitative and intangible features, the performance of material and immaterial activities that are difficult to quantify, the existence of volunteer staff or the difficulty of measuring the impact of their actions, mean that neither what performance should be measured, nor the approach to measure it, nor the indicators to use for it and not even what is the justification for measuring that performance (Cordery and Sinclair, 2013; Moxham, 2014; Bach-Mortensen and Montgomery, 2018) is clarified. Adding to these difficulties is the fact that there is a lack of official data on the third sector, especially on NGDOs due to the limitation of having to quantify their performance in third countries. In addition, the strong dependence of NGDOs on donations and the need to present a good image in order to continue to have the support of their funders, together with the lack of economic experts within these entities, can lead to misreporting, an incorrect imputation of accounting items that distort the operability of the data collected (Martín et al., 2012).

The limitations listed above mean that most of the published works approach the study of the efficiency of third sector entities from a common approach (Lecy et al., 2012; Cordery and Sinclair, 2013 and Mouchapms 2014), using various tools, all of them coming from the for-profit sphere, so that their validity for the third sector is questioned, and with indicators of achievement established subjectively by the authors of the study, generally based on the data available to carry it out. In this sense, Mouchamps (2014) states that traditional performance measurement tools (in his study he focused on three of them: the balanced scorecard, the social return on investment and the Global Reporting Initiative) would need to be adapted in such a way to the peculiarities of the third sector that, if this could be done successfully, we would be faced with a completely new tool.

Among the studies dedicated to analysing the performance of NGDOs or the third sector in general (excluding, therefore, those dedicated exclusively to social action or humanitarian aid organizations), there are studies dedicated to analysing the efficiency of one of the activities carried out by the organizations or of one aspect of their activity. Thus, Ribar and Wilhelm (2002) analysed the efficiency in fundraising and in carrying out interventions by American NGDOs, concluding that efficiency in one of the activities detracts from efficiency in the other. Hernangómez, Martín Cruz and Martín Pérez (2006) studied the objective pursued by NGDOs from the point of view of satisfying the interests that stakeholders place in the organization and, based on this objective of creating value for participants, analysed the efficiency of Spanish NGDOs. García and Marcuello (2007) studied the efficiency of NGDOs in attracting resources; the results obtained showed that the influence of size, measured by NGDO income, is very relevant in their efficiency and that the NGDOs that specialize in public funds or private funds obtain better results.

In 2011, Berber, Brockett, Cooper, Golden, and Parker, studied the efficiency of fundraising and service provision in non-profit organisations, taking as their objective the social benefit generated by these organisations. In their study, they state that to date no rigorous method has been presented to analyse the efficiency of non-profit organisations, with potential donors being satisfied with analysing the ratio of funds raised to administrative expenses, concluding that there are service provision costs that the organisations did not account for, in order to improve their image in the eyes of potential donors, which influenced the measurement of efficiency.

Martín, Martín, and Serrano (2015) use DEA to study the efficiency of Spanish NGDOs' development interventions. They conclude that efficiency is to some extent independent of the specific environment where the actions are implemented and depends, to a greater extent, on the type of cooperation instrument through which the aid has been channelled. They also conclude that the country results highlight the importance of focusing on smaller areas of action.

Gálvez, Caba and López (2016) studied the efficiency of Colombian NGOs, focusing on transparency as a means to legitimise these entities. They conclude that only 16.83% of the organisations studied are efficient, those that employ the fewest workers and volunteers and those that receive the least public funding. They also note a strong correlation between the transparency of the organisation, especially financial and performance transparency, and its efficiency, with the influence of governance transparency and the profile of the organisation on the organisation's efficiency not being significant.

Ferreira et al. (2017) focus on the study of fundraising activity and how the quality and quantity of fundraising can influence the performance of NGDOs. They conclude that the greater the diversity of funders, the lower the dependence of NGDOs on their donors, providing management autonomy that favours a better performance of the entity.

There are also studies dedicated to analysing the methodology used to study organisational efficiency and the most appropriate indicators for this purpose. Kaplan (2001) attempted to adapt the balanced scorecard of the for-profit sector to non-profit organisations. The same effort was made by González and Cañadas (2008) for Spanish organisations and Greiling (2010) for German third sector organisations. However, the use of the balanced scorecard has not taken off in the non-profit sector (Retozala, 2010), questioning whether the dimensions used to measure the performance of NGOs are appropriate, due to the fact that they do not include the evaluation of the decision-making process and due to the difficulty of evaluating the indicators established by the balanced scorecard to measure the outputs obtained and the necessary inputs for this (Mouchamps, 2014). Another criticism is that it tempts managers to set unambitious targets in order to be able to publicise the success of their achievements (Bisbe and Barrubés, 2012).

The tool commonly used by donors for the design and monitoring of interventions is the logical framework approach, which has also been widely criticised for its rigidity, to the extent that it has become a mere administrative procedure rather than a method of planning by objectives (Sainz, 2007; Cordery and Sinclair, 2013).

Mouchamps, in 2014, made an attempt to find the most suitable tool for measuring the performance of third sector organisations, and came to the overwhelming conclusion that "kitchen scales (tools designed for the for-profit sector) are being used to weigh elephants (non-profit organisations)", and that specific systems need to be designed for the third sector. However, there is an absence of studies examining how such systems should be developed (Moxham, 2014). Bach-Mortnsen and Montgomery (2018) went as far as to state that non-profit entities do not analyse their performance due to a lack of consistent guidelines on how such an evaluation should be carried out, with what criteria and indicators.

As we have seen, there is no agreement on which tools, criteria and indicators are the most appropriate for analysing the good work of third sector entities and, therefore, of NGOs. However, it is recognised in the studies analysed (Medina, 2009), as well as by the cooperation agencies themselves (European Commission, 2016; USAID, 2012; Resolution of 17 September 2013, of the Presidency of the Spanish Agency for International Development Cooperation) that the regulations and indicators established by donors to manage and choose the organisations to which they grant subsidies are rigorous and guarantee the good work of these organisations. Taking this statement into account, that is, if the criteria used by donors to select the NGDOs that will receive a grant are strict and guarantee the good work of the chosen organisations, they could serve as indicators of the good work of these entities and could give rise to indicators that are not only valid but also accepted for the third sector. Consequently, by analysing donor eligibility criteria, the indicators used by public donors to grant subsidies to some entities or others, it will be possible to obtain performance indicators for NGDOs that will allow us to classify entities according to their better or worse performance. We will devote the next section of this paper to this analysis.

3. Analysis of The Three Agencies

In this section, we establish a series of NGDO performance indicators based on the study of the eligibility criteria of public donors. For this purpose, an analysis of the regulations of three cooperation agencies has been carried out: Spanish Agency for International Development Cooperation (AECID), the European Agency (Europe-Aid) and the United States Agency for International Development (USAID). This study has allowed us to synthesize the indicators of good performance used by each one of the agencies. By comparing the criteria, indicators, requirements, weightings and sources of verification used by the three agencies, we will deduce which criteria and indicators can be used to verify the internal performance of NGDOs. These common indicators will allow us to demonstrate the performance of NGDOs to any public donor and, by extension, to the stakeholders of these entities.

3.1. Analysis of the AECID´s Eligibility Criteria

From the analysis of the general regulatory framework to which AECID grants are subject, it can be concluded that the eligibility criteria are defined in an imprecise and unclear manner, scattered throughout the regulations and without establishing precise relationships between the criteria, the indicators used to verify compliance and the sources of verification used for each indicator.

In all the regulations there is only one criterion relating to the entity's performance, which is to have a structure capable of sufficiently guaranteeing the fulfilment of its objectives. However, the indicators used by the donor to analyse compliance with this sufficiency criterion must be deduced from the sources of verification listed in article 7 f) of Royal Decree 193/2015, regulating the prescriptive registration in the AECID Register, as the Agency requires the organisation chart of the organisation and its delegations, the number of partners, the staff hired and the number of volunteers and development workers dedicated to activities related to international development cooperation. In sections g) and h) of the same article, and without relating them directly to any criteria, a certificate is requested accrediting the funds received from membership fees, donations and income generated by private fundraising activities that could finance part of the entity's activity, and the annual report corresponding to the last year of activity, if any.

All the indicators listed, although not explicitly stated in the regulation, are clearly related to the analysis of the material and human resources that the entity possesses and its capacity to carry out the development actions it undertakes, and can therefore be considered indicators of the criterion of sufficient structure.

This lack of legislative clarity in establishing the indicators used to verify compliance with the criterion of having a structure capable of sufficiently guaranteeing the fulfilment of its objectives means that it is not possible to be sure exactly which are the real indicators used by the Agency to verify this sufficiency. It is not known whether it takes these data in an absolute sense or whether it takes into account other indicators derived from them. Nor does the legislation make explicit what requirements these indicators must meet in order to consider that the entity has sufficient structure (a minimum number of partners, staff, projects carried out, etc., a reference threshold that sets the boundary between a bad and a good structure). It is known that the weight given by law to these requirements is 100%, as failure to meet the criterion of having a sufficient structure will result in refusal of registration in the AECID Register. Therefore, to calculate the weight given to each indicator of the criterion, this 100% is divided among all indicators (see Table 1).

In the general regulations, the rules governing subsidies according to the type of development cooperation action (agreement, project or action) establish four eligibility criteria, three of which refer to the formulation of the specific intervention to be subsidised: the quality and effectiveness of the aid, the complementarity of the project with the objectives of Spanish cooperation and the quality of the formulation, viability and sustainability of the proposed actions, and only one criterion referring to the internal functioning of the NGDO, institutional solvency, which is weighted with 20% of the total of the four criteria.

Furthermore, it is pointed out that, in the case of bidding for project subsidies, bidding NGDOs must have carried out at least five cooperation projects in the last eight years, which is why a new criterion is identified: the entity's previous experience, with the indicator being the number of projects carried out with a formal weighting of 100%. In the case of tenders for agreements, the NGDO is required to be qualified, a qualification requirement to which we will refer later (article 5 Order AEC/2909/2011).

After analysing the general regulations, it remains to study the specific regulations contained in the calls for proposals for each of the Agency's funding instruments. The AECID annually calls for grants for the implementation of projects, every four years it calls for agreements and periodically calls for development cooperation actions.

Taking into account all the indicators established by the Agency in the different funding instruments analysed, for the institutional solvency criterion, the only one referring to good internal functioning, the following indicators can be established: 1. having the following published on the organisation's website: activity reports, audited accounts and the composition of the governing bodies. 2. the number of employees according to the entity's funds. 3. indicators relating to sources of funding, 4. indicators referring to the quality of justification of previous AECID subsidies, 5. the existence of rules of conduct for employees, 6. the existence of mechanisms for processing complaints and suggestions, and 7. the alliances made.

In both calls for projects and agreements, the criterion of institutional solvency has a weight for the AECID of 20% in the evaluation (article 9 of Order AEC/2909/2011), but it is not known what specific weight the Agency gives to each of the indicators of this solvency, so for the purposes of this study, it is considered that they all have the same weight, i.e. each indicator listed counts 1/7 of the 20% (2.86%) corresponding to each solvency indicator.

In addition, some of the indicators listed involve data that are not publicly available, so the associated indicators 4, 5, 6 and 7 should not be taken into account. In summary, Table 2 shows the indicators extracted from the reading of Order AEC/2909/2011 as well as the different resolutions calling for grants for the three AECID cooperation instruments, except for the one referring to the required qualification.

The procedure to obtain the qualification is divided into two phases. In the first phase, it is analysed whether the organisation verifies a series of general requirements; without the accreditation of these requirements, it will not pass to the next phase and qualification will be denied. The second phase involves subjecting the organisation to a quantitative and qualitative assessment of its performance, giving each item analysed a certain score.

In order not to be exhaustive in listing all the indicators included in the Resolution regulating the rating of entities, Table 3 and Table 4 list only those indicators included in the assessment of the rating that are related to the internal functioning of the entity, leaving out the formal indicators and those whose verification requires unpublished data. Nor are the minimum compliance thresholds established by the Agency to consider the indicator passed taken into account, since, as they are established for large NGDOs, if these thresholds were taken into account, they would cease to be valid as indicators of good performance for medium-sized and small NGDOs.

For the purposes of this research, in order to weight these indicators, and given that the Agency gives 100% weight to compliance with each of the associated requirements, it will be considered that together they weigh 100% for the Agency, so that each of them will have a weight in this study of 1/8 of 100%, i.e. 12.5%.

Once the first phase has been completed, the Agency carries out a quantitative and qualitative assessment of the entity. The criteria and indicators extracted from the regulations that can be verified with the published data are shown in Table 4. It should be noted that those of the indicators established by the Agency that are closely related to each other have been grouped together in the same indicator, and that in other cases the wording of the indicator has been adapted so that it can be verified with public data; these points are indicated in Table 4.

Analysing all the indicators derived from the regulations that set out the AECID's eligibility criteria, the following conclusions can be drawn. Firstly, that there are repeated indicators or indicators that value similar concepts, which have been grouped together in a single indicator, adding the weighting given to each of them. Thus, for example, Table 5 shows how the number of workers is valued in indicators 3, 13, 22 and 25, which can be grouped together in a single indicator, called average workforce, with a weighting that is the result of the sum of the weights of each indicator grouped together. The average workforce would be an indicator with a total weight of 30.11 percentage points out of 260.78 possible.

Secondly, it is concluded that the indicators can be summarised, by similarity, in four criteria: experience, transparency and internal functioning, human resources and financial resources. Table 6 summarises the indicators extracted from the AECID's eligibility criteria, classified into criteria and summarised under a single criterion for all indicators that are similar. For each indicator, those to which it would correspond for the Agency are indicated.

3.2. Analysis of the European Union's Eligibility Criteria (Europe-Aid)

The European Union has been engaged in development cooperation since its creation as the European Economic Community in 1957. EU development cooperation policy is managed by the Commission's Directorate-General for International Cooperation and Development (DG DEVCO), which since 2015 has brought together the former Europe-Aid Cooperation Office (AIDCO, created in 2011) and the Directorate-General for Development and Relations with ACP (African, Caribbean and Pacific) States.1

The EU accompanies each call for proposals with a guide for applicants, which describes, among other things, the purpose of the call for proposals, the eligibility requirements for applicants and the evaluation criteria for the selection of competing NGDOs and the award of the grant. In general, in order to be eligible to tender for grants, the EU requires NGDOs to meet certain requirements related to: (i) the nationality of the entities (contracts are only concluded and grants awarded to entities from certain countries) or (ii) to the origin of the goods they intend to purchase with EU funds. In addition, it asks entities to provide a declaration of honour certifying that they have not committed any irregularities in their previous operation. These irregularities have to do with bankruptcy proceedings, fraud, corruption or terrorist offences in previous contracts with the EU. All these requirements cannot be considered indicators of good performance. In general, in order to check the financial and operational capacity of NGDOs, the EU requires applicants to be registered in the Potential Applicant Data On-Line Registration (PADOR). Organisations wishing to obtain EU funds must provide PADOR with data on the organisation's profile, values, target population, sectorial and geographical experience, as well as financial data, human resources, and the composition of the governing body. However, the regulations do not make explicit to what extent and how these data are used (Europe-Aid Online Registration Service, 2008).

Once registered in PADOR, an evaluation committee carries out a first analysis of the technical formulation of the project submitted by the NGDO to obtain the grant. In this first analysis (called analysis of the synthesis formulation), only aspects of the project are rated, according to criteria that are individually assessed on a scale of 1 to 5 (1 would be very poor compliance and 5 very satisfactory compliance). Only the summary documents that have obtained a certain minimum total score will be taken into account for pre-selection purposes. These criteria, as they refer to specific aspects of the project submitted by the NGDO for the grant (relevance and design), do not provide information on the general functioning of the organisation but only on how it will implement this particular project. Therefore, they cannot be considered as indicators of the organisation's performance.

After the pre-selection, in the next step the EU reassesses, through the data provided in a 'complete' application form, aspects related to the specific action to be carried out. In this step the goal is reassessing its relevance and also assessing its effectiveness, viability and sustainability, devoting a single criterion, to which it awards a weighting of 20 points out of 100, to the assessment of the operational and financial capacity of the applicants.

The indicators established by the EU linked to this criterion and their weighting are listed in Table 7. The scoring scale for each criterion ranges from 1 to 5, where 1 = very poor; 2 = poor; 3 = acceptable; 4 = satisfactory; 5 = very satisfactory. The NGDO must score a minimum of 12 points on this criterion to pass the selection. The description of the indicators is ambiguous as terms such as "sufficient experience", "sufficient capacity" or "sufficient sources of funding" are used, so the final score given to each indicator will depend on the experts.

After the evaluation of the financial and operational capacity, the NGDOs will be analysed by the evaluation committee by comparing the data declared to PADOR with the information contained in the statutes of the entity. These statutes include the financial reports, the audit report of the last year available, a legal entity form (showing the legal form of the entity and its personal data), a financial entity form (showing the bank in which the grant will be deposited) and a declaration of the entity stating that it has the means to carry out the action and commits itself to do so.

In no case does the EU provide information on the indicators, and the requirements associated with them, that serve as a basis for the assessment of NGDOs. It is not possible to know on what basis the EU finally decides which organisations to subsidise out of all those that have passed the previous stages analysed.2

The analysis of the data requested by the EU has made it possible to identify possible indicators that have been classified into four criteria (see Table 8). As the financial and operational capacity criterion has a weighting of 20% for the EU in the subsidy scale, it has been divided among all the indicators identified in this study, giving them all the same weighting.

3.3. Analysis of USAID Eligibility Criteria

The United States regulates the awarding of development cooperation grants in the Foreign Assistance Act of 1961 and in Regulation 2 CFR 200: Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards (U.S. Government Publishing Office [GPO], n.d.a). USAID, as the agency in charge of US foreign policy in the area of development cooperation, works under a series of rules, directives, encompassed in a system called ADS (Automated Directives System).

When USAID issues a Notices of Funding Opportunity (NOFO), any NGDO that wants to apply for a USAID grant must have a DUNS (Data Universal Numbering System) number, which identifies the organisation. It also must have a NATO Commercial and Government Entity Code (N Cage) and be registered in the United States Government Grants Management System (SAM) (US Federal Register of Contractors, USFCR, 2020). Obtaining the DUNS and (N) Cage does not require any additional requirements other than providing the organisation's identification data.

Registration with SAM also involves providing information on the organisation's age, fiscal year-end date, total revenues for the last three years, number of employees for the last year, compliance with environmental laws, tax obligations, and federal antitrust and corruption laws (affecting the organisation's officers and directors). These include that the organisation's representatives and officers must not be debarred from working with the federal government or have been convicted of antitrust violations, compliance with previous contracts with the US government, declaration of non-procurement of goods and services from countries that do not guarantee labour rights or have restricted foreign policies3, and compliance with US laws on non-discrimination on the basis of race, sex, religion, colour, sexual orientation or national origin. Upon registration with SAM, organisations seeking to receive a USAID grant must sign a declaration of honour4 certifying compliance with certain legal and ethical obligations.

Once these pre-requisites for registration have been met, USAID subjects the eligibility of organisations to a four-stage screening process.

The first phase consists of passing general criteria, usually related to the NGDO's experience, resources and compliance with US legislation. The second phase involves the fulfilment of the criteria set out in the specific call for proposals for a grant. These usually refer to the applicant organisation's previous experience both sectorially, and geographically, taking as indicators the number of successful actions previously carried out in the same sector and the number of years working in the country. Once the two previous phases have been completed, USAID submits the NGDO to an initial review. The criteria for passing this initial evaluation are established in each call for proposals, although, in general, they are in line with those listed in Table 9.

USAID does not establish the specific indicators that will be studied to verify compliance with each of the criteria, nor the weight to be given to each indicator. Thus, for example, one of the criteria that appears is previous experience in activities similar to those of the project with a weight of 10%. The fact is that it does not say whether experience will be measured by the number of projects previously carried out in the same sector, by the number of projects carried out in the same country, or whether other indicators will be taken into account. Nor does it indicate what the requirements are for each indicator to be considered positive, or what weight each indicator has when assessing the criterion in question if there is more than one indicator to check the same criterion. Furthermore, USAID gives the scale with the weights associated with each criterion, but does not establish the minimum score to be achieved in order to pass this preliminary phase. The only thing it adds is that the applicant will be informed in writing after the initial review.

In relation to the criteria set out in Table 9, those referring to the technical approach of the project (technical aspects) will not be taken into account in this study. The reason is that they are criteria related to a specific project in time and therefore represent a declaration of intent, but do not demonstrate the real functioning of the entity.

The criteria set out in the management and organisational analysis (management and institutional capacity) are considered, but their indicators are not provided by the Agency. It is known that the data on which the decision is based are contained in SAM and in the documents requested from the entity. In consequence, the analysis of previous experience will be based on the data on the seniority of the organisation contained in SAM and on the applicant's performance history. It is not possible to know, from the data in the performance history, which specific indicators are used to verify this experience, which has a weighting of 10% in the grant analysed. In this work, two indicators are considered: the length of service of the organisation and the number of projects previously carried out, both with a weight of 5% each (see Table 10).

To check human resources, USAID uses the indicator number of employees in the last year. Although as the criterion is worded, human resources - staff, partners and/or consultants - to implement all project components, including technical, administrative, financial and monitoring and evaluation, it seems to only analyse human resources. Considering that total income for the last three years is also requested (in SAM) and that audit reports are required (in the grant application), it is concluded that the analysis also covers financial resources. However, it is not known what data is analysed from these documents and therefore which indicators are associated with this requirement, so we will only consider the USAID explicit indicator of the number of people employed in the last year, with an associated weight of 25% (see Table 10).

If the NGDO passes the initial assessment, the three previous phases, it will go on to a new, more detailed analysis of its capacity, generally by means of a "pre-assessment survey" (NUPAS - Non-U.S. Organisation Pre-Award Survey). The NUPAS involves the analysis and evaluation, by a group of experts appointed by the Agency, of 29 criteria, classified into six categories or blocks, of equal weight, i.e. each of the blocks has a weight of 1/6 of the total score.

To verify compliance, each criterion may have one or more indicators, which will be evaluated according to a scale ranging from a score of 1 (inadequate compliance) to 4 (strong compliance). The score of the corresponding criterion is calculated as the average of the score obtained for each indicator. The block score is calculated as the average of the scores for each criterion and the overall score, the score obtained by the organisation, as the average of the scores in each of the 6 blocks. Depending on the score obtained, the experts conducting the evaluation will issue an audit report, with three possible results: (1) totally positive, the organisation is eligible to receive a grant; (2) positive, but with caveats, the organisation has operational deficiencies in certain aspects that USAID considers key and must be corrected to be eligible; and (3) negative, the organisation does not meet the minimum operational requirements to ensure proper management of USAID funds and the implementation of the activities to be funded. It is not known what average score the organisation must obtain for the report to be of one type or the other.

It should be noted that the NUPAS in some cases specifies the indicators of the criterion (for example, the solvency ratio as an indicator of the absorption capacity criterion) but in several cases, compliance with the criterion or the associated indicator is defined generically, using as a source of verification the observation by a group of experts of the organisation's internal procedures. This is done without specifying which variables are used by the group of experts or what must be included in these procedures for the indicator to be considered positive. Table 11 shows those indicators specified by the NUPAS, taking into account that those that the experts verify by means of data collected through observation have not been considered.

Having analysed the criteria, indicators, requirements, weightings and sources of verification used by the Spanish, European and American agencies, in the following section we will deduce, by comparing them, which criteria and indicators can be used to verify the internal performance of NGDOs.

3.4. Choice of Criteria and Indicators to Study the Good Performance of NGDOs

Having analysed the eligibility criteria of the Spanish, European and American agencies, it only remains to deduce, by comparison, which criteria are used to verify the good internal performance of the entities, the indicators and requirements associated with them, and the weights of these indicators.

The indicators obtained from the three donors analysed and the standardised weights calculated for each indicator (Table 6, Table 8 and Table 11) have been summarised in a single Table 12. The weights of each Agency for each indicator have been rounded and the indicators not covered by the different Agencies are listed as not applicable (N.A.).

From the study carried out in the previous section, we conclude that the criteria most frequently used by donors are four: (1) experience, (2) transparency and internal organisation or functioning, (3) human resources, and (4) resources and financial management. We now classify these indicators into the criteria listed above, establishing for each of them its realisation value or ideal value from the donor's point of view, in accordance with the regulations analysed in the previous section.

The first criterion is that of experience, collected by the agencies by means of the following indicators: 1. number of projects carried out previously; 2. length of service of the NGDO, which includes indicators related to the number of years the organisation has been involved in cooperation; 3. volume of funds managed and 4. number of countries in which it operates.

For donors, experience is a criterion to be maximised, awarding more points in the grant access scales to those organisations that offer higher numbers in different criteria. These are the number of projects executed, volume of funds managed, countries in which they operate or the organisation's seniority, which is logical, as the greater the experience, the better the internal functioning of the organisation due to the lessons learnt.

The second criterion is transparency and internal organisation. The assessment of the internal functioning of the organisation is usually exhaustive by the three donors analysed. From the point of view of transparency, the Spanish and American agencies assess whether the entity has a website or electronic headquarters. In the Spanish case, it is also made explicit that in order for the entity to be considered transparent, the webpage must include "activity reports, audited accounts, the composition of its governing bodies and technical teams" (AECID, 2019, article 11.2). In the case of USAID, it only refers to transparency within the criterion "governance and practice of good corporate governance", assessing "disclosure of data and transparency" but without defining what this consists of (USAID, n.d.).

On the other hand, if only the documents used by the three donors are taken into account to analyse the internal functioning of the organisation, without considering those that are used by a specific donor (such as the investment policy requested by AECID -AECID, 2017-), it can be deduced that the internal functioning is based on the analysis of different documents. These are the statutes (where it is also possible to assess whether the organisation has the functions of the board members, a procedure for the renewal of these members and a term limit), the strategic planning (which also includes the organisation's mission, vision and values), the activities report (which also allows us to know whether the actions carried out and the annual planning are monitored), the organisation chart (which, if complete, will also allow us to know whether the organisation's lines of responsibility, authority and communication are delimited), the annual accounts and the audit reports.

For donors, transparency and internal functioning is a maximisation criterion, with higher scores being awarded to those NGDOs that achieve higher scores on these indicators.

The third criterion covers human resources. Human resources are considered to be those people who work for the organisation, with or without an employment contract.

Another indicator that is considered interesting, and which is only assessed by the AECID, is the temporality rate, calculated, as defined by the AECID (2017), as a percentage of permanent contracts over the total staff. However, this definition does not correspond to that usually used in academic literature or by official5 bodies, which define the temporality rate as the ratio of temporary contracts over the total. This rate makes it possible to evaluate whether the organisation has a stable workforce, retaining their learning and experience, and also assesses whether the staff turnover is low, which can denote good staff management (good training, good remuneration, transparency in decisions, etc.).

The last criterion is financial resources. As with the criterion of transparency and internal organisation, the organisation's financial resources are assessed by donors using indicators and requirements that are not clearly publicised. For this, there is a group of people who are experts for the donor who analyse the organisation's policies, internal functioning and financial documents by observation.

However, from the comparison of the three agencies analysed, it can be deduced that the most important thing for the donor is to verify the existence of sufficient resources in the organisation to carry out the subsidised project and the stability of the organisation's financial resources.

Table 13 lists the indicators that have been deduced to analyse the performance of the institutions. We indicate whether for this group it is an indicator to be maximised or minimised, the indicators that have been discarded, the weights attributed to each indicator by each agency and the total weight that this indicator has for the donors. Indicators not considered by the different agencies are shown as not applicable (NA).

The indicators discarded are due to the fact that only one donor has taken them into account. The reason is that they are not considered to be an effective indicator for analysing the performance of the entities, such as the publication of the investment policy on its website, the number of sectors in which the NGDO operates or the number of networks to which it belongs.

As far as profits are concerned, in a non-profit sector, not making profits is not considered as a factor to be taken into account when analysing efficiency. The distribution of profits is limited by law (Article 3 of Law 49/2002 of 23 December 2002 on the tax regime for non-profit organisations and tax incentives for patronage), so that making a profit cannot be the reason for collaborating with the organisation or the objective expected of it.

4. Empirical Results: Promethee Approach

The efficiency of the 74 NGDOs that form part of the State NGDO Coordinating Committee is analysed using the indicators obtained from the donor eligibility criteria listed in previous tables. The data were obtained from those published by the Coordinating Committee in its recent sector report (Coordinadora de ONGD-España, 2017b), from the reports that NGDOs send to the Coordinating Committee and that the latter publishes on its website (Coordinadora de ONGD-España) and from the data on the website of each of the NGDOs in the sample. For the application of the method, the free software Visual Promethee 1.4 Academic Edition, (available at http://www.promethee-gaia.net/software.html) will be used. The steps are as follows: (i) Configuration of the multi-criteria decision matrix (ii) The different alternatives (the 74 NGDOs) and the indicators (listed in previous tables) that will be used to determine the efficiency of each NGDO, their values and weights, are loaded into the programme. Each indicator is assigned its unit of measurement and it is indicated whether it is an indicator to be maximised (the higher the indicator, the higher the value) or minimised (the lower the value, the higher the value). (iii) Two groups are created, the outputs (represented in the study by a green circle) and the inputs (represented in the study by a blue square), classifying each indicator in the corresponding group. Table 14 shows the NGDOs ranked according to their efficiency and the grants (in euros per project -€/pyto-) they have received from Spanish public donors during the analysed period, obtained from the BDNS. It also shows the order of the NGDOs from the highest to the lowest volume of grants received (in €/per project).

Taking the data to a spreadsheet and drawing a Cartesian plane where the virtual input is represented on the abscissa axis and the virtual output on the ordinate axis, Figure 1 (Appendix) is obtained.

The efficient frontier has also been represented by joining the NGDOs that occupy the best position (Oxfam Intermón, Manos Unidas and Cáritas Española), those with the highest net flow of output, given their inputs, or the lowest net flow of inputs given their outputs. It is important to take into account that we are working under the assumption that any positive convex linear combination of efficient NGDOs also belongs to the frontier, so the efficient frontier will be the one represented in Figure 1 by the blue line.

The virtue of the graphical representation is that it allows, in an intuitive way, to know what position each entity occupies, which entities are closer to it and what is the distance that separates it from the frontier. Moreover it also shows in which direction (if the frontier is closer to it by increasing the outputs, as would be the case of FAMME, or by decreasing the inputs, as would be the case of ACF-E).

By means of the graphical representation, NGDO managers can have a clear vision of which variables (inputs or outputs) they must act on to improve their position and, by analysing the functioning of the NGDOs located closer to the frontier, they can approach the decisions to be taken to improve the results obtained with their management (Figure 1 in Appendix).

The results obtained show that 44.6% of the NGDOs (33 out of the 74 studied) operate inefficiently, compared to 29.7% which are efficient. Moreover, 9.5% of the entities operate effectively, achieving their objectives, but at the cost of wasting resources, and 16.2% manage their resources well, but do not manage to satisfy the interests of all their participants, taken as the objectives of the NGDOs in this work.

Considering only the achievement of objectives (efficient and effective entities), 39.2% of NGDOs are efficient on the output side. On the input side, 45.9% of the NGDOs have a better management of their resources (efficient and frugal).

5. Conclusions

This paper aims to study the efficiency in the development cooperation sector, define the objectives and resources of the NGDOs and find general indicators that allow verifying to what extent the entities are achieving their goals and using their resources to be able to apply them to an efficiency study.

The efficiency indicators are inferred from those used by public donors when they select the NGDOs to receive funding, the so-called eligibility criteria published in different norms and laws. Thus, efficiency is measured in this paper using the indicators collected in the eligibility criteria of three of the main cooperation agencies: AECID, EuropeAid and USAID.

Given this opacity and in order to know if the indicators published by donors were sufficient to determine the award of a public grant to a NGDO, a simulation of granting public funding was applied. For this, the 74 NGDOs that in 2016 (up to 2018-2020) belonged to the Spanish Coordinator were taken. With the donor variables, a study of the efficiency of the coordinator´s NGDOs is carried out, using a technique that is an adaptation of the Promethee multi-criteria method known as PROMETHEE Productivity Analysis (PPA).

Our efficiency study reveals that 44.6% of the NGDOs (33 out of the 74 studied) operate inefficiently, compared to 29.7% which are efficient. Moreover, 9.5% of the entities operate effectively, achieving their objectives, but at the cost of wasting resources, and 16.2% manage their resources well, but do not manage to satisfy the interests of all their participants, taken as the objectives of the NGDOs in this work. Considering only the achievement of objectives (efficient and effective entities), 39.2% of NGDOs are efficient on the output side. On the input side, 45.9% of the NGDOs have a better management of their resources (efficient and frugal). Our results are important for academics, donors and managers of these type of organisations because they must be efficient in the use of the money they receive.