Linking Entrepreneurship to Productivity: Developing a Composite Indicator for Farm-Level Innovation in UK Agriculture with Secondary Data

1. Introduction

Producing more food, fibre and fuel with fewer inputs, such as land, fertilizers and water, requires change to the efficiency with which these inputs are used (Soteriades et al., 2016; van der Gaast et al., 2022). Productivity improvement not only helps achieve government goals of “sustainable intensification” (SI) (i.e. producing more with less), s exemplified in the UK by the Foresight Report on the Future of Food and Farming (Foresight, 2011), but it is also beneficial for the individual farm business, as it results in reduced input and resource costs per unit of output, leading to higher income (Poudel & Pandit, 2020; Warr & Suphannachart, 2021). Furthermore, productivity improvements can also lead to reductions in vulnerability to some risks (e.g. droughts, pests etc.), reductions in escape of damaging inputs to the environment, improvements in product quality and enhanced social responsibility (OECD, 2005; Pretty et al., 2020; Scognamillo et al., 2022).

Productivity thrives within the dynamic landscape of entrepreneurship, particularly in the context of UK agriculture. Entrepreneurs in the agricultural sector play a crucial role in driving innovation, adapting to evolving market demands, and optimizing operational efficiency (Ang et al., 2013). Their strategic vision and willingness to embrace new technologies contribute significantly to increased yields, sustainable farming practices, and enhanced overall productivity (Bjorklund, 2018). Nurturing a supportive environment for agricultural entrepreneurship in the UK is essential, as it not only ensures the sector's resilience in the face of challenges but also positions it for long-term success in meeting the growing demands of a dynamic and competitive global market (Bjorklund, 2018; Savastano et al., 2022; Thephavanh et al., 2023; Yoon et al., 2021).

Various studies have identified, at the farm-business level, a number of drivers of productivity change, for example: structural change (e.g. increasing scale) (Deininger et al., 2021), access to education and information (Rahman et al., 2022), cooperation between farmers and with upstream and downstream actors (Bizikova et al., 2020), access to new/improved resources (Deininger et al., 2021), climate change (Fujisawa et al., 2015; Ortiz-Bobea et al., 2021; Rahman et al., 2022), technological diffusion and innovation (Rahman et al., 2022; Zezza et al., 2017), among others. While there are a number of these drivers of productivity change, and the importance of these can vary from farm to farm (Buckwell et al., 2014), the OECD concludes that the most important driver of productivity improvement is innovation (OECD, 2013).

1.1. Defining the indicators for innovation

In its very broadest sense, innovation is the generation, diffusion and exploitation of knowledge (OECD, 2013). In business/farm management terms, innovation is defined as the introduction of novelty, i.e. some significant change, to any of several areas of activity within a business. Traditionally, both the practice and study of innovation has been limited to the adoption of science and technology, but innovation is now perceived in much broader terms, impacting such areas as: technological development (for example of new products); production techniques; changes to organisational structures and practices; and new marketing operations (OECD, 2005, 2013). As with most small firms in the non-agricultural sector (excluding the R&D sector), innovations in farm businesses do not generally originate on the farm itself but are acquired through diffusion of novelty onto the farm from elsewhere.

In EU states and beyond, various governmental and non-governmental policy documents delve into strategies for fostering innovation in agriculture, shedding light on the pivotal connection between farm-level entrepreneurship and productivity (Martinho, 2020). These initiatives aim to decipher the barriers hindering innovation and emphasize the role of policy in not only incentivizing but also removing obstacles to entrepreneurial activities at the farm level (Savastano et al., 2022). By aligning policy frameworks with the unique challenges faced by agricultural entrepreneurs, there exists a transformative potential to boost productivity, drive technological adoption, and ensure the sustainable growth of the farming sector. (see e.g. Zezza et al., 2017). However, policy instruments identified as providing encouragement to innovation are often quite broad in their scope and diffuse in their effects, i.e. directed towards achieving multiple desirable objectives simultaneously, only some of which might be targeted at the type of innovation that leads to productivity improvements. To illustrate, the New Entrants Scheme under the Rural Development Regulation aims to encourage younger farmers into agriculture, on the grounds that they might be more innovative than older farmers (Lediana et al., 2023). But to what extent and in what ways would this innovation lead to productivity improvement? Also, what is the relative value of this innovation, compared with other sources of innovation, such as increasing levels of training, purchase of new equipment, or increasing the scale of operation, in terms of driving productivity improvement? Before more targeted and informed policy instruments can be designed, the role of different types of innovation in driving productivity change must be better understood.

Additionally, to assess the efficacy of such innovations it is necessary to be able to measure (quantify), both the innovations (actions) themselves and their outcomes, especially in terms of productivity change, as well as measure the statistical relationship between them. This raises an important question: what metrics are available to measure both innovation (as an activity, or input) and productivity (as an outcome) specifically for the agriculture sector in publicly available datasets that would allow these innovation-productivity change relationships to be illuminated? In agriculture, while it is generally understood that innovation, in some way, drives productivity change, it has proven difficult to quantify this relationship. This difficulty arises not only because of heterogeneity in the population of farm businesses being studied, but also because innovation is a continuous and dynamic process that can occur at any time in many different areas of fam activity. Moreover, it is also that innovation can occur in different areas of the business, affecting different inputs/resources and can also be diffuse, i.e. not directed at any one activity (for example, farmer acquisition of new critical decision-making skills). It is simply not the case that innovation only occurs at set times in a business cycle, or when major investments programmes are instituted. Innovation occurs when a single new piece of knowledge is brought onto the farm, or a new tool, or a planned small-scale adaptation to management is instituted, leading to change. The piecemeal, random and multi-scaled nature of innovation presents real challenges for understanding and quantifying nature of its relationship with productivity and entrepreneurship. For this reason there have been few attempts to quantify/map this relationship, with some past studies as notable exceptions (Bergevoet et al., 2005; Byma & Tauer, 2010; Hansson, 2008; Kilpatrick, 2000; Läpple et al., 2015; Manevska-Tasevska & Hansson, 2011; Simar & Wilson, 2011). However, even in the studies cited above, innovation is not studied as a holistic effect. Rather, one or more factors (e.g. a new piece of technology) or actions that might be deemed to be innovations, are included alongside a larger list of possible non-innovation-based drivers of productivity change to study their relative effects.

More often, innovation itself is not identified, but rather proxies for innovation are used, i.e. metrics for activities which might facilitate innovation, or where innovation might conceivably occur. For example, expenditures on consultancy services and training courses are taken to be proxies for innovation because they are assumed to make innovation more likely. However, there is no assurance that in individual cases these proxy activities have led to innovation. Investment in new plant and machinery can be more clearly identified as innovation and it is for this reason that many early studies of the impact of innovation on productivity have been focussed on adoption of new technologies (Ajzen, 1991; Stefanides & Tauer, 1999, 1999).

Several measures of productivity change have been defined in the agricultural economics literature. The measure known as Partial Productivity is defined as the rate of output produced per unit of each input. This measure is obviously too simplistic for use with multi-product firms and so the more holistic measure known as Total Factor Productivity (TFP) was developed. This expresses the ratio between an index of aggregated outputs and an index of aggregated inputs. According to production theory, the determinants of rate of output are the technology used, the quantity and quality of the production factors and the efficiency with which these factors are employed in the production function (Melfou et al., 2013). Thus, for any firm, change in TFP is the result of the net effect of: changes in efficiency; shifts in the production frontier; and changes to scale of production (Färe et al., 1992).

To undertake a study of the impact of innovation on productivity in a more holistic way, it will be first necessary to identify, from the literature, the broad areas of innovation that can occur and which are relevant as drivers of productivity change. Within these broad typologies it will be necessary to identify specific innovation metrics that might also be represented in official farm datasets. For these purposes, the universe of innovation has been divided into two broad typologies, i.e. (i) innovation in management (including investment in human capital and entrepreneurial competencies, i.e. training); and (ii) innovation through investment in new technologies. Table 1 below presents the results of this review, identifying each activity found in the literature that might be identified with innovation (i.e. it is innovation directly, or an activity that makes innovation more likely) with its broad typology, as well as providing the source literature.

From of the perspective of drawbacks, perhaps the most restrictive barrier to a holistic analysis of the role of innovation in driving farm productivity, is that in the datasets that might be used to derive such metrics, particularly official datasets, there are very few apparent indicators of innovation applicable at the farm level, and consequently, for some dimensions of innovation, no indicators at all. A further complication is that there is no standard metric for measuring productivity change and consequently, studies in the literature use a range of different metrics of farm performance, such as Total Factor Productivity, output, Net Margin, or simply profitability.

1.2. The purpose and contributions of the current study

While it is generally understood that innovation drives productivity change, the exact nature of this relationship is not well understood and there have been few attempts to quantify/map it. One of the barriers to doing this is that there exist few adequate indicators of innovation available in public datasets applicable at the farm level (Läpple et al., 2015).

This study aims to comprehensively explore the role of innovation, particularly within the context of entrepreneurship in agriculture, and its influence on driving transformative changes in farm productivity. The research objectives encompass various facets: (i) thorough literature review, the study aims to map the variables, specifically management actions, that can be reasonably identified as innovation or as factors increasing the likelihood of innovation within agricultural practices. (ii) These identified variables will be classified into either innovation in management or technical change, providing a nuanced understanding of the diverse forms innovation can take. (iii) The study plans to scrutinize data from the Farm Business Survey (FBS) across multiple years, seeking analogues for the indicators pinpointed during the literature search. This approach integrates real-world data to validate and contextualize theoretical frameworks. (iv) To measure the impact of innovation, the study intends to identify an appropriate measure of productivity, justifying the chosen metric as a robust indicator of the outcomes resulting from innovative practices. (v) The research aims to separately evaluate the impacts of innovation on productivity arising from (a) enhancements in the quality or state of existing farm assets, including human capital, and (b) changes in the scale of farming activity. This differentiation adds granularity to the assessment of innovation's multifaceted effects. (vi) Lastly, the study seeks to provide recommendations for future collections of alternative Farm Business Survey indicators, addressing any deficiencies in the coverage of innovation. This forward-looking approach aims to refine data collection strategies for a more comprehensive understanding of the evolving landscape of agricultural entrepreneurship and innovation.

This study contributes to the literature, primarily, by helping to bridge the gap on our understanding on the relationship between innovation and productivity in agriculture. By mapping variables associated with innovation in farm management and technical changes, and correlating these with productivity measures, we contribute to the exploration of how innovation manifests and drives efficiency in agricultural practices. The use of longitudinal data from the to validate theoretical constructs provides empirical grounding to the study, enhancing its applicability and relevance for future research.

2. Material and Methods

2.1. Aims and overview of methodology

The goal of the study will be achieved in two stages. In the first stage of the analysis the Malmquist Index (MI) of total factor productivity (TFP) (Maudos et al., 1999; O’Donnell, 2012) is employed to explore changes in productivity, technical and economic efficiency of farms in the panel over time. The component distance function in the technical change index is then used to identify innovators within the sample i.e. farms that shift the frontier outwards (Färe, Grosskopf, & Lovell, 1994). In the second stage of the analysis, the factors that enable this innovation will be identified. To facilitate this, the MI of TFP will be decomposed by introducing to the regression analysis the components of change in scale of technology i.e.the product of change in scale efficiency and change in scale of technology (Simar & Wilson, 1998a, 1998b, 1999; Wheelock & Wilson, 1999).

2.2. Data sources

Data for the use in the modelling exercise comes from a representative sample of 60 Cereal Farms over the period 2003-2014. The data have been obtained from the Farm Business Survey (FBS) 1 which provides comprehensive information on the structure and physical and economic performance of farm businesses in England and Wales. The restriction of the sample to specialist cereal farms ensures it is relatively homogenous in terms of farm systems and their complexity 2. The inclusion of 60 cereals farms over a 12-year period yield a panel of 720 observations available for the efficiency analysis. For the evaluation of the MI of TFP this provides 660 observations (since the analysis utilises data from two adjacent years at a time).

2.3. Construction of variables for the TFP analysis

Farm output is based on market returns from all farm-based enterprises. Non-market sources of revenue (e.g. savings, or aid and subsidy payments) are excluded on the grounds that they do not vary in response to changes production scale, or the quality or quantity of farm inputs used. The production technology composite variable, i.e. as used in the estimation of technical and sub-vector efficiency, as well as the MI of TFP, is based on the following components: farmed area, crop costs (including fertiliser, crop protection, seeds and other agricultural costs) and total labour (number of paid and unpaid workers). All inputs expressed in money terms (£/ha) have been deflated to constant price terms based on 2010 prices,3. Specifically, the following indexes have been used: Fertilisers and soil improvement index, seeds index, plant protection products index, farm machinery and installation index, other costs index.

2.4. The Malmquist Index of Total Factor Productivity

Since the objective here is to measure the relative performance of farms over time a dynamic setting is required. Therefore, the time series dimension is used to estimate shifts in the frontier over time, thereby providing a measure of technical change and movements of the individual farms towards the production frontier and hence providing a measure of efficiency change. We have adapted an input-orientation Malmquist index since farmers have more control over the adjustment and efficient use of inputs rather than the expansion of output (Balcombe et al., 2008). Specifically, the MI between period $t$ and $t+1$ is defined as the ratio of the distance function for each period relative to a common technology estimated by Data Envelopment Analysis (DEA), following past studies (Balcombe et al., 2008; Simar & Wilson, 1999). Therefore, the MI based on an input distance function is defined as:

$${\mathit{M}}_{\mathit{I}}^{\mathit{t}}=\frac{{\mathit{D}}_{\mathit{I}}^{\mathit{t}}\left({\mathit{x}}^{\mathit{t}+1},{\mathit{y}}^{\mathit{t}+1}\right)}{{\mathit{D}}_{\mathit{I}}^{\mathit{t}}\left({\mathit{x}}^{\mathit{t}},{\mathit{y}}^{\mathit{t}}\right)}$$

(1)

Equation (1) express the ratio between the input-distance function for a farm observed at period $t+1$ and $t$, respectively, and measured against the technology at period $t$. Values of the $M_I<1$ indicate negative changes in TFP, values of the $M_I>1$ indicate positive changes in TFP while values of $M_I=1$ indicate no change in productivity. However, since the choice of period $t$ or $t+1$ as the base year is arbitrary (i.e. the base year can be either period $t$ or period $t+1$), Färe et al. (1992) defined the MI of TFP as the geometric mean of the $t$ and $t+1$ Malmquist indices. Therefore, for each farm the input orientation Malmquist index is expressed as follows:

$${\mathit{M}}_{\mathit{I}}^{\mathit{t},\mathit{t}+1}={\left[\frac{{\mathit{D}}_{\mathit{I}}^{\mathit{t}+1}\left({\mathit{x}}^{\mathit{t}+1},{\mathit{y}}^{\mathit{t}+1}\right)}{{\mathit{D}}_{\mathit{I}}^{\mathit{t}}\left({\mathit{x}}^{\mathit{t}},{\mathit{y}}^{\mathit{t}}\right)}\frac{{\mathit{D}}_{\mathit{I}}^{\mathit{t}}\left({\mathit{x}}^{\mathit{t}+1},{\mathit{y}}^{\mathit{t}+1}\right)}{{\mathit{D}}_{\mathit{I}}^{\mathit{t}+1}\left({\mathit{x}}^{\mathit{t}},{\mathit{y}}^{\mathit{t}}\right)}\right]}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}$$

(2)

where $M_I^{t,t+1}$ refers to the MI of TFP from period $t$ to period $t+1$; $\left(x^t,y^t\right)$ is the farm input-output vector in the $t^{th}$ period; $D_I^t\left(x^{t+1},y^{t+1}\right)=max\left\{\theta >0:\left(\raisebox{1ex}{$x^{t+1}$}\!\left/ \!\raisebox{-1ex}{$\theta $}\right.\right) \in P\right\}$ is the input distance from the observation in the $t+1$ period to the technology frontier of the $t^{th}$ period with $P\left(y^{t+1}\right)$ the input set at the $t+1$ period and $\theta$ is a scalar equal to the efficiency score. The indices are calculated with the use of the nonparametric DEA method (see Supplementary material, section A) in order to construct a piecewise frontier that envelopes the data points (Charnes et al., 1978). The technology assumption made to estimate the MI of TFP is constant returns to scale (CRS). Otherwise, the presence of non-CRS does not accurately measure productivity change (Grifell-Tatjé & Lovell, 1995). The main advantage of the DEA method is that it avoids misspecification errors and it enables the investigation of changes in productivity in a multi-output, multi-input case simultaneously (Balcombe et al., 2008). Furthermore, the use of the DEA method for the estimation of the MI of TFP makes it easy to compute since DEA does not require information on prices.

In addition, the index in equation (2) can be decomposed into two components: efficiency change and technological change, as follows:

(3)

The first part of equation (3) is an index of relative technical efficiency change, $(∆Eff)$, showing how much closer (or farther) a farm gets to the best practice frontier, i.e. it measures the “catch up” effect (Färe et al., 1992). The second component is an index of technical change, $(∆Tech)$, and measures how much the frontier shifts. Both components take values more, less, or equal to unity as it in the case of the MI of TFP indicating improvement, deterioration, and stagnation respectively. In addition, as Färe, Grosskopf, & Lovell (1994) and Färe, Grosskopf, Norris, et al. (1994) demonstrated, the index of $∆Eff$ is further decomposed into two factors, pure technical efficiency ($∆PureEff)$ and scale efficiency change $\left(∆ScaleEff\right).$

(4)

where the $D_I^{Vt+1}\left(x^{t+1}, y^{t+1}\right)$ and $D_I^{Vt}\left(x^t,y^t\right)$corresponds to distance functions estimated under VRS assumption. It must also be noted that $∆Eff=∆PureEff*∆ScaleEff.$The decomposition of Färe, Grosskopf, & Lovell (1994) and Färe, Grosskopf, Norris, et al. (1994) enables the identification of shifts in the CRS frontier over time ($∆Tech$) and changes in pure efficiency and scale efficiency that correspond to variable returns to scale (VRS) frontiers from two different periods. Moreover, the component distance functions in the technical change index of the MI of TFP identifies the farms responsible for the frontier shift (Färe, Grosskopf, Norris, et al., 1994). Specifically:

• if technical change $\left(\mathsf{\Delta }Tech\right)$ of farm $i$is greater than 1; and
• the distance function estimates, under CRS, for the farm in the period $t+1$relative to estimated technology in period $t$ are also greater than 1; and
• efficiency estimates, under (CRS, at time $t+1$ relative to technology at time $t+1$ equals 1;
• then that farm has contributed to a shift in the frontier between the two periods. Formally, this is expressed as follows:

  

  (5)

Kneip et al., (1998), Simar & Wilson, (1998a, 1998b) and Wheelock & Wilson (1999) proposed a further decomposition of the MI of TFP to estimate changes in technology by changes in the VRS estimate. Specifically, if the position of the farm remains fixed in the periods $t$ and $t+1$ in the input-output space, and the only change that happens is in the VRS estimate of technology, then the ($∆Tech$) in equation (4) will be equal to unity, indicating no change in technology. Therefore, to indicate a change in technology, the CRS estimate of technology should change. Hence, Kneip et al. (1998), Simar & Wilson (1998a, 1998b) proposed the following decomposition, based on the assumptions of Kneip et al. (1998) that the VRS estimator is always consistent:

(6)

where the first two components indicate $∆PureEff$ and $∆ScaleEff$ and the $∆Tech$ is decomposed into pure technical ($∆PureTech$) and scale technical change ($∆ScaleTech$). Also, $∆Tech=∆PureTech\ast ∆ScaleTech$. The index of pure technical change is the measure of the geometric mean of the two ratios indicating shifts in the VRS frontier between the two periods. Values of $∆PureTech$ greater than unity indicate an expansion in pure technology, values less than unity indicate a deterioration and values equal to unity indicate stagnation in pure technology. Information derived from the scale technology change index is used to describe the change in returns to scale of the VRS frontier between two time periods. Values of ($∆ScaleTech$) greater than unity is an indication that the farms operates either below or above the optimal scale, values less than unity indicate that the technology is moving towards CRS and when it is equal to unity there are no changes in the shape of technology.

2.5. Panel Data econometric models

A set of econometric models are estimated using the random effects and the Feasible Generalised Least Squares (FGLS) procedures for panel data. Financial and management characteristics data is derived from the FBS and used in a second stage regression to explore the effects of innovation in management and investment on the efficiency and productivity measures obtained by the MI of TFP and its various components. The decomposition of the MI of TFP to its various components and how these are used to capture innovation at a farm level are illustrated in Figure 1.

The efficiency change component $(∆Eff)$ of the MI of TFP and its two factors, pure technical efficiency ($∆PureEff)$ and scale efficiency change $\left(∆ScaleEff\right)$ are regressed to a set of variables in order to explore further how technically efficient managers capture the productivity gains observed in the various periods. Hence, conclusions in regard to innovation in management and through investment in human capital could be derived. In particular, $∆Eff$ and its two factors were regressed against the form of business (Sole trader, Partnership, and Farming Company), the age of the farmer, the level of education (basic education i.e. school only and further education), a dummy variable indicating paid or not managerial input, the size of the farm (based on the FBS classification of size). In addition, a dummy variable was used to indicate those farms that are owner occupied or tenanted and an index to define the level of specialisation for each farm in producing arable crops was designed. The index of specialisation considered the output derived from arable enterprises and the output derived from livestock enterprises. A farm will receive an index of 1 when all its output is derived from arable enterprises and any other number less than that will identify the percentage of other enterprises contributing to the total output of the farm business. Hence, three levels of specialisation were defined for all farms through the periods under consideration (Level 1: 0.7 - 1, Level 2: 0.5 – 0.69 and Level 3: 0 – 0.49).

Descriptive statistics of the variables used for the innovation in management and human capital panel data models are available in Table 2. Key insights show that the average Efficiency Change is slightly above 1 (Mean = 1.04, SD = 0.25), indicating a general improvement in efficiency over the period studied. The Pure Efficiency Change, which represents efficiency improvements excluding scale and mix effects, is close to 1 (Mean = 1.01, SD = 0.14), suggesting modest gains. The Scale-mix Efficiency Change, reflecting changes due to scale and mix of outputs, is also slightly above 1 (Mean = 1.03, SD = 0.18).

Regarding farm structure, Table 2 shows it is predominantly either sole traders (47%) or partnerships (45%). Limited companies comprise a smaller portion (8%). This distribution suggests a dominance of traditional and family-run farm businesses. The education and management aspects of farmers show a significant majority of the farmers holding higher education (64%), with fewer having only basic education (17%) or A-level qualifications (19%). However, most farms (95%) operate without paid managerial input, highlighting the reliance on the farmers' own expertise. The distribution between large (45%) and medium-sized (44%) farms is fairly even, with small farms making up a smaller proportion (11%). The majority of farms are owned (65%) rather than tenanted (35%). Also, a majority of farms (89%) have more than 70% of their output in crops, indicating a strong focus on crop production in the sample. Data on farmer’s age suggest an apparent aging of the farmer population, with the average age increasing from 53 in 2003/2004 to 62 in 2013/2014. Overall, descriptive data suggests that while there have been slight improvements in farm efficiency, these are more pronounced in scale and mix changes rather than pure efficiency. The demographic data points to an aging farmer population, a predominance of higher education among farmers, and a major reliance on crop production.

A series of specification tests have been performed for panel data models (Hausman-type tests). In addition, a series of diagnostic checks were used regarding serial correlation, heteroscedasticity and also cross-sectional dependence. Hence, the model specified for the $∆Eff$ and its two factors ($∆PureEff, ∆ScaleEff)$ was a random effects model. Moreover, since heteroscedasticity has been detected in the case of innovation in management and human capital ($∆Eff,∆PureEff, ∆ScaleEff$), a robust covariance matrix has been used to account for it.

3. Results

3.1. The MI of TFP and its components

The statistical inference of the MI of TFP between 2003 and 2014 is presented in Supplementary Material, Table S1. Values of the MI above unity indicate improvement, while values below unity indicate deterioration in productivity. In addition, the significance of these changes is reported for each farm. Confidence intervals (CIs) were calculated for 5% and 1% levels of significance (A detailed discussion on statistical inference for MI of TFP and their components is available in Supplementary Material, section A.2). Most of the MI estimates are significantly different from unity at the 99% or 95% level. A farm is reported to have experienced significant progress between the two time periods if its confidence interval lower bound is greater than unity. A farm has significantly regressed during the period if its upper bound is less than unity and there is no statistically significant change if unity is included in its confidence interval.

The most important shifts in productivity are identified in period 2008 - 2009 (MI=1.248) and 2011 – 2012 (MI=1.27). The lowest average level of productivity is observed in the period between 2009 and 2010 (MI = 0.791). The variation in the average value of the MI and its components (efficiency and technical change component) is shown in Figure 2. We can observe significant regressions or advancements of the MI are mainly caused by the technical change component rather than the efficiency change component, which is approaching unity in most periods. A significant deterioration of the technical change component is observed between the periods 2003/2004 and 2006/2007. With some fluctuation, MI is constantly under improvement after the 2009/2010 period as all scores are above unity. The product of efficiency and technical change should by definition be equal to the MI in each period.

Table 3 provides further information in relation to TFP change for each farm size group over time. To explore any statistically significant differences between farm size groups in terms of productivity changes the Kruskall-Wallis test (one way analysis of variance by ranks) was used. The null hypothesis, i.e. that the sub-samples originated from the same distribution, could not be rejected for any period. This indicates that no significant differences exist in productivity change between different farm sizes in the study period. All farm sizes have an MI value of less than unity. Furthermore, the average MI for the 11-year period for the large, medium and small farms is 0.98, 0.98 and 0.97 respectively, indicating a slight deterioration of productivity over the period. The relevant geometric means for efficiency change per farm size group for the same period are 1.01 (Small size), 1.02 (Medium size) and 1.03 (Large size). In terms of the technical change component, the average $∆Tech$ for the large, medium and small farms is 0.96, 0.98 and 0.98 respectively. Therefore, the geometric means for the $∆Tech$ and the $∆Eff$ indicate that any progress in the MI of TFP over the period is mainly driven by innovation in management and investment in human capital rather than innovation through investment in new technology.

3.2. Test for innovators in the sample

During the periods 2005/2006, 2006/2007 and 2009/2010 no farm caused any outward shift to the frontier since technical change was less than unity for all farms. In total, 25 farms have been identified as responsible for the outward frontier shift in the remaining accounting periods (in particular farms 1, 2, 9, 13, 14, 18, 21, 26, 30, 31, 32, 33, 34, 35, 36, 38, 39, 42, 43, 45, 46, 51, 55, 59 and 60). Based on the principle outlined in section 2.4 these farms can be identified as the “innovators” in the sample.

3.3. Decomposition of the efficiency change index into pure efficiency change and scale efficiency change

The efficiency change index can be further decomposed into pure efficiency and scale efficiency change, thereby allowing for the isolation of the impact of changes to farm scale on efficiency change. Table 4 reports the distribution of pure and scale efficiency estimates over the review period (estimates of pure and scale efficiency per farm are presented in Tables S2 and S3 in the Supplementary Material).

According to Table 4, the results for 2009/2010 indicate that scale efficiency index has improved for more than 71% of the sample farms. Conversely, the pure efficiency index deteriorates for 51% of the farms in the sample. Figure 3 shows that scale efficiency deteriorates immediately following the 2008/2009 period (Perhaps due to a loss of confidence, or tighter money supply following the financial crisis), but recovers to form an upward trend thereafter. In addition, the improvement in aggregate efficiency for the 2008/2009 is primarily driven by improvements in pure efficiency and thereafter tracks closest to the pure efficiency trend. In conclusion, pure efficiency is the main factor in the improvement of the efficiency change index for the 2010/2011 period.

3.4. The determinants of innovation in management and innovation through human

Table 5 presents the results from the three panel data regression models accounting for random effects using the DEA estimates of the change in aggregate efficiency and its two components as dependent variables. The purpose of each regression model is to identify the parameters which have a significant impact as determinants of innovation in management and innovation through investment in human capital (the proxy for this is the presence of paid managerial input). The following model has been estimated using the efficiency change component and the pure efficiency and scale efficiency change sub-components respectively as dependent variables:

where ${\alpha }_{\iota } ~ iid\left(0, {\sigma }_a^2\right)$ and $u_{it} ~ (0, {\sigma }_u^2)$. When the $∆Eff$ component is considered as the dependent variable for the model (MD1), estimation results reveal a positive and statistically significant effect when the form of business is a company, compared to a partnership or sole trader (${\beta }_2=0.059, p-value<0.05)$. The magnitude of the effect is reduced (${\beta }_2=0.029)$ when the $∆PureEff$ factor is considered as the dependent variable in the model (MD2) but it remains statistically significant at $\alpha =0.05.$However, when the$∆ScaleEff$is considered as the dependent variable of the model (MD3), the effect although positive is no longer statistically significant ($p-value>0.05$. For both MD1 and MD3 the effect of an increase in the age of the farmer by one unit is positive across time and across individual farmers however, it is small in magnitude (${\beta }_{3MD1}=0.001$ and ${\beta }_{3MD3}=0.001$, $p-value<0.05$ and $p-value<0.10$, respectively).

In regards of the farmer being both the owner and the manager of the farm, Table 5 shows for all three models that the effect is positive (${\beta }_{4MD1}=0.070, {\beta }_{4MD2}=0.030, {\beta }_{4MD3}=0.038$) and, significant (p<0.05). Basic education (i.e. school only), has also a positive and significant effect ${(\beta }_5=0.030, p-value<0.01)$ for MD1 and for MD2 ${(\beta }_5=0.011, p-value<0.10)$ when compared with higher levels of education (i.e. degree, college and post-graduate studies). Interestingly, the effect of A-level or equivalent studies is negative for both the MD1 and MD3 models${ (\beta }_6=-0.035, {\beta }_6=-0.024, p-value<0.05)$. In terms of innovation through investment in human resources, the paid managerial input has a significant and positive effect in all three models and is the one parameter with the strongest in terms of magnitude of the coefficient (${\beta }_{7MD1}=0.132)$,indicating that farms with trained and experienced farm managers make a better use of the existing technologies and are able to retain this over subsequent periods.

With respect of farm size, results from Table 5 suggest that medium and small farms are less able to achieve a positive effect on all three DVs than large farms, but only medium size is significant at the 5% level$\left({\beta }_{8MD1}=-0.026, p-value<0.05\right)$. That is medium size farms drive a smaller change in $∆Eff$ than large farms and their average efficiency change across time is 0.026 less than of the average of large farms. Moreover, the results indicate that tenanted farms, on average, across time and across individuals drive a higher level of efficiency change when compared with owned farms for all three models (${\beta }_{10MD1}=0.026, {\beta }_{10MD2}=0.014, {\beta }_{10MD3}=0.013,{ p-value}_{MD1,MD2}<0.05, {p-value}_{MD3}<0.10)$. In addition, the estimation results regarding the level of specialisation (i.e. the business output derived from crop enterprises or other enterprises) indicate that the more diverse the farm business output is (less than 50% crop output) then, the average efficiency change across time and individual farm business is higher when compared to farm business where the percentage of crop output is more than 70% (${\beta }_{11MD1}=0.090, p-value<0.05)$. In contrast, a negative average change of efficiency is estimated by MD1 for the level of specialisation between 50% and 70% however, this is only statistically significant for MD1 ($p-value<0.10)$.

4. Discussion

4.1. What has been driving productivity change?

The decomposition of the MI of TFP has permitted further exploration of the drivers of productivity change in the specialist cereals farm sector over the study period (Färe, Grosskopf, Norris, et al., 1994) and, in particular, highlighted the impacts on innovation of improvements in management efficiency and technological progress (Darku et al., 2013; Song et al., 2016). Significant variation, i.e. periods of improvement and regression, in the MI of TFP is observed over the 11-year period. The fact that productivity change is, in some periods, negative, highlights the warning given by Glendining et al. (2009), that maintaining productivity per unit area is an important requirement for the future sustainability of arable farming systems. Of great relevance to policy makers is the finding that, over the study period, it is the innovation in management and in human capital that drives positive productivity changes in the UK cereal sector, rather than technological innovation.

4.2. Managerial and entrepreneurial efficiency

The geometric mean of the ΔEff component for the 11-year period is above unity. This strongly suggests that cereals farmers have been successful in adopting innovations sufficient to improve management and enhance human capital, so that they can solve problems and make relatively efficient resource allocations at the farm level (O’Donnell et al., 2017). Focusing on the two sub-components of theΔEff index, it is noted that it is the scale efficiency (ΔScaleEff) component that is actually driving positive productivity change. This observation confirms the conclusions of O’Donnell (2016), that over the study period, management efficiency gains have been driven largely by increasing the scale of operations, rather than by improving the quality of management.

According to Färe, Grosskopf, Norris, et al. (1994), analysis of the different components of the MI of TFP allows for the identification of the specific decision making units driving positive shifts in the efficiency frontier and also allows for the identification (and description) of the best performing farms (in terms of productivity change) in the sample. Positive productivity change occurs because innovation occurs on farms, leading to more efficient use of resources. Therefore, the best performing farms (in terms of productivity change) are, by definition, the most innovative. In theory, by describing these ‘leading’ farms using key variables, it would be possible to use them for benchmarking purposes, i.e. identifying those changes to management practices (i.e. innovations) that would improve the productivity of ‘lagging’ farms. In practice, however, benchmarking attempts break down, because farms that innovate, do not do so consistently, i.e. in some years they contribute to a positive shift in the productivity frontier and in others they pull it back. This is due to the fact that while a farm may innovate, perhaps through investment in training, in year 1 and so enjoy productivity improvements in years 2 and 3, if no further innovations are made lagging farmers catch up.

4.3. A word on economies of scale

A number of studies in the academic literature comment on the relationship between resource use efficiency and growth in farm size (see e.g., Thirtle et al., 2004, 2008). The conclusion of these studies is that greater productivity gains are observed in farms expanding the scale of their operations. Realising economies of scale at a farm level is therefore considered as an important means to improve the productivity of agricultural systems (Fuglie, 2008). In designing its own agricultural policies post Brexit, UK policy makers will be confronted with the challenge of ongoing market-driven consolidation in the agricultural sector and will have to take decisions on whether to allow this process to continue (Grant, 2016). Grant (2016) has shown that increasing scale does appear to increase management efficiency and lead to productivity gains. The question is, how long can this continue? The UK already has among the largest average farm sizes in Europe. Is there an optimal farm size, in terms of management efficiency, in a UK context, beyond which regression occurs? Whether this is the case or not, it is clear from this analysis that UK farmers have had a kind of monomania in looking to economies of scale as a means to increasing productivity, at the expense of alternatives. It should be clear to policy makers, therefore, that there are unrealised opportunities to further improve farm productivity through a policy focus on programmes designed to improve the quality of management. On the one hand, a good step in this direction would be government incentives to increase use of decision support tools for agriculture at farm level, as these have been demonstrated to significantly contribute to the improvement of productivity (agricultural outputs) and environmental outputs (Gadanakis et al., 2015a, 2015b). On the other hand, although a plethora of these tools are available , their rate of uptake by UK farmers is low (Rose et al., 2016).

4.4. Management and technology as drivers of innovation

The ΔEff factor and its two sub-components were used in a second stage regression analysis to explore the role that farm and farmer characteristics may have on level of innovation in management and capital investment in human capital. Although findings in the literature of productivity and technical efficiency at a farm level indicate that productivity, and hence efficiency, of farmers decreases with age (Läpple et al., 2015; Tauer & Lordkipanidze, 2000). We found a positive relationship between age and ΔEff . This positive effect is mainly derived from the positive and statistically significant scale efficiency change. The latter indicates that in terms of scale efficiency, the technically efficient farmer in the sample has the experience and knowledge accumulated over the years to capture the productivity gains associated with changing the scale of operations. Nowak et al. (2015) and Gadanakis et al. (2015a) both also report that length of management experience is positively correlated with improvements in productivity and technical efficiency. Furthermore, the positive relationship of basic education with management efficiency, in combination with the findings regarding age, suggests that experience and knowledge accumulated over the years can substitute for higher levels of education (Nowak et al., 2015; Pérez Urdiales et al., 2016).

However, these trends are not replicated when we consider ΔTech, where higher levels of education have a positive influence in improving the technology and positively shifting the technical efficiency frontier (Läpple et al., 2015). These observations means that policy interventions designed to encourage the adoption of innovation in the agricultural sector, must be nuanced enough to capture some of these apparent contradictions. For example, a policy instrument to encourage innovation based solely on vocational training/knowledge transfer may yield desirable results in terms of adoption of new technologies but have very little impact on innovation in management.

An area of particular interest is the impact of investment in innovation in human capital. In terms of efficiency change of MI of TFP, as indicated by the sign of the coefficient for the paid managerial input and for the case when the farm holder is also the manager, we expect further improvements in productivity and technical efficiency. The same is concluded for both the pure and scale efficiency change factors. These findings though require further investigation in order to explore further the management style and the decision-making process at a farm level. According to Pollak (1985) and Gallacher et al. (1994) professional management might be more conducive to productivity gains compared to management provided by a family member . This is mainly because professional managers tend to be better educated and exhibit greater managerial ability, including greater attention to detail than their owner-occupier counterparts.

Moreover, Byma (2010) also found that measured inefficiency is influenced by managerial ability and that older and more educated farmers show higher efficiency, as do larger farms which is in line with the findings presented here. However, Byma (2010) also suggests that more work is required in understanding the determinants of managerial ability. In addition, it is also important to consider the fact that farms operating under a company status are more likely to observe an improvement in technical efficiency- presumably because there is greater pressure to make profits. Once more, the management style and the decision-making process requires further understanding to capitalise these outcomes into specific strategies and recommendation for policy makers.

Higher levels of specialisation in terms of farm activities were expected to be linked to positive improvement in ΔEff. However, the results show an inverse relationship, i.e. farms with a percentage of crop output less than 50% are more likely to realise a positive efficiency change. This is mainly due to the scale-mix efficiency change. Since the output considered in the DEA linear programming problem is the farm business output (excluding subsidies) it can be said that the more diversified the business is, then the lower the risk to the farm of shocks to any one activity. Nevertheless, for the purposes of this study further investigation is required to validate these findings. This validation might also benefit from the availability of different indicators of farm specialisation as the FBS indicator currently available is rather crude, with limited coverage.

In addition, in terms of ΔTech , equity and capital intensity are positively associated with technology change and shifts of the efficiency frontier. However, their impact is very small and hence further disaggregation of the two ‘pure’ and ‘scale’ components is not required. For example, we could disaggregate fixed capital into machinery and building and further into other technologies adopted at a farm level. Nonetheless, this is an important finding since it indicates that the future development of farms and their ability to realise technological progress is linked to their investment activity and accumulation of capital (Nowak et al., 2015). This conclusion is supported by Mugera & Nyambane (2015), who have shown that capital investment has a positive impact on technical efficiency and hence could contribute to the overall productivity improvement of the farm.

4.5. A word on entrepreneurial competencies in agriculture

Entrepreneurship plays a crucial role in the agricultural sector, linking to farm management, human capital innovation, and productivity in several ways (Yoon et al., 2021). When business development and innovation is linked to farm management it is necessary to consider the functions of planning and control of farm systems. Thus, explore the dynamics in decision making, resource allocation and risk management. Entrepreneurs in agriculture make critical decisions regarding crop selection, land use, resource allocation, and technology adoption (Savastano et al., 2022; Thephavanh et al., 2023; Yoon et al., 2021). Effective farm management involves strategic planning, risk assessment, and efficient utilization of resources to maximize productivity. This involves optimizing the use of land, water, fertilizers, and other inputs to ensure sustainable and profitable farming operations. Furthermore, agriculture is inherently risky due to factors like weather conditions, market fluctuations, and pest outbreaks. Therefore, entrepreneurial skills are essential for managing and mitigating these risks through diversification, insurance, and other risk management strategies.

Human capital innovation is a driver of technology adoption, diversification, training and skill development. Farmers acting as entrepreneurs, lead in the adoption of innovative technologies including the use of precision farming techniques, data analytics, and other advanced tools to improve productivity and reduce resource wastage (Rahman et al., 2022; Zezza et al., 2017). Moreover, entrepreneurial farmers invest in the continuous training and skill development. This enhances the human capital in agriculture by improving the efficiency and effectiveness of farm operations. In addition to the adoption of new technologies, entrepreneurial farmers are also prone to explore new crops, farming techniques, and value-added products. Thus, linking this back to training and skills development in the agricultural sector since these activities require a well-trained and adaptable workforce capable of embracing and implementing new ideas and technologies.

Entrepreneurship in agriculture drives efficiency improvements by constantly seeking ways to increase productivity and minimize waste throughout farm operations, encompassing streamlined supply chains, optimized logistics, and judicious use of inputs. Entrepreneurs in this sector also maintain a market-oriented approach, producing goods in line with consumer demand to enhance competitiveness and boost sales, ultimately leading to increased profitability (Poudel & Pandit, 2020; Warr & Suphannachart, 2021). In summary, agricultural entrepreneurs foster innovation by actively adopting progressive practices, such as experimenting with novel crop varieties, implementing sustainable farming techniques, and leveraging data-driven approaches to optimize overall production processes.

5. Conclusions

This study used a panel data set of cereal farms derived from the FBS in order to assess variations in productivity change in the sector based on the estimation of a MI of TFP. Both ΔEff and ΔTech indicators were employed, along with their sub-components to explore further the drivers of innovation in management and innovation through investment in human capital. The MI of TFP revealed significant variation in productivity over the 11-year period, but with a moderate overall improvement over the whole period. One limitation of the index used here is the fact that is not accounting for a sequential productivity change and hence future work in the area will need to consider recent developments in this area, for example by O’Donnell et al. (2017). Moreover, although the FBS is a comprehensive and detailed database it lacks information on specific management practices and decision-making process that might be used as indicators of innovation in management. Without these, it must be conceded, there is no way of providing data on drivers of innovation detailed enough to inform policy design focussed on encouragement of innovation. In terms of future research, a good starting point would be a study to investigate the possibility of including in the FBS data collection exercise measures for a far wider range of managerial behaviour, together with follow-up analysis of the impacts of these varied behaviours on different types of innovation leading to improvements in productivity. A parallel investigation would be required to gain further insights into the role of technology investments in driving shifts of the efficiency frontier and technological progress.

In addition, new agricultural policies, resulting from Brexit, will need to focus on ways to capitalise on the existing knowledge and experience of farmers to design educational programmes that facilitate innovation through management efficiency and a move away from the traditional ‘bigger is better because costs are spread’ mentality to a more nuanced approach which asks, what is the optimal scale for my farm to maximise productivity gains? This more nuanced approach would, of course, be more cognitively challenging, and such decisions would need to be better supported by advisory services and planning tools, but Government can certainly do more to encourage, or incentivise, farmers to become better informed, both concerning current research, and the decision-support community.

Recent studies highlight these challenges and opportunities post-Brexit (see e.g., Gittins et al., 2020; Ojo et al., 2021). Some emphasize the substantial contribution of CAP direct payments to farm business income and the vulnerability of farms to their removal, underscoring the need for new policies that mitigate these risks and enhance farm productivity (Ojo et al., 2021). Also, it has been discussed the challenges and uncertainties faced by arable farming in the UK due to Brexit, highlighting the need for resilience and competitive strategies in the new policy landscape. Regarding livestock management, technological tools can assist farmers in adapting to policy changes and developing growth strategies (Gittins et al., 2020). Overall, it has been demonstrated the importance of adapting to the post-Brexit era with policies and tools that support farmers in optimizing productivity through informed decision-making and efficient management practices. They suggest that a shift in mindset from traditional approaches to more nuanced strategies, backed by government support and technological innovation, is key to the long-term success of UK agriculture in the global market.