Comment on HEA’s “Towards a Performance Evaluation Framework: Profiling Irish Higher Education”

In Ireland, government policies have given increasing importance to knowledge transfer activities in the higher education sector, as means of driving the knowledge-based economy. There lacks, however, much empirical evidence showing the width and depth of interactions between Irish universities and businesses.

Efforts such as the Community Innovation Surveys are essential to help us understand how enterprises in Ireland source knowledge and build innovation. Nevertheless, these surveys are mainly focused on the business sector and not designed to reveal the full picture of university-industry interactions. An urgent issue which needs to be addressed now in Ireland is to start building empirical evidence on the university side, i.e. how academics engage with businesses and the wider community.

Furthermore, when focusing on university-industry relationships, there has been a shift of research focus from the knowledge transfer of intellectual property (IP) to multifaceted channels and mechanisms of knowledge exchange. As many studies have asserted, IP-related activity is too specific and narrow, a wider view of interactions between academics and businesses is better capable of capturing the comprehensive roles of universities.

In the U.S. and Canada, the Association of University Technology Managers (AUTM) has been releasing its annual surveys for a couple of decades, with its special focus on licensing, patenting, and establishing spin-out companies. The Higher Education Funding Council for England (HEFCE), on behalf of all UK HEIs and the national funding bodies, has been publishing the Higher Education-Business and Community Interaction Survey (HE-BCI) since the academic year 1999/2000. In comparison to the AUTM surveys, the HE-BCI surveys take a more holistic approach and consider the following types of engagements: collaborative research, contract research, consultancy research, facilities and equipment related services, courses for business and the community, and IP-related activity.

Even so, there are critics that these six types of activities only account for a small share of partnerships through which knowledge is exchanged between organisations. More recently, a large research project conducted by the Centre for Business Research (CBR) at the University of Cambridge in 2009 grouped possible modes of interaction into the following categories: people based, problem solving, community based, and commercialisation. In total, the research team considered 27 types of knowledge exchange activities.

As mentioned earlier, there has been little research compiling these data in Ireland, except for a recently report released by the Higher Education Authority (HEA) in December 2013. Entitled “Towards a Performance Evaluation Framework: Profiling Irish Higher Education”, the report represents a milestone in Irish higher education policy. This study, the first of its kind in the country, has attempted to profile Ireland’s higher education sector in the three dimensions of its core mission: teaching and learning, research, and engagement.

The main strengths of the report lie in two aspects. First, it appreciates the diversity of higher education institutions in Ireland and groups them into universities, institutes of technology, and colleges. To develop a framework in which to evaluate the quality of engagement, this is an important step to take as it takes account of institutions’ historical, geographical, socio-cultural specificity and diversity, which would to a large extent determine the type (width), intensity (depth) and locationality (length) of academic engagements.

Second, it puts much effort in collecting data from a variety of sources, in the aim of building a comprehensive performing evaluation framework. In particular, it presents data on topics such as student enrolment, research funding, staff profile, as well as knowledge transfer activities. More importantly, it indicates, in its framework, which data is currently available, under development, or needs to be articulated in the future, the last type of which is what much of the engagement dimension has been labelled as.

Whilst the report has an intensive literature review, in which it searches for and describes the indicators used in different countries when evaluating research, the framework it proposes in the end still has many limitations. In particular, the indicators proposed in the knowledge exchange section are not quite clearly defined. For example, it calls for data collection in joint research programmes with enterprise, but does not specify the nature of those programmes, such as whether they are collaborative research or contract research. The nature of research programmes could have a big impact on what type of knowledge is exchanged, and how intensively that knowledge is exchanged. Also, the framework suggests HEIs to report their involvement in the community, which, again, is too broad and may be really difficult, if not impossible, for all HEIs to provide effective and comparable answers.

All in all, this work has pointed us in the (right) direction of building evidence-based practice in Ireland’s higher education sector by, at least, showing some evidence.



For the purpose of Eurostat and EU Structural Funds, two NUTS 2 regions of Ireland were established and came into effect in July 1999. One is the Southern and Eastern region, and the other one is Border, Midland and Western region. Both regions had “Objective 1” status in the early years, meaning they were both qualified to receive the Structural Funds, which are one of the financial instruments of European Union regional policy intended to narrow the development disparities among regions and Member States. The “Objective 1” status of the Southern and Eastern region ended in 2005, one year earlier than the Border, Midland and Western region.

These two regions may be similar in terms of size, but the Southern and Eastern region shows a much higher level of population density as well as economic activity measured as GDP (PPP) per capita. Figure 1 below analyses how the GDP per capita of the two regions compares against that of Ireland between 1996 and 2010.

Figure 1 GDP per capita as % that of Ireland by region, 1996-2010, Ireland=100


Source: OECD (2014), Main Science and Technology Indicators.

Note: Per capita, US $ constant PPP, constant (real) prices (year 2005).

Clearly, Ireland has an unbalanced economy, which could be termed as a “North South Divide”, similar to what has also been found in the UK. In both countries, the southern regions show more advanced economic development than the northern regions. In 1996, GDP per capita of the Border, Midland and Western region in Ireland was just under three quarters of that of Ireland. More importantly, the gap between this region and the rest of the country was further widened across the years. Up to 2010, its GDP per capita was less than two thirds of the national average.

It may not come as a big surprise that the Southern and Eastern region is more developed in the Irish context. Indeed, it is this part of the country that houses the majority of firms, in particular those high-tech multinational companies. In 2012, OECD published its Cluster Scoreboard which compared the entrepreneurship of 80 local enterprise clusters in two key innovative sectors with important roles in local economic growth: high-technology manufacturing and knowledge-intensive service activities. Three of the four clusters from Ireland being selected in the report were from the Southern and Eastern region, including: Bio-pharma cluster in Dublin; Med-Tech cluster in Cork; and ICT cluster in Dublin. The Atlantic Technology Corridor from Galway to Shannon was the only one located within the Border, Midland and Western Region.

What is a bit surprising is the fact that the overall economic development level of the Border, Midland and Western region has lagged further behind the national average during the recent decade. Since the publication of the National Development Plan 2000-2006, the Irish Government has started addressing the regional disparities, with more and more resources being dedicated to supporting infrastructure development and business development in the weaker areas.

In Figures 2 to 5 we illustrate the R&D landscape in Ireland between 2002 and 2009. As Figure 2 demonstrate, the total R&D expenditure performed within the Border, Midland and Western region accounted for under 20 per cent of the total R&D expenditure in Ireland in 2009, showing a significant increase from less than 14 per cent in 2002.

Figure 2 Total R&D expenditure as % that of Ireland by region, 2002-2009


Source: OECD (2014), Main Science and Technology Indicators.

Note: Million US $ constant PPP, constant (real) prices (year 2005).

Figure 3 R&D expenditure performed by the business sector as % that of Ireland by region, 2002-2009


Source: OECD (2014), Main Science and Technology Indicators.

Note: Million US $ constant PPP, constant (real) prices (year 2005).

Figure 4 R&D expenditure performed by the government sector as % that of Ireland by region, 2002-2009


Source: OECD (2014), Main Science and Technology Indicators.

Note: Million US $ constant PPP, constant (real) prices (year 2005).

Figure 5 R&D expenditure performed by the higher education sector as % that of Ireland by region, 2002-2009


Source: OECD (2014), Main Science and Technology Indicators.

Note: Million US $ constant PPP, constant (real) prices (year 2005).

Figures 3, 4 and 5 show more or less similar trends to what is revealed in Figure 2. Therefore, although the Border, Midland and Western region has made gradual progress in terms of performing R&D activities, those efforts have not turned into economic benefits or final outputs. Understandably, it takes time for a region to adjust its industrial structures, and maybe needs more patience to build a functional system of innovation, in which institutions actively engage with each other in knowledge exchange and collaboration.

At the first glance, Figure 1 and Figures 2 to 5 may have shown something contradictory, but one needs to bear in mind that, although it is generally claimed that R&D expenditure is associated with economic development, that impact has to be examined within a longer period, depending on the specific context of regions. Ireland, as small as it may be, still has localities which present heterogeneous features of economic development, industrial structures, and organisational relationships.


While the Irish Government has made a clear commitment to supporting the research and development (R&D) sector and to building the smart economy in the past decade, now is an appropriate time to know where it currently stands in comparison to its counterparts. In particular, we want to compare Ireland’s R&D levels (GERD, BERD, HERD, and GOVERD) in relative to the levels of UK, EU-15, and OECD respectively.

Our analysis mainly draws upon the OECD’s Main Science and Technology Indicators database, as well as the Forfás’ publications related to R&D data in Ireland since 2001. These publications include, among many others, Research and Development in Ireland 2001 – At a Glance, Statistics at a Glance 2011, State Investment in Research and Development 2011-2012.

According to a study by the Centre for Business Research (CBR) in 2011,

‘Ireland’s economy is unusual in structure. The majority of its manufacturing industry is foreign owned with very high recorded profits, of which large amounts are repatriated each year resulting in a wide divergence between GDP and GNP. For this reason, when we refer to output in Ireland, we are referring to GNP.’

Therefore, our analysis considers GNP in the case of Ireland when calculating the shares of R&D expenditures; in the cases of all the other countries, it also uses GDP. The time frame chosen for the comparisons spans two decades since 1993, which enables us to understand how Ireland’s R&D levels have evolved in a longer period.

Furthermore, in order to focus on the relative performance of Ireland to the others, in each comparison, we have set all the R&D levels (GERD, BERD, HERD, and GOVERD) of that country (such as the UK in Figure 1) at the value of 100. The values shown by Ireland were the ratios between the R&D levels of the two countries. For example, Ireland’s GERD in 1993 was valued at 66 while the UK’s GERD in 1993 was valued at 100, meaning Ireland’s GERD as a percentage of GNP was at a much lower level than UK’s GERD as a percentage of GDP in that year.

The R&D intensity data has its limitations; for example, even if the real R&D expenditure remains the same, a fall in GDP (or GNP) could lead to an increase in the intensity. In the case of Ireland, Forfás has clearly pointed out in its Business Expenditure on Research and Development (BERD) 2011/2012 that,

‘The increase in BERD intensity from 2009 to 2011 is entirely due to a fall in GNP as BERD remained unchanged during that period.’

Considering the significant impacts of the 2008 Economic Crisis on the outputs of so many countries, Ireland may not be alone in this regard. One should bear in mind how the real expenditures of R&D in a country have changed, in addition to the development of its R&D intensity.

Figure 1 below shows how Ireland compared with the UK in investing in R&D activities between 1993 and 2012.

In general, Ireland started at a much lower level than the UK, has been catching up ever since, and showed better performance in the most recent years. The value for Ireland’s GERD in 2008 was 97 – still lower than the UK level – but it increased to 114 in 2009, leading the UK for the first time. In 2012, that value stood at 123, while the values for Ireland’s HERD and BERD were 120 and 135 respectively. During the past two decades, however, Ireland’s GOVERD has always been at a lower level than that that of the UK, although it has also narrowed the gap between the two.

Figure 1 Ireland’s R&D levels in comparison to the UK, 1993-2012, UK=100


Sources: OECD (2014), Main Science and Technology Indicators.

In figure 2 we present the comparison between Ireland and EU-15. EU-15 area countries are: Austria, Belgium, Denmark, Finland, France, Germany, Greece, Ireland, Italy, Luxembourg, Netherlands, Portugal, Spain, Sweden and United Kingdom. Ireland’s R&D levels have also been catching up with EU-15 ones in all aspects but GOVERD intensity. Its GERD level was the same to that of EU-15 in 2012, while its BERD and HERD levels were higher than EU-15 in that year.

In the first decade – 1993-2003 – Ireland’s BERD showed a relatively smaller gap between EU-15 than its HERD did. Since 2004, Ireland’s HERD intensity increased sharply to catch up with the HERD level of EU-15 in a few years. In 2009 – one year after its HERD intensity became higher than that of EU-15 – BERD intensity in Ireland was also above the EU-15 average.

The Ireland-EU-15 comparison obviously shows a two-stage process, in the first of which BERD was relatively stronger than HERD while in the second HERD became relatively stronger than BERD. A plausible explanation for this could be related to the great effort given by the Irish Government in the early 2000s in developing world class research and expanding its investment in universities and other third level institutions.

Figure 2 Ireland’s R&D levels in comparison to EU-15, 1993-2012 (EU-15=100)


Sources: OECD (2014), Main Science and Technology Indicators. 

Figure 3 below compares Ireland’s R&D levels with those of OECD countries. An obvious finding is that, Ireland’s HERD intensity has always shown better relative performance than its BERD and GERD, as its HERD line has always been above its BERD and GERD lines. This is an interesting point, as it implies that in comparison to the OECD average, Ireland’s relative R&D strength has consistently been lying in its higher education sector. That figure 2 and figure 3 showing different patterns suggests it matters which country/countries we choose to compare Ireland with.

Ireland’s HERD intensity became higher than the OECD average in 2004 for the first time, and increased even further in the recent years, standing at the value of 125 in 2012 (OECD=100). BERD and GERD of Ireland have changed in a similar pattern – the two lines to a large extent overlap. In these two indicators, Ireland did not essentially catch up with its OECD counterparts until 2009, before when the values of its BERD and GERD fluctuated between 60 and 75, much lower than the OECD average. Similar to what figures 1 and 2 have shown, Ireland’s GOVERD has become further behind the OECD countries. It demonstrates a continuous deterioration of its government expenditure on R&D in relative to many other countries. 

Figure 3 Ireland’s R&D levels in comparison to OECD, 1993-2012 (OECD=100)


Sources: OECD (2014), Main Science and Technology Indicators.


From the late 1990s onwards, the Irish Government embarked on a strategy of significantly enhancing the scientific, technological and innovative capacity of the country, as part of its aim to develop as a knowledge and innovation-based economy. In a way, these endeavours were built on many years of support from the European Union (EU) mainly through its Structural Funds, and were originated at a period when the so-called ‘Celtic Tiger’ growth spurt seemed to have slowed down, if not ended. Ireland has seen, since 2000, the release of a series of national policies targeting investment in science and technology.

The National Development Plan 2000-2006, which sets out an overall development strategy for the country, has as one of its objectives the consolidation and improvement of Ireland’s international competitiveness. It has been argued that this stage saw a policy shift towards funding research in universities and other third level institutions, as a means of maintaining sustainable national economic growth. Following the implementation of the NDP, the Science Foundation Ireland (SFI) was founded in 2001 to undertake and support strategic research of world class status. As well, the Programme for Research in Third Level Institutions (PRTLI), administered by the Higher Education Authority (HEA), was further expanded, representing a significant commitment of State resources to research in higher education.

The importance of science, technology and innovation (STI) was emphasised again in the Strategy for Science, Technology and Innovation 2006-2013, which represents Ireland’s first comprehensive strategic approach to developing STI on a whole of Government basis. The Strategy seemed to have proposed a wider perspective on STI than what was addressed in the NDP six years ago. In particular, the Strategy called for actions not only in delivering world class research, which was strongly recommended in the NDP, but also in areas such as capturing, protecting and commercialising ideas, strengthening R&D for enterprise, innovation and growth, and facilitating all-Island and international R&D collaboration.

In March 2012, the Research Prioritisation Steering Group launched the Report of the Research Prioritisation Steering Group. The idea of the Report was to build on the strengths that have emerged from considerable public and private investment in STI during the last decade, and to target the priority areas that will become the focus of future State investment in research and innovation. In total, the Steering Group identified 14 priority areas, incorporating input from the research community, the enterprise sector and research funding departments and agencies.

It could be therefore concluded that the approach of developing R&D undertaken by the Irish Government has become more ‘holistic’ on the one hand and more ‘specific’ on the other hand. A ‘holistic’ view addresses the importance of various types of innovation actors in the economy, including universities, enterprises, and public and private research organisations; furthermore, it highlights the close interactions and collaborations between those actors. A ‘specific’ view attempts to identify the key research areas that have the greatest potential to deliver sustainable economic return; and it guides the concentration of Irish research and development effort aligned with national priorities.

To a certain degree, the ‘holistic’ aspect of Ireland’s R&D development approach is actor-based, while the ‘specific’ aspect is sector-based. The ‘holistic’ aspect forms the foundation of the approach, which clearly states the role of innovation systems. The ‘specific’ aspect forms the strategy of the approach, which calls for collaborations in a number of specified areas.