Evaluation and Impact Analyses

The Houska Prize, awarded annually since 2005, is the highest endowed private prize for applied research in Austria. The aim of the evaluation is to determine the extent to which the Houska Prize has rewarded outstanding science and research projects that have led to economic success. To this end, among other things, publication and patent data will be evaluated, qualitative interviews, an online survey and case studies will be conducted. Conclusions and recommendations for the future design of the prize will be drawn from the results.

Project duration: 2022/12 - 2023/10
Client: B&C Foundation
Project partner: Inspire Research
Contact: Eva Heckl

The Directorate for Innovation and Technology of the Austrian Federal Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK) has a heterogeneous portfolio of thematic priorities, RTI policy instruments and accompanying measures, and implements these with the support of agencies (e.g. FFG, aws) via strategic governance of public research centres (i.e. AIT, SAL). With the development of an overarching monitoring system, the aim is to create an efficient and transparent system for measuring impact and supporting knowledge development to facilitate strategic intelligence at the department level. The various interventions of the BMK - in particular the funding of programmes, networking and institutions – are to be recorded and be verifiable concerning their effects.

Project duration: 2022/12 – 2023/06
Client: Federal Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK)
Project partner: technopolis group
Contact: Peter Kaufmann

The Austrian Research Promotion Agency FFG commissioned this short study in order to examine the status quo with regard to the representation of women in the projects it funds on the one hand, and on the other hand to analyze which measures an agency like the FFG can take or which measures already implemented seem most promising.

The project undertakes a comprehensive evaluation of the funding programmes License.IP (2013-2020) and IP.Market (2017-2022) of Austria promotional bank (aws). The evaluation serves to systematically record, evaluate, and reflect on the strengths and weaknesses of the programmes to be able to make any necessary improvements and adaptations for possible future funding of IP transactions. The evaluation thus primarily has a supporting character for the management of future programmes and informs the programme commissioners about the progress made so far.

Project duration: 2022/09 – 2023/04
Client: Austria promotional bank (aws)
Project partner: Inspire Research Beratungsgesellschaft m.b.H.
Contact: Harald Wieser

The trustworthy use of artificial intelligence (AI) has the potential to change the economy in key future industries for the benefit of society and to positively influence the positioning of Austria and Europe in international competition. First, the terminology is clarified in an overview study and possible potentials and challenges are considered. The focus is on the necessary digitalisation of the areas of production, mobility and medicine.

In addition, the climate friendliness of AI and its potential for climate protection is increasingly gaining interest. Against this background, a second study provides a knowledge base for assessing the climate protection potential of AI applications and provides an overview of existing developments and practices in business support.

The policy insights are a result of the accompanying research in the context of the accompanying monitoring and evaluation of the call "aws Digitalisation - trustworthy artificial intelligence" on behalf of aws - Austria Wirtschaftsservice Gesellschaft mbH.

Client: Austria Wirtschaftsservice (AWS)
Contact: Peter Kaufmann

The study analyses which factors influence the commercial exploitation success of applied research projects of enterprises. From this, recommendations for action are derived as to how the agency can further develop its services to promote the realisation of exploitation potential. The analysis thus focuses on a core function of the FFG - selecting projects with high potential and supporting them in their implementation.

Project duration: 2021/09 - 2022/02
Client: Austrian Research Promotion Agency (FFG)
Contact: Peter Kaufmann

The subject of the evaluation is Switzerland's space policy as it has developed over the past thirteen years on the basis of the institutional strategy "Swiss Space Policy 2008". The appropriateness and effectiveness of selected guidelines are described and assessed, taking into account past and future developments. Furthermore, the evaluation serves as an information base for the development of a future Swiss space policy. Part of the evaluation is, among other things, the analysis of the complex landscape of actors including the start-up ecosystem around Swiss space policy.

Project duration: 2021/09 - 2022/10

Project partner: econcept

Client: Swiss State Secretariat for Education, Research and Innovation (SERI)

Contact: Peter Kaufmann

The funding programme "EXIST – university based business start-ups" aims to increase the number and success of technology-oriented and knowledge-based start-ups and to improve the start-up climate at German universities and non-university research institutes.
This accompanying evaluation focuses on the EXIST - Potentials Programme from 2018, the evaluation of EXIST - Start-up Grants and EXIST Research Transfer from 2020 and the overarching consideration of the three programme lines from 2022 to 2025.
EXIST is a funding programme provided by the German Federal Ministry for Economic Affairs and Climate.

This accompanying and ex-post evaluation is implemented on behalf of the Federal Ministry of Finance based on the evaluation plan approved by the European Commission in the period 2022 to 2025.

Due to the accompanying character, there is an ongoing examination of implementation processes, target group outreach, as well as an in-depth analysis of the effects in the target groups and for Germany as a business location. This is implemented methodically using a mix of qualitative and quantitative (econometric) methods.

This evaluation will examine the effects that the "Stadt der Zukunft" (“City of Tomorrow”) programme has achieved since its inception. The evaluation is based on extensive surveys of research project participants and various multipliers from application areas such as civil engineering and urban planning. The evaluation thereby creates an important basis for the design of future research funding.

More information on the project (in German)

Project duration: 2021/11 - 2022/08
Client: Federal Ministry for Climate Action (BMK)
Contact: Harald Wieser

Trustworthy artificial intelligence (AI) not only has the potential to change the economy in key future industries as a new basic technology, but could also positively influence the positioning of Austria and Europe in international competition.

In this project, the Austrian Institute for SME Research accompanies and examines the underlying assumptions, potentials, and the design and implementation of the pilot initiative trustworthy AI on half of the Austrian promotional bank - aws.

Project duration: 06/2021 - 12/2022
Client: aws - Austria Wirtschaftsservice GmbH
Contact: Peter Kaufmann

The climate neutrality of the mobility system is more urgent than ever. Therefore, it is also necessary to use suitable monitoring to show the progress of research projects up to their implementation in social practice and thus also to identify supporting and hindering influencing factors.
With this project, Austrian Institute for SME research, in cooperation with the BMK, AustriaTech, FFG and ongoing projects in the mission-oriented programme Mobility of the Future (MdZ), is developing proposals for impact monitoring, accompanying projects and the programme within the framework of an ‘ImpactLab’.
This will ensure a systematically organised learning process that meets the conceptual demands of modern mission orientation and identifies possible barriers to implementation. The proposals will be continuously tested and implemented as part of the implementation of the RTI Strategy Mobility 2040 from 2021.

The overall objective of this study was to gather evidence on the effectiveness of public support for innovation in Europe. A key finding of the study is that SMEs in the EU continue to face barriers to innovation; for micro-enterprises, these are primarily financial barriers; for small and medium-sized enterprises, access to skilled labor is the most significant barrier. On the other hand, with regard to the effectiveness of support services, different contextual conditions in the individual member states make it difficult to draw a blanket conclusion, but most SMEs see public support as crucial for carrying out their innovation activities.

Project partner: Centre for Strategy & Evaluation Services (CSES)

Client: Executive Agency for Small and Medium-sized Enterprises (EASME)

Study Download

The impact monitoring of RTDI funding for Austrian companies and research organisations, managed by the Austrian Research Promotion Agency (FFG), assesses direct impacts of the funded projects on RTDI inputs, behavioural changes, and outputs. The results of the analyses serve as an input for the further development of the portfolio of the funding agency.

The analysis covers the design and implementation of the 'Funding Announcement for Applied Non-nuclear Research Funding' and the guideline for 'Funding of Energy Transition Real Laboratories' within the 7th Energy Research Programme 'Innovations for the Energy Transition' of the Federal Ministry for Economic Affairs and Energy. The evaluation intends to provide a basis for steering and continuously improving the programme. The focus of the analysis is a proportionality check of the aid as well as the target achievement and effectiveness of the measures. Within the framework of the comprehensive evaluation design, a variety of qualitative and quantitative methods are used.

Within the framework of the project, the klimaaktiv climate protection initiative was evaluated together with the Wuppertal Institute for Climate, Environment and Energy. Internal and external reports, personal interviews with key stakeholders, an online survey of the partner network and a strategy workshop formed the basis for the evaluators. They evaluated the klimaaktiv program period 2013 to 2020. Based on the results, indications were developed on how to design the program phase until 2030. The results show that klimaaktiv is effective, well organized and has built up a powerful network. The activities are to be expanded and intertwined with upcoming measures and funding.

Project partner: Wuppertal Institute for Climate, Environment and Energy

Client: Federal Ministry of Agriculture, Regions and Tourism

Further information & study download

The study explores the competitiveness of the engineering industries in the EU against its major competitors USA, Japan, South Korea and the BRICs countries. It compares regulatory frameworks of EU and major third country competitors in the field of engineering industries. Furthermore, it analyses the impact of digitalisation on the EU engineering industries and its effects on the global market position. The study also provides a strategic outlook and an assessment of the challenges and opportunities ahead for the European engineering industries.

The evaluation analyses the instrument "BMVIT endowed professorship" and develops recommendations for the further development of the instrument. The focus is on the organisational and structural framework conditions for endowed professorships from the perspective of the sponsor and the universities. The evaluation focuses on questions regarding the appropriate design of the instrument with regard to the intended funding objectives. In addition, the integration of the instrument into the national funding environment and the positioning of endowed professorships in the Austrian university and higher education landscape is examined.

German Report

Within the framework of the "funding focus on talents" ("Förderschwerpunkt Talente), (potential) researchers are funded in the application-oriented, scientific and technical RTI area. The funding instruments are aimed at children and young people (internships for pupils, regional talents) as well as researchers (career grants, job exchange) and address equal opportunities in research (FEMtech student internships, FEMtech career (check), FEMtech research projects). The Research Partnerships ("Forschungspartnerschaften") program supports dissertation projects in companies and non-university research institutions. The evaluation focuses on the analysis of the funding concepts, the implementation and the effects of the programmes.

Project duration: 05/2020 - 02/2021
Client: Federal Ministry for Climate Protection, Environment, Energy, Mobility, Innovation and Technology
Contact: Eva Heckl

Downloads (only executive summary available in English):

• Report Talente - Evaluierung des Förderschwerpunkts
• Report Forschungspartnerschaften – industrienahe Dissertationen - Programmevaluierung 

The evaluation of this support programme of the Federal Ministry of Economic Affairs and Energy (BMWi) covers the period 2011-2018. Programme implementation is largely embedded in the Federal Government's space strategy and Germany's involvement in the European Space Agency ESA.

The study focuses on the achievement of the objectives and the effects of the programme, as well as on the cost-effectiveness of programme implementation by the Space Management (RFM) of the German Aerospace Agency (DLR). The analysis focuses on questions of internal organization and governance of programme implementation, as well as the implementation of large-scale missions. Apart from the actual programme performance, we also develop a concept for an increased future evaluability of the programme.

Contact: Mag. Peter Kaufmann

Project partners: Dr. Christiane Kerlen Evaluation und Beratung, Institut für Innovation und Technik (iit) in der VDI/VDE Innovation + Technik GmbH

Download the German Report (including an English Executive Summary)

The German government wants to use the R&D tax credit to help increase the share of R&D expenditure in gross domestic product to 3.5 % by 2025 and to supplement the existing project funding. In this project, the Austrian Institute for SME Research and the Institute for Advanced Studies (IHS) as well as the Ifo Institute Munich investigate the possible databases and methodological challenges for a future evaluation of the measure. The results lead to a coordinated evaluation plan, which should allow a deep analysis of the impacts.

In this project, The Austrian Institute for SME Research (KMFA) supports Innosuisse in developing a concept for the agency's impact analysis. In this context, impact logics for the most important funding instruments and the agency as a whole are developed. These form the basis for the development of an indicator system that can provide information on its own activities and the resulting effects to the client, policymakers and the public in a clear systematic manner.

The outcomes of the impact analysis concept are used for strategic optimisation and learning for the whole organisation.

The evaluation of the German Central Innovation Programme for SMEs (ZIM) with its three R&D programme pillars - individual projects, cooperation projects and innovation networks concerns one of the most important instruments of innovation policy in Germany.

The evaluation focused on the programme period 2015 to mid-2018 and integrated a theory-based approach with a strong empirical basis. Methodologically, the evaluation implemented a coordinated set of analytical-quantitative and qualitative approaches, such as a control group approach as well as a broad set of qualitative methods, including the analysis of spillover-effects of the different programme pillars.

This evaluation commissioned by the Federal Ministry for Climate Protection, Environment, Energy, Mobility, Innovation and Technology (BMK, formerly BMVIT) examined the Austrian space commitment in the period 2012 to 2018 and derived corresponding recommendations for further development. This includes not only the design of the future space strategy, the relevant commitment at ESA and at the level of the European Union, but also the Austrian Space Applications Programme (ASAP), Austria's national space programme.

The evaluation was coordinated by the Austrian Institute for SME Research (KMU Forschung Austria) and implemented in close cooperation with the Institute for Innovation and Technology (iit) in VDI/VDE Innovation + Technik GmbH and inspire research GmbH.

The study on behalf of DG GROW assessed the feasibility of including societal criteria (environmental, innovation, employment-related) in upstream space procurement within the European Union Space Programmes, Copernicus, Galileo and EGNOS. The study required carrying out three tasks: Task 1) Investigate the legal feasibility of including societal criteria in the procurement of infrastructures for the EU Space Programmes, Task 2) Estimate the impact of including selected societal criteria and Task 3) Draw conclusions regarding the most appropriate societal criteria to be included in future procurement.

The accompanying impact analysis and evaluation of the Swiss "CTI Entrepreneurship" Training Programms assesses its conception, implementation and short to medium-term effects. The analysis, carried out by the Austrian Institute for SME Research in cooperation with B, S, S Volkswirtschaftliche Beratung AG in Basel, is based on theories of social psychology and their empirical implementation against the background of heterogeneous support infrastructures in the Swiss regions.

This accompanying evaluation of the implementation of the European Regional Development Fund (ERDF) 2014-20 in Austria involves a consortium of six partners, and applies a broad evaluation design.

From the variety of methodologies applied in this evaluation, the Austrian Institute for SME Research implements, apart from some qualitative analyses, a conditional difference-in-difference to investigate the impact of ERDF subsidies on enterprises.

This project implements the accompanying research of a technical research project, which shows the economic and technical potential of the external component activation for the renovation of existing buildings with prefabricated façade elements and defines the respective conditions and develops new methods. Above all, the accompanying research includes analyses along the project-relevant value chain, such as analyses, of needs as well as potential impacts etc.

Climate and Energy Model Regions (KEM) is a programme of the Austrian Climate and Energy Fund (KLIEN), which supports regions in the development and implementation of climate change-related measures. The evaluation focuses on the design, implementation, the outcomes as well as the impacts of the program, and derives recommendations for the follow-up period.

The interim evaluation analyses the quality of the mission-oriented RTDI programme „Mobilität der Zukunft“ (mobility of the future). Main focus of the evaluation is on the analysis of the programme design and the processes with special attention to the learning processes within the programme. The output and the preliminary outcomes of the programme are reviewed compared to the objectives (targets and indicators). At the same time indicators of the programme selected for an internal impact assessment are evaluated. The evaluation is based on a mix of qualitative and quantitative methods.

The subject of the project is the evaluation of the programme(s) for the promotion of studies for commercial, international technology transfer" (kit4market, kit4market 2.0). The evaluation focuses on answering evaluation questions on programme design, the integration of the programme into the relevant funding portfolio, the programme processes and the effects achieved so far. To this end, interviews are conducted with programme owners and Austria Wirtschaftsservice Gesellschaft mbH (aws), and the funding recipients are asked about their experiences with the program and the effects achieved.

In 2017 and 2018 the implementation of HORIZON 2020 and ERA, as well as EUREKA, COSME and EEN in Austria was evaluated by a consortium consisting of AIT, Joanneum Research, KMFA and ZSI. The evaluation focused on three key areas: 1. Evaluation of Austrian Research Promotion Agency’s (FFG) advisory and support services for HORIZON 2020, EUREKA, COSME, EEN and ERA 2. EU performance monitoring of FFG 3. The ERA Observatory Austria, which forms an overarching framework for the integration of the individual tasks in the field of governance for EU-RTI policy in Austria on the part of the BMBWF (Federal Ministry of Education, Science and Research).

The evaluation analyses to what extent the concept for innovation promoting public procurement (IÖB) established in 2011 has been successfully implemented or whether the corresponding setting enables efficient implementation. The evaluation is based on a mixture of quantitative and qualitative methods and also develops recommendations for further implementation with the help of an international comparison.

The BRIDGE programme funds basic science research projects at universities and other public research organisations that already demonstrate realistic application prospects. Thus, the BRIDGE funding will facilitate the transfer of basic science results into industrial applications.
Companies participate in these projects financially or non-financially by providing equipment and guidance to the research institutes. The evaluation focuses on the different facets of impacts of the programme and on the more recent modifications in the programme design during the period 2009 up to 2016.

This evaluation aims to analyse the portfolio of Upper Austria’s funding and support measures provided to companies. At the centre of this exercise is the analysis of the interplay of different measures with regard to their conceptual and design-based orientation towards additionality and broadness/specificity. Based on the results, the evaluation will develop recommendations for the client’s future funding and support policies after 2020.

The evaluation of the AWS and FFG promotion agencies essentially addresses the question whether and to what extent ambitions and expectations of the predecessor organisations consolidations, which happened in the context of the structural reforms more than a decade ago, have been achieved. Applying a theory-based evaluation approach (TBE) the evaluation analyses organisational and content-related functionalities, inherent properties, functional logics and impacts of both institutions, in order to draw action-relevant conclusions.

Contact: Peter Kaufmann

Download Report (German only)

The overall objective of this study was to evaluate the results and impacts of the participating SMEs within two initiatives of FP7: the cooperation programme and the research for the benefit of SMEs schemes under the capacities programme, including the impacts on economic performance, European added value, behavioural additionality and innovation. The evaluation study assessed the relevance, efficiency and effectiveness of the two initiatives.