How often do causal analysis methods provide reliable findings on the impact of research, technology and innovation policy (RTI) in Austria? KMU Forschung Austria, in collaboration with WPZ Research, Joanneum Research and fteval, investigated this question between July 2025 and July 2026. The study analysed Austrian evaluations, international examples and the perspectives of commissioning bodies, evaluators and stakeholders.

Key findings

The study identified 17 Austrian RTI evaluations that included counterfactual impact analyses. Thirteen of these were carried out in the past ten years, which points to the growing importance of this method. At the same time, however, only six studies meet the minimum standard for robust causal evidence. The analysis cites as key causes small, highly specific funding programmes with few beneficiaries and control groups that are scarcely comparable; the simultaneous use of multiple funding streams by the same organisations, the impact of which can hardly be attributed to any single programme due to a lack of available data; and delays in data availability. Around 6.5 years can elapse between the start of funding and the appearance of visible effects in statistics.

Significance for the economy, politics and society

The study identifies significant potential for the wider use of counterfactual impact analyses, provided that the framework conditions are improved. These include the early incorporation of evaluations into programme design, adequate budgets, comprehensive access to funding data and easy access to microdata. Furthermore, the authors recommend targeted capacity-building for evaluators and commissioning bodies. fteval supports this through training programmes. In the fteval podcast "REvaluation" on the future of research policy, Jakob Kofler, who co-authored the study as lead author, explains the findings, recommendations and challenges of counterfactual impact analyses. At the same time, the study emphasises that counterfactual analyses can complement, but not replace, qualitative and theory-based methods.

to the Preprint Paper