FP7 Project: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

Methodology Development for the Evaluation of Policy Instruments to Promote Servicizing

-Deliverable 3.1-

Elah Matt, Moshe Givoni, Bryan Epstein (Tel Aviv University) Eran Feitelson (Jerusalem Institute for Israel Studies, Hebrew University) With contributions from:        

Surrey University, UK Oxford University, UK The Finnish Environment Institute, Finland University of Santiago de Compostela, Spain Lund University, Sweden Jerusalem Institute for Israel Studies, Israel Tel Aviv University, Israel Ben-Gurion University of the Negev, Israel February 2013

© SPREE. All rights reserved. This report was produced by SPREE consortium with the financial assistance of the European Union and is available on SPREE website free of charge to visitors for non-commercial use, provided that the European Union is duly credited. The contents of this document are the sole responsibility of the SPREE Consortium and can under no circumstances be regarded as reflecting the position of the European Union. SPREE - Servicizing Policy for Resource Efficient Economy - is a project within the Seventh Programme and is co-funded by the European Union (Grant Agreement 308376). The project duration is from July 1, 2012 – June 30, 2015.

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

Contents List of Tables .................................................................................................................................... 3 Executive summary .......................................................................................................................... 4 1. Introduction ................................................................................................................................. 6 2. Policy evaluation in the public policy literature ............................................................................. 8 2.1 What is a policy evaluation? ....................................................................................................... 9 2.2 Who evaluates, why, and how? ................................................................................................ 10 2.3 What is evaluated? ................................................................................................................... 13 3. Evaluation of policy instruments in the SPREE project ................................................................ 15 3.1 Impact evaluation of policy instruments ................................................................................... 17 3.1.1 Economic impacts............................................................................................... 17 3.1.2 Environmental impacts ....................................................................................... 18 3.1.3 Social impacts ..................................................................................................... 20 3.1.4 Potential effect on decoupling ............................................................................ 21 3.2 Evaluating Implementation ....................................................................................................... 22 3.2.1 Implementation feasibility: financial, technical, technological and pre-condition considerations .................................................................................................. 22 3.2.2 Institutional settings ........................................................................................... 24 3.2.3 Political legitimacy .............................................................................................. 25 3.2.4 Social acceptability ............................................................................................. 26 3.2.5 Flexibility & Reflexivity ....................................................................................... 27 4. Summary and next steps ............................................................................................................ 28 4.1 Using the evaluation framework in the SPREE project .............................................................. 30 References ..................................................................................................................................... 32

2

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

List of Tables Table 1: Aspects of administrative evaluation........................................................................................ 14 Table 2: Evaluation criteria proposed by policy instruments perspectives ............................................. 15 Table 3: Main criteria for the evaluation of policy instruments .............................................................. 16 Table 4: Examples of environmental impact indicators .......................................................................... 19 Table 5: Summary of evaluation methodology for the SPREE project..................................................... 29

3

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

Executive summary Policy evaluation is an inherently political activity, influenced by the interests, values, beliefs and norms of the evaluator. Positivist policy evaluations, based largely on quantitative research methodologies that were advocated during the 1960s and 1970s, were therefore over time complemented by more qualitative research methodologies. Today, many policy evaluations rely on a mix of qualitative and quantitative methods. In this deliverable, we aim to construct a broad methodological framework for the evaluation of existing policies that promote servicizing. Broadly, servicizing can be defined as a transaction where value is provided through the provision of services or functions, rather than sale and purchase of products. We distinguish between servicizing systems, which induce economic, environmental and social impacts, and policy instruments, which may facilitate the uptake and functioning of these systems. In this deliverable, we focus on the evaluation of the latter. This task promotes two of the SPREE project's objectives. First, the evaluation of individual policy instruments is a necessary first step in building policy packages to promote servicizing. This evaluation will be carried out in the context of the specific case studies chosen in each of the three SPREE sectors (mobility, water and agri-food) using quantitative – Agent Based Modelling framework, and qualitative methods. Second, the guidelines outlined in this document will also inform the evaluation of servicizing systems at later stages of the project. Our evaluation framework comprises two levels of analysis. In the first level, we suggest criteria for evaluating the impacts of individual policy instruments in terms of their economic, environmental and social impact, as well as decoupling effect. In the second level of analysis, we introduce criteria to examine the "implementability" of policy instruments in terms of their financial, technical and technological feasibility and any preconditions necessary for their uptake, institutional settings, political legitimacy, social acceptability, flexibility and reflexivity of these instruments. The suggested methodological framework comprises a list of criteria that inform a multi-criteriadecision type analysis. Each criterion is further divided into a number of guiding questions that can be utilized in the ex-post evaluation of existing policy instruments, as well as ex-ante evaluation of proposed policy instruments. The evaluation methodology is developed in parallel to the ex-post policy analysis on the national levels (task 2.5 of WP2), with reciprocal knowledge transfer between them. In 4

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

addition, the evaluation methodology developed is to be used in the next steps of the project and further tailored to each of the sectors which are the core focus of the SPREE project. It should also support the development of the Agent Based Modelling framework and tools and to construct policy packages ('Servicizing Policy Packages") to promote servicizing as a way to achieve resource efficient economy.

5

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

1. Introduction In a world where multiple economic, environmental and social crises dominate the public agenda, there is a need for new thinking on these matters. In the SPREE project, it has been argued that a shift towards a service-based economy can result in the improvement of economic, environmental and social circumstances, and ultimately to a decoupling of economic growth from environmental degradation. The SPREE project aims to examine this assumption. Another objective of the project is to propose policy packages, defined as a combination of individual policy instruments aimed at addressing one or more policy goals (Givoni et al., 2013), which could facilitate and promote the shift to servicizing and consequently contribute to decoupling and resource efficiency. Before policy packages can be assembled, a wide range of individual policy instruments must be considered and evaluated. The aim of this deliverable is therefore to outline a methodological framework that will allow for the evaluation of policy instruments to promote servicizing to be carried out later in the project, and in the three sectors considered - the water, agri-food and mobility sectors. Policymaking and hence policy evaluation are undertaken in increasingly complex and uncertain realities. A shift from one-way governing systems towards two-way governance systems between those governing and those governed has been purported (Kooiman, 1993). The often-conflicting interests of a range of public and private policy actors must therefore be taken into consideration when prescribing and evaluating policy instruments. Moreover, increasing uncertainties regarding, for example, the effects of global climate change, the volatility of financial markets and geo-political instability pose further challenges to public policymaking. In light of these challenges, policy evaluation has become a growth industry, undertaken by academics, governmental and non-governmental organisations, private consultancy firms and lobby groups (Huitema et al., 2011). As such, policy evaluation is an inherently socio-political practice shaped by the interests, values and beliefs of policy evaluators and participants in the policy process (Guba and Lincoln, 1989). The challenge for policy evaluators is therefore to strive for objectivity in policy evaluation, while acknowledging the inherent complexities and value-laden nature of this practice. In this context, the aim of this deliverable is to develop a general methodological framework for the evaluation of policy instruments to promote servicizing. It is perhaps prudent to provide definitions here for some key terms used throughout this deliverable, notably Methodology, Servicizing and Policy instruments. Methodology can be understood as 'the tasks, 6

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

strategies, and criteria governing scientific inquiry' (Gerring, 2012: 6). This is in contrast to the definition of a method, which can be understood as 'a specific procedure for gathering and/or analyzing data' (ibid). In this deliverable, we are therefore concerned with the construction of a broad methodological framework, rather than prescribing specific tools to carry out the evaluation. The methods for the implementation of this framework will be chosen by the project partners, in particular within task 3.5: "Development of data collection methods". Servicizing can be defined as a transaction where value is provided through services or functions and not just through selling products (a more account has been developed in the context of task 2.3: "Establishing a common understanding of the notion of servicizing"). Servicizing systems include both the environmental and societal systems that support/facilitate or act as barriers to servicizing solutions and provide the context for the desired implications and effects of servicizing solutions. In order to promote servicizing - the development and success of servicizing systems, policy instruments can be utilised, individually or preferably as policy packages. Policy instruments can be defined as ‘the techniques or means through which states attempt to maintain their goals’ (Howlett, 2011: 22). More broadly, policy instruments can be understood as ‘an identifiable method through which collective action is structured to address a public problem’ (Salamon, 2002: 19). This definition acknowledges the involvement of public and private actors in shaping the choice, design, and implementation of policy instruments. In this view, policy instruments are determined by, and determine which actors will be involved in the policy process, what role they will play, and how their (often conflicting) interests will shape the policy process (Salamon, 2002: 10-11). Therefore, in order to understand and evaluate public policies, we need to evaluate policy instruments and their impact and implementabiltiy in practice (Eliadis et al., 2005: 5). Our evaluation methodology comprises two levels of analysis. The first examines the economic, environmental and social impacts of policy instruments to promote servicizing and their potential effects on decoupling. Second, we examine the implementation feasibility (implementabiltiy) of individual policy instruments. We measure implementabiltiy through the consideration of financial, technical and technological feasibility, and necessary preconditions for the implementation of the policy instrument. We further identify institutional settings, political legitimacy, social acceptability and 7

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

flexibility as important determinants of the implementabiltiy of policy instruments. For each of these categories we provide a number of guiding questions to inform the instrument's evaluation. The methodological framework proposed in this document is intended for use throughout the remainder of the SPREE project. It will be used for evaluating policy instruments to promote servicizing in the mobility, agri-food and water sectors. Although the stated aim of this task is to propose a methodology for the evaluation of existing policy instruments, the framework will also provide the basis for ex-ante evaluation of policy instruments, and for assessment of their inclusion in policy packages to promote servicizing, adopting both qualitative and quantitative methods (including the Agent Based Modelling). The methodology will need to be adapted to the three sectors examined in the SPREE project, and further refined to in order to evaluate servicizing systems in different countries and localities. These objectives will be pursued at later stages of the project. This deliverable continues as follows. Section 2 provides a review of the existing public policy literature on policy evaluation. The section answers questions about what is policy evaluation, who evaluates, why, and what is evaluated. The section concludes that the evaluation of policy instruments requires the use of a broad methodological approach, and a mix of qualitative and quantitative research methods, supporting the choice of the methodological approach of the SPREE project. Section 3 then outlines evaluation criteria in two categories. First, the evaluation of impacts, or the effectiveness dimension of policy instruments is outlined. Second, the implementabiltiy aspect of policy instruments is explored. Section 4 concludes this deliverable with a summary of the main findings, the proposed evaluation criteria.

2. Policy evaluation in the public policy literature In order to develop the methodological approach of this study, it is important to first outline the concept of policy evaluation. This section attempts to answer the following questions: what is policy evaluation; when is policy evaluated; who evaluates policies and why; how are policies evaluated; and what is evaluated. The section concludes that policy evaluation is an inherently political practice, which requires the application of a mix of qualitative and quantitative research methods.

8

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

2.1 What is a policy evaluation? Definitions of policy evaluation abound and vary. Broadly, policy evaluation can be understood as ‘learning about the consequences of public policy’ (Dye, 1987: 351). Policy evaluation can be conceptualised as either an ongoing activity throughout the policymaking process, or as a separate stage of the policy cycle. The purpose of policy evaluation is to assess whether the policy objectives pursued were (or will be) met, and the means employed to achieve these objectives (Howlett, Ramesh and Perl, 2009: 178). Three different types of policy evaluation can be discerned. Understanding policy evaluation as a discrete stage of the policy cycle, results in the conceptualisation of two types of policy evaluation. Exante policy evaluation implicates the assessment of policy instruments before they are chosen, designed and implemented. This practice aims to inform policymakers of the possible consequences of various policy instruments before these are adopted. Ex-ante evaluation measures include environmental impact assessment, strategic environmental assessment and regulatory impact assessment (see for example Turnpenny et al., 2009; Persson and Nilsson, 2006). Ex-post policy evaluation occurs following the implementation of the policy instrument, and often (but not exclusively) ahead of its reformulation or termination. Ex-post evaluation investigates 'how a public policy has actually fared in action’ (Howlett, Ramesh and Perl, 2009: 178). Continuous evaluation occurs at all stages of the policy process. Continuous evaluation ‘involves investigating a policy program to obtain all information pertinent to the assessment of its performance, both process and result’ (Wollmann, 2007: 393, emphasis added). Also termed ‘formative evaluation’, this type of evaluation is undertaken intermittently at all stages of the policy cycle, and particularly during policy implementation (Parsons, 1995: 546-7). Formative evaluation requires analysis of ‘the extent to which a program is being implemented and the conditions that promote successful implementation’ (Palumbo, 1987: 40, see also Parsons, 1995: 547). This description corresponds with the two evaluation dimensions presented in the introduction to this deliverable, and outlined in section 3. These are the considerations of social, economic, environmental and decoupling impacts alongside considerations of the implementability of policy instruments. Continuous evaluation challenges the earlier perceptions of evaluation as a distinctive stage of the policy cycle (Hill, 2013: 298). There are clear benefits to this approach, such as ensuring the assessment of public policies not only before or 9

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

after implementation but on a regular basis, maintaining dynamic policy implementation, and dealing with uncertainties and unexpected consequences. However, in practice, continuous evaluation remains the least implemented type of policy evaluation. These notions of policy evaluation and particularly the conception of ex-ante and ex-post evaluation, assume that policymaking is a linear process defined by the discrete stages of the policy cycle. However, this is rarely the case, as policymaking is never a rational, linear process. As Parsons (1995: 79) notes: The idea of dividing policy-making in such a way greatly overstates the rational nature of policymaking and gives a false picture of a process which is not a conveyor belt in which agenda-setting takes place at one end of the line and implementation and evaluation occurs at the other.

In practice, policymaking in general, and policy evaluation specifically, seldom occur in a linear fashion. Still, the stagist model of policymaking provides a useful heuristic device 'with which to orientate and situate a particular academic contribution' (Givoni et al., 2013: 5). We therefore maintain the distinction between ex-ante and ex-post evaluation, although in practice policy evaluation can be understood as a continuous process. 2.2 Who evaluates, why, and how? Policy evaluation is conducted by governments and other interested policy actors, who wish to influence the policymaking process (Howlett, Ramesh and Perl, 2009: 178). Policy evaluators include academic scholars, independent research institutes, consultancy firms, international and national governmental bodies, non-governmental organisations and business groups (Huitema et al., 2011). Each of these actors has their own interests, values and beliefs that may shape the outcomes of the evaluation process. Two opposing views on the execution of policy evaluation exist. The rational model supports evidencebased policymaking. In this view, policy evaluation should objectively assess the achievements of a given policy instrument (or policy packages), creating a ‘natural feedback cycle to subsequent policy improvement (Hill, 2013: 296). However, this ideal seldom occurs in practice, and there is ‘rarely a systematic connection between evaluation and policy improvement’ (Hill, 2013: 298). Interpretivist views of policymaking therefore conceive policy evaluation as a political tool which shapes all (or some) stages of the policymaking process. These two conflicting views subsequently lead to diverging methodological perspectives on policy evaluation. 10

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

The public policy evaluation literature evolved since the 1960s, in line with trends in the broader public policy literature (Wollmann, 2007). From the 1960s through to the early 1980s, the evaluation literature was largely positivist, and based on two predominant assumptions. First, it was thought that in order to assess policy outcomes, it is necessary to identify in advance policy goals and political intentions. Second, in order to identify objective casual relation between observed change and policy program, there was a need for quantitative, quasi-experimental research design (Wollmann, 2007: 398). Evaluation was viewed as ‘a neutral, technical exercise in determining the success (or failure) of government efforts to deal with policy problems’, and was therefore seen as an ‘objective, systematic and empirical’ process (Howlett, Ramesh and Perl, 2009: 178). However, this practice was fraught with difficulties. For example, it is difficult to identify goals and objectives that will serve as a measuring stick. Policy goals often come in bundles and are difficult to translate into measurable indicators. Empirical data to fill indicators is difficult to get, “and the more meaningful an indicator is, the more difficult it is to obtain viable data’ (Wollmann, 2007: 398). Sideeffects and unintended consequences are hard to trace and measure. Further, methodologically robust research design can often not be applied as the “large N small amount of variables” conditions for quantitative research are often replaced with “small N large amount of variables", necessitating qualitative research methods (ibid). Positivist analysis proved difficult to conduct in the study of public policies, in which real-world experiments can rarely be conducted. The analysis of policy impacts and their distribution therefore proved to be a challenging task for the positivist researcher, since the distinction ‘between who gets and who does not get the benefit of a new policy’ can rarely be made (Hill, 2013: 296). From the early 1980s, the second phase of evaluation studies was influenced by constructivistinterpretative thinking. Scholars began questioning the premises that policies should be evaluated according to policy goals and political intentions. Instead, a plurality of perspectives, interests and values were recognised by these perspectives. For example, Stufflebeam (1983) suggested the CIPP model, based on the evaluation of Context, Input, Process and Product. This line of research acknowledged that ‘developing adequate and acceptable measures for evaluating policy was more contentious and problematic than was previously believed’ (Howlett, Ramesh and Perl, 2009: 179). This literature also acknowledged that evaluation could be employed to conceal some governmental 11

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

shortcomings. Thus, evaluation was increasingly perceived as a political tool in the hands of policy actors aiming to influence policymaking processes (Howlett, Ramesh and Perl, 2009: 179). Extreme postpositivist thinking therefore maintained that since similar conditions could be interpreted differently, ‘there is no definitive way of determining the correct evaluation mode’ (Howlett, Ramesh and Perl, 2009: 179). Instead, the prevailing interpretation of evaluation is determined by ‘political conflicts and compromises among the various actors’ (ibid). Evaluation was thus understood as a political activity, with little ‘systematic connection between evaluation and policy improvement’ (Hill, 2013: 298). Accordingly, qualitative evaluation methods were developed during this period. The third wave of policy evaluation, since the 1990s, was inspired by the New Public Management literature, and the idea of creating more efficient public administration. Increasingly, scholars and practitioners are combining positivist and post-positivist views on policy evaluation. This perspective is captured by, Howlett, Ramesh and Perl (2009: 179): [It] is not to suggest that policy evaluation is irrational or a purely political process, devoid of genuine intentions to assess the functioning of a policy and its effects. Rather, it served as a warning that we must be aware that relying solely on formal evaluation for drawing conclusions about a policy’s relative success or failure will yield unduly limited insights about policy outcomes and their assessment. To get the most out of policy evaluation, the limits of rationality and the political forces that shape it must also be taken into account, without going so far as to believe that the subjective nature of policy assessments allows no meaningful evaluation to take place (Howlett, Ramesh and Perl, 2009: 179).

Thus, policy evaluation needs to rely on both qualitative and quantitative research methods in order to provide an encompassing understanding of the outcomes of policies, but also to capture the inherently political and subjective nature of this process. According to Wollmann (2007: 397), evaluation research faces two main conceptual and methodological tasks. The first is the conceptualisation of the intended and unintended consequences that are to be assessed. These consequences are the dependent variables. Secondly, evaluation research needs to determine "whether and how the observed changes are causally linked to the policy and measure under consideration". These instruments are the independent variable. In the bid to answer these questions, the positivist and interpretivist camps have come closer over the years, and it is now widely agreed that [T]here is no “king road” in the methodological design of evaluation research; instead, one should acknowledge a pluralism of methods. The selection and combination of the specific set and mix of methods depends on the evaluative question to be answered, as well as the time frame and financial and personnel resources available (Wollmann, 2007: 398)

12

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

In summary, from the positivist approaches to policy evaluation that prevailed in the 1960s and 1970s, policy evaluation studies evolved first towards constructivist-interpretivist approaches based on qualitative research methods, and more recently towards a mix of qualitative and quantitative research methods. We now aim to answer the questions at the beginning of this section, with respect to the SPREE project. The evaluators employed under the SPREE project are the project consortium, which includes academic, governmental, and non-governmental organisations, from a range of EU Member States and Associated Countries. The evaluation is undertaken under the European Commission's Seventh Framework Programme (FP7), with the aim of promoting economic growth, environmental resource-efficiency and social cohesion through the promotion of servicizing systems (as discussed above). Our research methodology is mixed in its nature; combining both positivist and constructivist elements. As such, we suggest the utilisation of mixed research methods, including both quantitative and qualitative techniques. The following section goes on to examine in more detail what is actually being evaluated by different evaluation approaches. 2.3 What is evaluated? As mentioned above, policy evaluation is an inherently political practice, influenced by the interests and values of the evaluator. The selection of evaluation criteria and the perspectives of different policy actors can therefore lead to different conclusions regarding policy outcomes (Hill, 2013: 297). Public policy scholars created numerous categorisations of evaluation criteria. These studies were often concerned with the process, content, or outcomes of policies and policy instruments, either separately or in combination. For example, administrative evaluation, as conceived by Howlett, Ramesh and Perl (2009: 186), addresses these three aspects, as summarised in Table 1. Similarly, Marsh and McConnell (2010) provide detailed criteria for policy evaluation, based on the examination of process, content and outcome. These criteria, they claim, are constructed in order to overcome the barrier between positivists and constructivists evaluation approaches. Fischer (1995) outlined four considerations that affect evaluation efforts. First, positivist verification examines whether the policy's stated objectives were met (in our case the objectives are mainly achieving absolute decoupling through servicizing). Second, validation of the policy in terms of its relevance to the problems it aims to address will affect the evaluation outcomes. Third, vindication 13

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

examines whether the policy contributes to society as a whole, while social choice examines the ideological concerns about the policy’s objectives. These considerations, Fischer argues, influence the choice of divergent evaluation criteria, and therefore may lead to different results of the policy evaluation process. Not only the choice of indicators to assess a specific impact, or criterion will affect the evaluation outcome, but of course, and first of all the choice of what is evaluated (before deciding on how it will be evaluated). Thus, evaluation can consider economic impacts and appear successful only since environmental impacts are not considered. Table 1: Aspects of administrative evaluation Type of evaluation

What is being evaluated

Process evaluation

Organisational methods: Can policymaking processes be streamlined and made more efficient? Implementation divided into tasks and evaluated Program input: how much effort did government put into achieving policy goals in monetary terms Policy outputs: What is the policy producing?

Effort evaluation Performance evaluation Efficiency evaluation Effectiveness evaluation

Economic efficiency: Can policy goals be achieved more efficiently by different means? Is the performance adequate? Are policy goals met?

Source: Howlett et al. (2009: 186) From the above, it can be seen that research on policy evaluation often focuses on one or more evaluation aspects, including the context, process, content or outcome of public policies. However, many studies evaluating policy instruments hold in common their focus on the outcome dimension of evaluation. These approaches often examine the economic, social and political impacts of policy instruments, as summarised in Table 2. The common themes discerned from this table are the consideration of effectiveness in reaching stated objectives; economic and resources efficiency; social legitimacy and equity and political acceptability. In the methodology developed in the following section, we address the outcome dimension of policy evaluation by suggesting criteria for assessing policy impacts. We also offer some means for appraising the process and content dimensions of policy evaluation (and to a lesser extent the context dimension) through the examination of implementabiltiy.

14

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

Table 2: Evaluation criteria proposed by policy instruments perspectives Policy instrument Sabatier and Pelkey Salamon (1989) attributes (1987)

Linder and Peters Treiblock (1994) (1989)

Theoretical approach

Policy design

Valuation criteria

Advocacy- coalition Policy framework implementation Public management 1. Economic 1. Efficiency resources 2. Political support 2. Political costs 3. Administrative 3. Economic costs feasibility and 4. Efficacy equity 4. Supply and targeting effectiveness

Public choice

1. Resource1. Efficiency intensiveness 2. Political 2. Political risk considerations 3. Financial and 3. Distributional ideological considerations 4. Targeting

Source: Landry and Varone (2007: 110)

3. Evaluation of policy instruments in the SPREE project In the remainder of this deliverable, we construct a methodological framework for evaluating policy instruments and their impact in the context of the SPREE project. All the policy instruments that will be evaluated in the course of the project will be expected to promote and support, directly or indirectly, servicizing systems and a shift to servicizing. Whether these policy instruments will also support resource efficiency, decoupling and desired social outcomes depends on whether the servicizing system they support contribute to these aspects (different methodologies have been developed for this purpose within task 3.1: "Methodology development for measuring economic and environmental impacts of servicizing activities" and task 3.2: "Methodology development for measuring social impacts of servicizing activities"). In assessing and evaluating policy instruments two dimensions are considered. First, the extent to which the policy instrument, by promoting servicizing, affects the economic, environmental and social performances of a particular servicizing system. In this respect, there is similarity between the evaluation of servicizing systems (by using the methodologies developed within WP3) and the evaluation of policy instruments to promote them, outlined in this deliverable. The second dimension 15

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

we consider is the extent to which a given policy instrument can be implemented. The consideration of these two dimensions will inform the evaluation of policy instruments, and their role in facilitating a transition to servicizing. Table 3: Main criteria for the evaluation of policy instruments Evaluation dimension

Evaluation criterion

Impact

Economic impacts

(effectiveness)

Environmental impacts Social impacts Absolute decoupling effect

Implementabiltiy

Feasibility (financial, technical, technological and pre-condition considerations) Institutional settings Political legitimacy Social acceptability Flexibility and Reflexivity

The methodological framework outlined in this deliverable covers both of the above dimensions. First, the effectiveness of servicizing policy instruments - policy instruments to promote servicizing, in terms of economic, environmental, social and decoupling impacts is considered. Effectiveness represents the degree to which a policy intervention influences the corresponding policy objective in a desirable manner. Second, the implementabiltiy of servicizing policy instruments is examined. Implementabiltiy is considered in terms of technical, technological and financial feasibility, preconditions, institutional settings, political legitimacy, social acceptability, flexibility and reflexivity. These criteria are summarized in Table 3, and discussed in more detail below. While the evaluation criteria are treated as discrete categories, in practice there is an overlap between them. Where possible, we have tried to indicate these overlaps, but have adhered to distinct evaluation criteria in order to facilitate the evaluation work which will be conducted later in this project. We suggest a series of questions to guide the evaluation of the different criteria. Based on these questions, specific indicators will be selected and considered for the actual, quantitative and qualitative, evaluation of policy instruments later in this project. 16

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

3.1 Impact evaluation of policy instruments As mentioned above, the first part of this evaluation methodology examines the economic, environmental, social, and decoupling impacts of policy instruments that promote servicizing. These impacts are outlined in more detail below. 3.1.1 Economic impacts Much work has been done on the economic impacts of environmental legislation and technological innovation (see for example Jaffe et al., 2005). Indeed, environmental economists have over the past decades created detailed methodologies for economic evaluation, based largely on cost-benefit analyses. In the SPREE project, our ambitions are more modest. We relate the concept of economic impacts to the economic effects of policy instruments on governments, service providers, consumers and other affected parties, or in other words, the value chain of the examined system. We suggest a series of guiding questions, which will inform an economic analysis. Economic impact can be considered to include the impact on economic growth, employment and changes in the costs and benefits to affected parties. In the context of the SPREE project, and especially the quantitative analysis, the focus is on the service providers and produces, consumers, and on the chain/sector level. To assess the economic impacts on these actors the following questions can be used to guide the evaluation. For service providers and producers: 1. Does the policy instrument creates additional profits, and reduces risks for the service provider? 2. Is innovation stimulated (dynamic efficiency)? 3. What are the costs in terms of compliance, investment, and marketing? 4. Are intellectual property rights secured? 5. Are environmental externalities internalised? For consumers: 1. What are the costs for consumers associated with the policy instrument? 2. Are there any utility gains in terms of reduced costs, time savings and/or improved quality of life (see also social impacts)? 3. What are the likely short and long-term impacts on consumer demand and consumption behaviour?

17

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

For the chain/sector level: 1. What are the overall profits and added value in the system (e.g. knowledge gained, incomes secured, expertise developed, networks constructed etc.)? 2. What are the labour employment consequences of the change in the system? 3.1.2 Environmental impacts Servicizing systems and the policy instruments that support their uptake are often intended to reduce the negative environmental impacts of economic development. One of the main arguments in favour of the creation of servicizing systems is that gains can be made in resource-use efficiency. Material products are treated as capital assets rather than consumables. Servicizing may thus create incentives to maximize resource-use efficiency and product durability, whilst minimizing costs to the producer, consumer and the environment. Environmental assessment of servicizing solutions demonstrates that, on average, single-digit-factor improvement was achieved in various case studies, including: washing centres (Weaver, Jansen et al. 2000; Hirschl, Konrad et al. 2003), car sharing (Sperling, Shaheen et al. 2000), ski rental services (Hirschl, Konrad et al. 2001), and integrated pest management (Schmidt-Bleek and Lehner 1998). The relatively modest improvements stem from a lack of focus on the systemic changes that need to be made in order to deliver services in a more sustainable way. At the same time, quantitative methods for evaluating environmental impacts of servicizing solutions are still in short supply. Studies employing life-cycle analysis (LCA) and similar methods for evaluating servicizing solutions still yield diverging results, owing to differences in the evaluation systems’ initial conditions, boundaries and functional units chosen. The evaluation of environmental impacts therefore constitutes an important challenge to be addressed throughout the SPREE project. In the SPREE evaluation of environmental impacts associated with the implementation of policy instruments to promote servicizing, we distinguish between two evaluation criteria. Maximising resource-use productivity and minimising environmental impacts. Resource-use oriented indicators represent the amounts of resources used, while environmental-impact based indicators relate to the environmental impacts of servicizing systems, including the production and consumption of goods and services. Some examples of these indicators are outlined in Table 4, with regards to materials, energy and water resources. Some of these indicators can be evaluated through LCA. However, we 18

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

acknowledge that it may not be possible to include all the suggested indicators in case-specific evaluations. For example, it may be difficult to assess the impacts of a given policy on air pollution. Likewise, current LCA techniques have not accounted for some resource-efficiency indicators, such as metal depletion. Therefore, the factors suggested in Table 4 are more for illustration while specific indicators for this category will be selected later in the project. Table 4: Examples of environmental impact indicators Resource Materials

Resource use-oriented consumption of biotic and abiotic resources, Domestic material consumption, imports

Energy

cumulative energy demand (CED), total Greenhouse Gas emissions, air pollution, renewable and non-renewable energy use carbon footprint Water consumption and water abstraction Water exploitation index1

Water

Environmental impact-oriented terrestrial acidification, freshwater eutrophication, photochemical oxidant formation, particulate matter formation

Source: Adapted from BIO (2012: 8) To guide the evaluation of changes in resource-use and environmental impacts of a specific policy instrument, several questions should be considered. These include: 1. Does the policy instrument promote better resource use associated with servicizing? 2. Does the policy instrument ensure better product durability and life-time maximization, and therefore greater resource-use efficiency, and positive environmental impact? 3. Are incentives created for reduction, reuse, and recycling/remanufacturing of material resources? 4. Does improved efficiency/reduced cost lead to a rebound effect and therefore possibly to an overall increase in resource consumption? 5. Can the overall environmental impacts of the policy instrument be discerned, with respect to the criteria suggested above? (resource use, GHG emissions, carbon footprint etc.)

Due to some trade-offs between different environmental impacts, it is crucial to evaluate multiple environmental impacts of servicizing systems based on an LCA methodology (as explained in detail in Task 3.1). In the case of the SPREE project, indicators are identified for each of the case-study sectors. Examples of these indicators include inventory data, such as greenhouse gas emissions, energy use, 1

The European Environment Agency suggests a water exploitation index whereby water abstraction should stay below 20% of available renewable freshwater sources (BIO, 2012: 13). This indicator may not be applicable in non-EU countries (for example, Israel), and needs to be better defined through expert consultation.

19

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

water use, and waste generated, as well as impact indicators such as Global Warming Potential (GWP), acidification, etc. 3.1.3 Social impacts Within the SPREE project, we examine the effect of policy instruments on the social impacts of servicizing systems and we examine the effect of policy instruments that promote servicizing on the behaviour of consumers and businesses. Different types of social impacts can be distinguished among them are quality of life, the function of well-being (Diener, 2005), access to resources, job opportunities and equity issues. The concept of quality of life refers to the extent to which a person’s life is desirable, often in relation to some external factors such as income or environment (Diener, 2005). However, the measurement of quality of life is not always based solely on objective measurements such as GDP, life expectancy and education. For example, Pukeliene and Starkauskiene (2011) differentiate between individual quality of life and societal quality of life. Individual quality of life is explicit and considers how well individuals live, societal quality of life is implicit and relates to the stability of society. For the purpose of this task, and in order to simplify the evaluation procedure, we aim to evaluate individual quality of life, based on the measurements of objective indicators. The perception of wellbeing is more subjective. Wellbeing can be understood as a function of how people feel and operate on a personal and societal level, together with an overall self-evaluation of a person’s life (Diener, 2005; Michaelson et al., 2012). Community wellbeing has been defined by Forjaz et al. (2011: 734) as “satisfaction with the local place of residence taking into account the attachment to it, the social and physical environment, and the services and facilities”. More broadly, Hall et al.(2010) define societal well-being as the sum of human well-being together with the condition of the ecosystem, which provides the necessary resources and ecosystem services to maintain well-being. Thus, there is clearly an overlap between the notions of quality of life, and well-being. However, for methodological purposes we distinguish between these concepts. In evaluating the social impacts of policy instruments to promote servicizing, equity must also be considered. While policy instruments that promote servicizing may improve the quality of life and the well-being of society as a whole, they may have negative impact upon certain individuals, or they might 20

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

not improve quality of life and wellbeing equally across society. Hall and his colleagues (2010) argued that when considering the sum of human well-being, it is important to consider inter-generational and intra-generation equity. Although average material well-being can rise in a society, if the distribution of well-being is skewed, progress may not be sustainable or equitable (Hall et al., 2010). For example, some women may feel less comfortable travelling on public transport late at night than travelling in their own car, and thus could be negatively affected by measures to improve public transport – a servicizing system. Disabled users, who previously owned specially-adapted equipment, may struggle to benefit from services geared towards the wider population and which do not take into consideration their particular needs. Therefore, considerations of equity and fairness must be taken into account when measuring the social impacts of policy instruments to promote servicizing (these aspects have been studied within task 3.2). Some guiding questions for evaluating the social impacts of servicizing and policy instruments that promote servicizing include: 1. Are there visible changes in quality of life which can be attributed to the policy instrument (e.g. changes in GDP, life expectancy, environmental quality, education and health services)? 2. Is people’s perceived sense of individual wellbeing and personal satisfaction affected by the instrument (e.g. Are people more satisfied/happier with their life; see also Table 5)? 3. Is the distribution of costs and benefits among different societal groups equitable? 4. Are certain societal groups likely to be adversely affected by the policy instrument? Can these groups be compensated? 5. Have there been changes in the accessibility to local amenities and services (local well-being, Morrison, 2010: 1046)? 6. Have there been changes to the affordability of social services, including accommodation, food, utility bills, health care and education?

3.1.4 Potential effect on decoupling In the SPREE project, we ultimately aim to evaluate whether the synergetic economic, environmental and social effects of servicizing systems create a decoupling of economic growth from environmental impacts, together with positive social effects. We are interested in absolute decoupling, whereby the overall negative environmental effects decrease, while economic growth continues. Some guiding questions for evaluation include: 21

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

1. Is there any evidence that continued economic growth has resulted in lower overall environmental impacts? 2. Can observable changes in environmental quality be related to the economic impact associated with the policy instrument? 3. Are there specific relational effects among environmental and economic impacts (e.g. impact on emissions of greenhouse gases and employment)? 4. Can positive social impacts be observed in relation to economic/environmental decoupling?

3.2 Evaluating Implementation The previous section examined criteria for evaluating the change in the impacts of servicizing systems as a result of policy instruments to promote servicizing systems. In this section, we turn to examine factors that may affect the implementation of these instruments. It is common knowledge that the impact of policy instruments depends on their implementation in practice (Parsons, 1995: 547). Or, as Rist (1998: 158-159) observes: Successful policy implementation involves the translation of policy intent into policy action. While the failure to craft a thoughtful policy involving the choice of appropriate instruments can reasonably predict that the policy will not have the intended effects, so also can the failure to appropriately implement the policy ensure that the intended effects do not occur. Intention without execution leads nowhere.

Thus, changes in the economic, environmental and social impacts of policy instruments depend on the implementation of these instruments. Still, most studies on policy instruments have not adequately addressed their implementation in practice (Jordan et al. 2011: 546). In the remainder of this section, we introduce some of the factors that may influence the implementabiltiy (that is the likelihood of successful implementation) of policy instruments. These factors include feasibility, institutional settings, political legitimacy, social acceptability and flexibility in dealing with uncertainty. These considerations and their evaluation are discussed in some detail below. The suggested criteria must not be read as a comprehensive review of the vast literature on policy implementation, but should be seen as providing general guidelines for the empirical work which will be undertaken later in this project. 3.2.1 Implementation feasibility: financial, technical, technological and pre-condition considerations In order to ensure the successful implementation of policy instruments, they must be feasible financially, technically and technologically. As put by McDonnell and Elmore (1987: 146), 22

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

The selection of policy instrument depends on the constraints a policymaker faces and the resources available either to diminish the force of those constraints or to enhance the effectiveness of a given instrument. In simplest terms, identifying resources and constraints is how policymakers assess what is feasible, given how they define a policy problem.

We therefore need to understand the material constraints under which policy instruments are chosen and designed. These include financial limitations on both public bodies and the private actors, who are responsible for the implementation in practice of policy instruments. A policy instrument cannot be implemented if there are no funds to implement it, if it is not financially viable, or it could only be partially implemented (e.g. unfinished project). Equally, policy instruments need to be technically feasible, so that their implementation is not only financially viable, but also relatively straight-forward to undertake in terms of technical feasibility. For example, financial resources might exist to build a new high-speed train line, but the expertise to build the required infrastructure, or the boring machines for the required tunnels are not available. Another consideration that must be taken into account when evaluating the implementabiltiy of policy instruments to promote servicizing is whether the instrument is technologically feasible. Some policy instruments are adopted in order to promote technological innovation (technology-forcing instruments). For example, the European Union car emissions standards have over several decades promoted the uptake of new technologies, such as catalytic converters in the 1980s and 1990s (Arp, 2000; Wurzel, 2002), and more recently the uptake of hybrid cars and other low-carbon car innovations, following EU legislation on car CO2 emissions. However, when assessing technological feasibility it does not suffice to examine solely the available technology. Rather, as suggested by Unruh (2000), we must also understand societal and institutional factors, as well as interdependencies with other technologies and infrastructure in order to adequately address this criterion. The matter of technological feasibility will need to be carefully monitored during the project and the implementation of this methodological framework, as some of these matters are in the heart of the SPREE project and the promotion of a resource-efficient servicizing economy. One of the main elements in formulating policy packages is accounting for the relations between individual policy instruments. An important relation is precondition relation, relating to a situation whereby the successful implementation of one policy instrument entirely depends upon the prior successful implementation of another. In other words, a policy instrument will only work if its 23

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

preconditions have been implemented (Givoni et al., 2013; Taeihagh et al., forthcoming). In this respect it is important to identify, when characterising and evaluating a policy instrument whether it has any preconditions. From the above, we can discern several guiding questions relating to the implementation feasibility of policy instruments. These questions include: 1. What is the cost of implementing the policy measure? 2. Are there sufficient financial resources to support the implementation of policy instruments? Can funds be raised or borrowed (e.g. from the private sector)? 3. Is the implementation of the policy instrument technically achievable? Is the instrument relatively easy to implement in practice? 4. Does the instrument require the prior implementation of other policy instruments (pre-conditions)? 5. Is technological innovation required for the implementation of a policy instrument? If so, are there barriers and time delays before the innovation can be used? 3.2.2 Institutional settings The implementabiltiy of policy instruments is influenced by the institutional settings within which policy problems are defined, instruments are chosen and designed, and later implemented. In the context of the SPREE project, we adopt a definition of institutions which conceives these as 'organizations that are expected to initiate policy actions' (Gysen et al., 2006: 99). Institutions in this context refer to the public organisations within which publicly-elected officials and politicians, bureaucrats and administrators operate. These aspects were not well addressed in the public policy literature until recently (Parsons, 1995: 158), but are detrimental to the implementabiltiy of policy instruments to promote servicizing. Particularly, institutional settings are thought to prohibit policymakers from taking action in new directions and departing from the status-quo and policymaking norms (McDonnell and Elmore, 1987: 146). Therefore, when evaluating the implementabiltiy of policy instruments, we need to carefully examine the institutional settings within which these instruments are implemented. This examination can be multi-levelled where necessary, and include the evaluation of local, national, regional and international institutions, as appropriate. Furthermore, questions of coordination among vertical and horizontal 24

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

levels of government (and governance) need to be addressed here (see e.g. Jordan and Schout, 2006). Some guiding questions that can be asked in this context include: 1. Who are the main institutional actors? 2. At what spatial level do these actors operate in and what level(s) they have influence on or control over (on scale from local to global)? 3. Do the institutional settings and culture facilitate or act as a barrier to implementation? 4. Does coordination among horizontal and vertical levels of government required? 5. Are institutional actors held accountable for the implementation? 3.2.3 Political legitimacy Questions of political legitimacy are important at all stages of the policy cycle, and affect the implementabiltiy of policy instruments. Foremost, under this heading we address issues of democratic legitimacy. Schmidt (2013) conveniently distinguishes between output legitimacy, input legitimacy, and throughput legitimacy, giving examples of EU governance. In her conception, output legitimacy refers to the effectiveness of policy instruments for the people. This notion relates to social acceptability, discussed in more detail in the following section. Input legitimacy refers to the 'responsiveness to citizen concerns as a result of participation by the people' (Schmidt, 2013: 2, emphasis in original). Throughput legitimacy refers to the efficacy, accountability, inclusiveness and transparency of the policymaking process. Schmidt's distinction addresses the main criteria for evaluating political legitimacy, namely the accountability of politicians, citizen participation, and transparency (Huitema et al., 2011: 5). To these criteria we add the consideration of the relative power of policy actors vis-à-vis one another. This evaluation of power relations can shed light on the democratic legitimacy of various policy instruments. Issues of equity and distributional effects also fall under this category, as does the legality of policy instruments. Thus, in evaluating the political legitimacy of policy instruments, we need to address the following questions: 1. Are policymakers held accountable for the implementation of the policy? Is corruption mitigated? 2. Is transparency and provision of information promoted? 3. Is public participation safeguarded? 4. Who are the main policy actors? What is their relative power vis-à-vis each other? (power analysis). Do certain actors have greater lobbying, and even veto-power? 5. How are the costs and benefits associated with the policy instrument distributed between policy actors? Is the distribution of these impacts equitable? 25

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

6. Do any of these policy actors have veto power over implementation? Have they acted to promote/hinder the uptake of the policy instrument? 7. Is legitimacy ensured? Have public and private policy actors accepted the policy instrument (See e.g. Mickwitz, 2003: 427). In order to answer questions of political legitimacy, we need to consult a range of sources. Often nongovernmental organisations and lobby groups provide good information and critique of these aspects of political legitimacy. Other sources that may be consulted include mass-media, policymakers and policy experts. 3.2.4 Social acceptability The issue of social acceptability constitutes an important evaluation criterion, outlined here and discussed in greater detail later in this project. Definitions of acceptability abound, yet there is no consensus as to the meaning of this term. Perhaps the most concise definition to date stems from the work of Vlassenroot and his colleagues (2010). The authors make an important distinction between the terms “acceptability” and “acceptance”, based on time and experience of the individual. In their view, [A]cceptance entails beliefs and attitudes, based on their behavioral reactions after the introduction of a measure. Acceptability describes the prospective judgment, based on attitudes and beliefs about a measure, without experience, to be introduced in the future (Vlassenroot et al., 2010: 176, emphasis added).

In this view, acceptance can be understood as an ex-post evaluation criterion, while acceptability can be perceived as an ex-ante criterion. However, we wish to diverge from this understanding, and define acceptability as a criterion for both ex-post and ex-ante evaluation. The evaluation of social acceptability naturally correlates with other evaluation criteria suggested in this deliverable. For example, there is a correlation between acceptability and impacts such as fairness and equity (Bristow et al. 2010, Cools et al. 2011, Eriksson et al. 2006 and 2008, Schade and Schlag 2000 and 2003). The measurement of acceptability is often based on qualitative surveys or user choice preference, coupled with a quantitative, statistical analysis of collected data. The most popular model for measuring acceptability was expressed by asking participants to evaluate and choose between different policy scenarios or measures. The question “To what extent are you in favor of or against the implementation of this policy measure/policy package?” (rated on a scale from -3 to 3) (Eriksson et al.,

26

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

2008: 1122) seems a useful starting point for the evaluation of acceptability. However, when evaluating social acceptability we will need to ascertain and refine this question. 3.2.5 Flexibility & Reflexivity Policy evaluation takes place in increasingly complex and uncertain environments (Mickwitz, 2003). As such, evaluation needs to ascertain whether policy instruments are flexible enough to address these dynamic conditions, and the often-conflicting interests of a multitude of policy actors. In this context, the last few decades have seen a growing advocacy of more flexible policy instruments. This trend is perhaps most evident in the field of environmental policy, where "new environmental policy instruments", which allow policy actors greater flexibility in complying with environmental regulations, have been strongly promoted (Jordan et al., 2003). For example, various economic policy instruments, including permit and emission trading schemes have allowed economic actors greater flexibility in complying with environmental legislation. When addressing uncertainty, we need to examine whether the precautionary principle has been applied, whether the instrument is adaptable, and whether it addresses possible economic, environmental and social changes. To this, we add the consideration of reflexivity. When dealing with scientific, social, environmental and economic uncertainties, policy evaluations need to ascertain whether a policy instrument and policy goal can be redefined in accordance with changing knowledge and background conditions. Reflexivity can therefore by defined as 'willingness to question formal policy goals' (Huitema et al., 2011: 15). This criterion, Huitema and his colleagues argue, relates more to the evaluation procedure per-se, rather than to the evaluated instrument. Therefore, when conducting our evaluation, we need to evaluate not only the implementation and impact of the policy instrument, but also question whether the proposed policy goal promotes servicizing and ultimately a resource-efficient economy. In this context, questions to be asked include:

27

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

1. Can the policy instrument be revised and adapted to changing and uncertain conditions (see Mickwitz, 2003: 426)? 2. How does the policy instrument address uncertainty and unpredictability? 3. Does the implementation of the policy instrument can increase, or result in, lock-in effects and irreversibility effects? 4. Are instruments flexible enough to encourage lower compliance costs? (Jaffe et al., 2005: 165). 5. Is there a reflexive evaluation of policy goals? Do the policy goals promote a servicised, resourceefficient economy? (Huitema et al. 2011) These are only some tentative questions that we are interested in addressing in the SPREE project. We hope that the questions listed in this section will stimulate debate among the project partners, and inform the methodological approach of this project, as well as the choice of methods.

4. Summary and next steps In this deliverable, we introduced some current debates and trends in the public policy evaluation literature. We noted that largely quantitative research methods that were advocated in the 1960s and 1970s were later replaced by calls for the employment of constructivist, qualitative research methods. Increasingly, these approaches are combined, and a mix of quantitative and qualitative research is often encouraged in the public policy evaluation literature. This is also the approach adopted in the SPREE project. In this light, we propose a methodological framework to evaluate policy instruments that promote servicizing. We distinguish between two dimensions in the evaluation. The first refers to the social, economic, environmental and decoupling impacts of these instruments – the effectiveness of the policy instruments. The second dimension examines the implementabiltiy of these instruments in terms of their feasibility, institutional constraints, political legitimacy, social acceptability, flexibility and reflexivity. The outcome of the framework proposed is a list of guiding questions to inform the evaluation of individual policy instruments later in this project. This list of questions is summarised in Table 5.

28

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

Table 5: Summary of evaluation methodology for the SPREE project Evaluation dimension 1. Impact

Criterion

Summary of guiding questions

Economic

1. What are the implications for economic growth? 2. What are the consequences for employment? 3. What are the costs and benefits to service providers, consumers and the chain/sector level? 1. What are the impacts in terms of resource-use efficiency? 2. What are the environmental impacts, in terms of air water and soil quality, and climate change? 1. Are there observable objective changes in quality of life (e.g. employment, wages, environmental quality) 2. Are there subjective changes in well-being and happiness? 3. Is the distribution of social impacts equitable? 1. Does continued economic growth result in overall reduction in negative environmental impact? 2. Are there any observable positive relational effect among economic and environmental impacts? 3. Can social impacts be associated with any observable decoupling trends? 1. Is implementation financially viable? 2. Is implementation technically feasible? 3. Are there any necessary preconditions for implementation? 4. Are technological solutions in place/needed for implementation? 1. Are there institutional barriers/enablers to implementation? 2. What spatial scales do these institutions operate on? 3. Is there vertical and horizontal coordination among these institutions? 4. Are these institutions held accountable? 1. Are politicians held accountable? 2. Has public consultation and participation been promoted? 3. Is transparency guaranteed 4. How is power distributed among policy actors? 5. Is the policy equitable? 1. Is the policy instrument accepted by the general public?

Environmental

Social

Decoupling

2. Implementabiltiy

Feasibility

Institutional settings

Political legitimacy

Social acceptability Flexibility

1. Is instrument flexible enough to address uncertainties? 2. Can it be modified in light of any unexpected changes? 3. Is there reflexivity in evaluating the policy goals?

29

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

4.1 Using the evaluation framework in the SPREE project The analysis in the SPREE project essentially compares between three systems: one based on the selling of products, one based on selling of services, and one where the selling of services is supported with various policy instruments (through the implementation of policy packages). Such comparison will be within the mobility, agri-food, and water sectors and through specific case studies chosen in each sector. The 'impact' dimension of the evaluation (section 3.1) can be used here to guide the comparison between the three states. The methodological framework developed and described in this deliverable is however aimed at evaluating policy instruments to assess the degree to which they can support the transition from a product system to a service system and evaluate how they can favourably alter the impacts (economic, environmental and social) of the service system to facilitate absolute decoupling. This requires attention also to the implementability of the policy instruments. Moreover, it is generally accepted, also in the SPREE project, that the implementation of individual policy instruments in isolation from one another is unlikely to result in achieving the defined policy goals - resource efficient economy achieved through a shift to servicizing. To increase the likelihood of achieving the defined policy goals, policy packages must be designed and implemented. To create policy packages, the characteristics of each policy instrument proposed and considered must be fully understood before it is selected for inclusion in a package. This is done by applying the methodological framework described above. To apply the framework in practice, the list of guiding questions need to be transformed into measurable indicators and data on these indicators need to be collected. The methodological framework will be used to inform both quantitative and qualitative evaluation within the SPREE project. Qualitative evaluation will be based on the expertise within the SPREE partners and by involving various stakeholders in the SPREE projects, for example through several workshops. A central element in the SPREE project is the development of an Agent Based Modelling (ABM) tool for quantitative analysis of servicizing and its impacts, especially on decoupling. This tool will also be used for a quantitative evaluation of policy instruments and policy packages. In this respect, the methodological framework described in this deliverable will inform the development of the ABM to allow the evaluation of policy packages. This requires that the criteria listed in Table 6 will be included in the ontology which is the basis for developing the ABM. For example, when evaluating policy 30

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

instruments or packages, we would like to test if the implementation of a policy instrument by the agent 'government' affected the economic outcome of each 'consumer' and/or 'business' agent and to what degree, in order to assess the effect on the system as a whole. This effect of the policy instrument depends also on its successful implementation, the likelihood of which should ideally be also simulated and examined using the ABM and based on the criteria outlined in Table 6. Similarly, when evaluating whether absolute decoupling has been achieved in the modelled system, we would like to see if the policy packages considered increase or decrease decoupling by recording changes in the relevant indicators that will represent the decoupling criterion. The design of a policy package starts by considering many and wide range of policy instruments that could potentially promote servicizing and influence the impacts of servicizing systems. The evaluation framework proposed here will be used to assemble an inventory of around 100 or more measures. Based on the same evaluation framework and using qualitative evaluation methods the most promising 30 or so instruments will be selected, and might be assembled into a policy package. At this point quantitative evaluation methods will be employed and an analysis of the instruments and/or policy packages will be carried out using the ABM. It is envisaged that the process will end with returning to use qualitative evaluation methods in the final assessment of 'Servicizing Policy Packages' in the three sectors. This document has encapsulated the crucial criteria for evaluating policy instruments, and eventually promoting a shift towards a resource-efficient, service-based system. These criteria and the methodological framework developed are the starting point for assessing and promoting servicizing.

31

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

References Abdallah, S., Mahony, S., Marks, N., Michaelson, J., Seaford, C., Stoll, L., & Thompson, S, 2011 Measuring our progress: The power of well-being, London, New Economics Foundation Arp, H, 2002 "Technical regulation of politics: the interplay between economic interests and environmental policy goals in EC car emission legislation", in: Jordan, A (ed), Environmental Policy in the European Union: Actors, Institutions and Processes, London, Earthscan, pp. 256-274. BIO Intelligence Service, Institute for Social Ecology and Sustainable Europe Research Institute, 2012, Assessment of resource efficiency indicators and targets, Final report prepared for the European Commission, DG Environment, available at: http://ec.europa.eu/environment/enveco/resource_efficiency/pdf/report.pdf, accessed: 12.2.2013 Bristow A, Wardman M L, Zanni A M, Chintakayala P K, 2010, "Public Acceptability of Personal Carbon Trading and Carbon Tax", Ecological Economics, 69, pp. 1824-1837 Cools, M., Brijs, K., Tormans, H., Moons, E., Janssens, D., Wets., G, 2011, "The Socio-cognitive Links between Road Pricing Acceptability and Changes in Travel-behavior", Transportation Research A, 45, pp. 779-88 Diener, E, 2005, "Guidelines for national indicators of subjective well-being and ill-being", Social indictors network news, 84, 4-6. Eliadis F P, Hill M M, Howlett M, 2005, "Intorduction", in Designing government: from instruments to governance Eds F P Eliadis, M M Hill, M Howlett, McGill-Queen's University Press, Montreal, pp 3-18 Eriksson L, Garvill J, Norlund A, 2006, “Acceptability of travel demand management measures: the importance of problem awareness, personal norm, freedom, and fairness” Journal of Environmental Psychology, 26, pp. 15-26 Eriksson L, Garvill J, Norlund A, 2008, "Acceptability of single and combined transport policy measures: the importance of environmental and policy specific beliefs" Transport Research Part A, 42, pp. 1117128 European Commission 2012 Life cycle indicators for resources: development of life cycle based macrolevel monitoring indicators for resources, products and waste for the EU-27. European Commission, Joint Research Centre, Institute for Environment and Sustainability, available at: http://lct.jrc.ec.europa.eu/pdf-directory/LBNA25517ENN.pdf ], accessed, 12.2.2013 Fischer F, 1995, Evaluating public policy, Nelson-Hall Publishers, Chicago

32

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

Forjaz, M. J., Prieto-Flores, M. E., Ayala, A., Rodriguez-Blazquez, C., Fernandez-Mayoralas, G., RojoPerez, F., & Martinez-Martin, P, 2011, "Measurment properties of the Community Wellbeing Index in older adults" Quality of life research, 20(5), 733-743 Gerike R, Gehlert T, Richter F, Schmidt W, 2008, "Think Globally, Act Locally - Reducing Environmental Impacts of Transport" European Transport 38, pp. 61-84 Givoni M, Macmillen J, Banister D, Feitelson E, 2013 "From policy measures to policy packaging" Transport Reviews (in press) Guba, E G., Lincoln, Y S, 1989, Fourth generation evaluation, Sage Publications, London Gysen J, Bruyninckx H, Bachus K, 2006, "The Modus Narrandi A Methodology for Evaluating Effects of Environmental Policy" Evaluation 12 95-118 Hall, J, Giovannini, E., Morrone, A., & Ranuzzi, G, 2010, A framework to measure the progress of societies, OECD Statistics Working Papers. Paris: OECD Publishing Hill M, 2013, The public policy process sixth edition (Pearson, Harlow) Hirschl, B, Konrad, W, 2001, New concepts in product use for sustainable consumption. 7 European Roundtable for Cleaner Production, Lund, International Institute for Industrial Environmental Economics at Lund University. Hirschl, B, Konrad W, 2003, "New concepts in product use for sustainable consumption" Journal of Cleaner Production 11(8): 873-881. Howlett M, 2011 Designing public policies: principles and instruments (Routledge, London) Howlett M, Ramesh M, Perl A, 2009 Studying public policy: policy cycles and policy subsystems, Oxford University Press, Oxford Huitema, D, Jordan, A, Massey, E, Rayner, T, van Asselt, H, Haug, C, Hildingsson, R, Monni, S, Stripple, J, 2009, “The evaluation of climate policy: theory and emerging practice in Europe”, Policy Science, 44, pp. 179-198 Jaffe, A B, Newell, R G, Stavins, R N, 2005, A tale of two market failures: technology and environmental policy, Ecological Economics, 54, pp. 164-174 Jordan, A, Benson, D et al. 2011, "Policy instruments in practice" The Oxford handbook of climate change and society, J. S. Dryzek, R. B. Norgaard and D. Schlosberg (eds), Oxford, Oxford University Press, pp. 536-549. Kooiman J, 1993, "Social-political governance: introduction", in Modern governance: new governmentsociety interactions Ed J Kooiman, SAGE, London, pp 1-19 33

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

Marsh D, McConnell A, 2010, “Towards a framework for establishing policy success”, Public Administration,88(2), pp. 564-583 Landry R, Varone F, 2005, “The choice of policy instruments: confronting the deductive and the interactive approaches”, in Eliadis F P, Hill M M, Howlett M, Designing government: from instruments to governance, McGill-Queen's University Press, Montreal, pp. 106-131 McDonnell, L. M. and R. F. Elmore, 1987,"Getting the Job Done: Alternative Policy Instruments" Educational Evaluation and Policy Analysis, 9(2), pp. 133-152 Michaelson, J., Mahony, S., & Schifferes, J, 2012, Measuring Well-being, New Economics Foundation, London Mickwitz P, 2003, "A framework for evaluating environmental policy instruments: context and key concepts", Evaluation, 9(4), pp. 415-436 Mickwitz P, Birnbaum M, 2009, “Key insights for the design of environmental evaluations”, in M. Birnbaum & P. Mickwitz (Eds.), Environmental program and policy evaluation: Addressing methodological challenges. New Directions for Evaluation, 122, 105–112. Morrison, P S, 2010, "Local expressions of subjective well-being: The New Zealand experience" Regional Studies, 45(8), 1039-1058 Palumbo D J, 1987, The politics of program evaluation, Sage, London Parkin J, Wardman M, Page M, 2007, "Models of perceived cycling risk and route acceptability" Accident Analysis and Perception, 39, pp. 364-371 Parsons D W, 1995 Public policy: an introduction to the theory and practice of policy analysis, Edward Elgar, Aldershot Persson A, Nilsson M, 2006, “Towards a framework for ex post SEA: theoretical issues and lessons from policy evaluation”, in Emmelin L (ed), Effective environmental assessment tools – critical reflections on concepts and practice, Blekinge Institute of Technology,Research Report No 2006:03, Report No 1 from the MiSt-programme Rist R C, 1995, “Choosing the right policy instrument at the right time: the contextual challenges of selection and implementation”, in Belmanns-Videc M L, Rist R C, Vedung E (eds), Carrots, sticks & sermons policy instruments & their evaluation, Transaction Publishers, London Salamon L M, 2002, "The new governance and the tools of public action: an introduction", in The tools of government: a guide to the new governance Eds L M Salamon, O V Elliott, Oxford University Press, Oxford, pp 1-47 34

FP7 PROJECT: SPREE SERVICIZING POLICY FOR RESOURCE EFFICIENT ECONOMY

Schade J, Schlag B, 2000, “Acceptability of Urban Transport Pricing” .Rep. no.PL97-2258, European Commission 4th Framework - Transport RTD, AFFORD Schade J, Schlag B, 2003, "Acceptability of Urban Transport Pricing Strategies”, Transportation Research F, 6, pp. 45-61 Schmidt V A, 2013, "Democracy and legitimacy in the European Union revisited: input, output and ‘throughput’" Political Studies, 61, pp. 2-22 Schmidt-Bleek, F. and F. Lehner, 1998, Das MIPS-Konzept. Weniger Naturverbrauch - mehr Lebensqualität durch Faktor 10 (The MIPS-Concept. Fewer consumption of natural resources - more quality of life through Factor 10), Munich, Droemer Sperling, D., S. Shaheen, et al., 2000, Carsharing - Niche market or new pathway? Berkeley, University of California Stahel, W, 1997, "The Service Economy: 'Wealth without Resource Consumption'?", Philosophical Transactions A, Royal Society, London 355(1728), pp. 1309-1319. Stavins, R N 1995, "Transaction costs and tradable permits", Journal of Environmental Economics and Management, 29, pp. 133-148 Taeihagh A., Givoni M., Bañares-Alcántara R. (forthcoming), "Which policy first? A network-centric approach for the analysis and ranking of policy measures", Environment and Planning B Turnpenny J, Radaelli C M, Jordan A, Jacob K, 2009, “The policy and politics of policy appraisal: emerging trends and new directions”, Journal of European Public Policy, 16(4), pp. 640-653 Unruh ,G. C. 2000, "Understanding carbon lock-in." Energy Policy 28(12), pp. 817-830 Vlassenroot S, Brookhuis K, Marchau V, Witlox F, 2010, "Towards defining a unified concept for the acceptability of intelligent transportation systems (ITS): a conceptual analysis based on the case of intelligent speed adaptation (ISA)", Transportation Research F, 13, pp.164-178 Weaver P, Jansen L, et al. 2000, Sustainable technology development, Greenleaf publishing, Sheffield Wollmann H, 2007, “Policy evaluation and evaluation research”, in Handbook of public policy analysis: theory, politics and methods, Fischer F, Miller G, Sidney M.S (eds), Taylor Francis, Boca Raton, pp. 393402 Wurzel ,R. 2002, Environmental policy-making in Britain, Germany and the European Union: the Europeanisation of air and water pollution control, Manchester, Manchester University Press.

35