Endeavour Fund Smart Ideas and Research Programmes

Assessor Guidelines for Smart Ideas 2017 Endeavour Round

November 2016

The Ministry of Business, Innovation and Employment reserves the right to withdraw or amend, at any time, this document or any part of it. All persons, entities, techniques and ideas contained in this document are fictitious and are for training purposes only. Any resemblance to real persons (whether living or dead), entities, techniques or ideas is unintentional and purely coincidental.

Contents Welcome ..................................................................................................................................... 1 How does this document help you? .............................................................................................. 1 PART A: ALL ASSESSORS ............................................................................................................... 2 What you need to know - at a glance ............................................................................................ 3 The 2017 Process ......................................................................................................................... 5 Roles and responsibilities ............................................................................................................. 7 Assessors ......................................................................................................................................... 7 Lead Assessors ................................................................................................................................ 8 MBIE ................................................................................................................................................ 8 Assessor training ............................................................................................................................. 8 The Application and Assessment Process ...................................................................................... 9 The Smart Ideas Application Process .................................................................................................. 9 Key steps of the assessment process .................................................................................................. 9 Assessing and scoring proposals ....................................................................................................... 10 Assessment comments ..................................................................................................................... 10 An introduction to the Scoring Guide ............................................................................................... 11 Structure and content of the scoring guide .................................................................................. 11 Differentiating assessment between concept proposals and full proposals ................................ 11 Using the scoring guide ................................................................................................................. 12 Weighing up ‘attributes’ of varying merit......................................................................................... 12 Other scoring tips.............................................................................................................................. 12 Your obligations ......................................................................................................................... 13 Conflicts of interest ........................................................................................................................... 13 Direct ............................................................................................................................................. 13 Indirect .......................................................................................................................................... 13 Managing conflicts ........................................................................................................................ 13 Confidentiality ................................................................................................................................... 13 Communications with MBIE.............................................................................................................. 14 Official Information Act 1982............................................................................................................ 14 Using the IMS portal ......................................................................................................................... 14 Assessing Excellence .................................................................................................................. 16 Area of research ................................................................................................................................ 17 Team ................................................................................................................................................. 18 Interpreting Criteria, Sub-criteria and Attributes ......................................................................... 19 Scoring guide for Smart Ideas – Excellence .................................................................................. 23 PART C: IMPACT ASSESSORS ....................................................................................................... 30

Assessing Impact........................................................................................................................ 31 Benefit to New Zealand .................................................................................................................... 32 Implementation Pathway(s) ............................................................................................................. 33 Interpreting Criteria, Sub-criteria and Attributes ......................................................................... 34 Scoring guide for Smart Ideas 2017 - Impact ............................................................................... 37 Appendix A ................................................................................................................................ 43 Investment Signals for 2016-19 ........................................................................................................ 43 Appendix B ................................................................................................................................ 45 Dimensions of Impact ....................................................................................................................... 45 Appendix C ................................................................................................................................ 46 Some guidance to help you strengthen your approach to Vision Mātauranga................................ 46 Appendix D ................................................................................................................................ 47 Assessing contributions to Vision Mātauranga................................................................................. 47

Welcome Welcome to the Endeavour Fund 2017. The Endeavour Fund will invest in science that has a strong potential impact on New Zealand’s economy, environment and society. Thank you for agreeing to contribute your time, expertise and experience to assess research proposals submitted to the Endeavour Fund. Excellent science and the impact it can deliver is reliant on strong peer support and constructive critique. If you have assessed proposals for us before, then a special welcome back. Smart Ideas are intended to catalyse and rapidly test promising innovative research ideas with scientific stretch and a high potential to benefit New Zealand. This mechanism encourages our scientists at all levels – from recent graduates to experienced researchers – to come forward with fresh ideas in science that have the potential to make a strong potential impact on New Zealand. We particularly encourage applicants to embody ‘stretch’ in their proposals or to take risks, with the aim of increasing the impact from our overall science investment. You will find this theme coming through in several sections of these assessment guidelines. Your role as an assessor is critical to identifying which of the many research proposals submitted to us demonstrate excellent science and have the potential to deliver credible impact on New Zealand’s economy, environment and society. Your input will help shape the recommendation to our Science Board of what Smart Ideas proposals to invest in for the next two to three years. The type of research MBIE is interested in funding is outlined in the investment signals section in the Investment Plan (and Appendix A). We expect proposals to contribute to one or more of the three objective areas (economic, environmental and social). The impacts we are seeking from these objectives are described in the Investment Plan (and Appendix B). The experience and expertise that you bring to the assessment process is greatly appreciated. Excellent science and the impact it can deliver relies on strong peer review and constructive critique. Thank you for your time and support of MBIE’s science investment processes.

Dr Prue Williams General Manager Science Investments

1

How does this document help you? This document has three parts so that you can focus on information relevant to your assessments and avoid unnecessary reading. What should you read? • Part A - All assessors • Part B - Excellence assessors • Part C – Impact assessors If you are assessing both excellence and impact, please read Parts B and C together. We will let you know in the lead up to proposals being assigned what we would like you to assess. Note that these two aspects are assessed at different stages in the process, so that you will only be assessing excellence or impact at any one time. We strongly recommend that you read this document before you begin assessing proposals. You may also want to read other relevant document particularly those set out in Figure 1 which will provide additional context. These are, or will be provided on the MBIE website, and you will be advised of the links as soon as possible. Figure 1: Document Map

An important note: Any change in the process will be notified via MBIE’s Alert e-newsletter for the Endeavour Round. If you do not already receive science investment Alerts you can sign up to receive them here

1

PART A: ALL ASSESSORS

2

What you need to know - at a glance Your role

Scope of the fund

As an assessor: • only assess and comment on what you are asked to assess and comment on. If you are assessing impact, do not comment or re-litigate the assessment of excellence. If you are assessing excellence, do not comment on impact. • your assessment score and comments must align with the scoring guides in Parts B and C of these guidelines. The Endeavour Fund invests in excellent research with high potential for impact in areas of future value, growth or critical need for New Zealand. The Fund can invest in research that is targeted basic research as well as research that is more applied. That said, the Government is looking to focus a greater proportion of investment in the Endeavour Fund in research with a potentially high, but longer-term transformative impact. The Fund invests in research that has the potential for impact in economic, environment, and/or social outcomes, as outlined in the investment signals for this fund (Appendix A).

Judgement, horizons of research and state of the sector(s)

The breadth of the Endeavour Fund’s scope across the research horizons means that you will have to apply judgement when assessing science excellence and impact, relative to the stage of research and the area of impact. The three horizons can be thought of as generating new ideas, developing emerging ideas, or leveraging proven ideas (National Statement of Science Investments, p31). For example, in the generate new ideas horizon, research might pose higher levels of risk than more developed research and have a less well defined or developed implementation pathway that refers to next users rather than end users. By comparison in the leverage proven ideas horizon, research might have a welldefined implementation pathway with proven engagement with end users. In undeveloped sectors e.g. a transformative new technology for which few or no end users currently exist, then planning for the development of new implementation pathways is appropriate. If assessing excellence, you might want to ask: • Is the size of risk and plans to mitigate that risk commensurate with the stage of research and state of the sector? • Is the proposed research excellent and fit for purpose for the proposed benefit impact sought? If assessing impact, you might want to ask: • Has the applicant described a vision of how this opportunity can be transformative for New Zealand in the future? • Is the proposed implementation pathway credible relative to the proposed stage of research, and the state of the sector?

3

Judgment and areas of impact

The breadth of the Endeavour Fund’s scope across economic, environment, and social outcomes means that you will have to apply judgement when assessing benefit and implementation pathways. For example, the potential benefits associated with research in the high-value manufacturing space, may be substantive in size but be narrowly focused because its implementation pathway involves commercialisation via a small number of companies. By comparison, research in the environmental or social space may have broad benefits for New Zealand that is implemented via engagement with public policy makers and regulators. You might want to ask: • Are the scale and breadth of proposed benefits credible given the area of impact? • Is the proposed implementation pathway credible for this particular area of impact? • Has the applicant described how the new science or technology has the potential to have breadth of application across multiple sectors, industries or user groups in the future? Refer to Appendix B for more detail of the dimensions of impact framework for the wider science systems.

Vision Mātauranga in proposals

Consider if the applicant has identified: • opportunities, needs, requirements, contributions or innovations from Māori knowledge, people or resources • relevant, specific Māori interests (collectives, businesses and communities) • a line of sight from research design to delivery of outcomes • appropriate and relevant elements that should be integrated throughout the proposal.

Relevant changes this year

The Gazette Notice has been updated and clarifies that those proposals with a primary focus on health outcomes are not eligible to apply, although a proposal where health is a secondary focus is eligible.

4

The 2017 Process What are the key dates?

These key dates are indicative and may be subject to change. We will notify you separately of specifics regarding assessment times and activities. Smart Ideas Call for Proposals and Application Guidelines available Roadshow sessions Closing date for registrations Closing date for concept proposals Proposals assessed Science Board decisions on concept proposals Applicants notified of Science Board decisions Closing date for full proposals Proposals assessed Science Board final decisions Applicants notified Contracts begin

Late September 2016 Early October 2016 12 noon, 2 November 2016 12 noon, 30 November 2016 February 2017 April 2017 May 2017 12 noon, 14 June 2017 Mid-June – July 2017 August 2017 September 2017 1 October 2017 What do I do if I think I have a conflict of interest?

If you think you have a direct or indirect conflict of interest with any proposal, please notify us as quickly as possible by: • emailing [email protected] to advise us of the circumstance leading to the conflict of interest and we will assess whether the conflict can be managed, or • declining the proposal in the Investment Management System (IMS portal). If we consider the conflict of interest is not manageable we will reassign the proposal to another assessor.

Why is the quality of my comments so important?

Your comments must: • align with the reference statements and criteria in the Scoring Guide • reflect your score and give guidance and the rationale for your assessment. Your scores and comments will be used to inform Science Board decisions and may also be the basis of feedback for applicants. It is important that your comment: • supports your score and fairly reflects the assessment • is accurate, professional, and honest. Therefore: • do not comment on the skill or expertise of fellow assessors • do not comment on the score and comments of fellow assessors (unless you are a lead assessor preparing summary comments, who may wish to reflect individual comments from other assessors) • do not comment on impact if you are assessing excellence • do not comment on excellence if you are assessing impact.

5

Why is information security vital?

We cannot overly stress the importance of information security and confidentiality. Do not discuss or communicate any aspect of the assessment you are working on with the applicant or anyone outside the process as you may undermine the integrity of the process. Only use the project note function in the IMS Portal to communicate with the lead assessor or other assessors. For security reasons, do not use your own personal communication channels to discuss proposals or communicate with other assessors.

What do I do if I have any questions?

Send any queries to us by email at assessors@[email protected]. Include your telephone contact details and time of availability if your query is urgent. Including the subject of your enquiry in the subject line of your e-mail will enable us to pass your question on to the right person quickly and provide a swift response.

What other supporting documents are available to me?

• • • • •

• •

National Statement of Science Investment 2015-2025 (the NSSI) outlines the government’s 10-year vision for New Zealand’s science system. Endeavour Fund Investment Plan 2016-2019 (the Investment Plan) outlines the government’s strategy for its science investment for the next three years. New Zealand Gazette Notice, Number 2016-go4875, August 2016 (the Gazette Notice) sets the criteria the Minister for Science and Innovation requires the Science Board to use in making funding decisions for the Fund. Vision Mātauranga policy outlines the government’s policy framework that aims to unlock the innovation potential of Māori knowledge, resources and people to assist New Zealanders to create a better future. International Science for New Zealand: Strategic Action Plan 2017 – 2017 provides guidance about government’s expectations of the contribution of international partnerships and collaborations to New Zealand science. (due for release early 2017) Guidelines for Applicants Completing a Proposal explains the purpose of each section of the application form and suggests information you may wish to include in your proposal. Portal Guidelines for Assessors explains how to use the IMS Portal to read proposals and record assessment details.

6

Roles and responsibilities Assessors Your role is to assess: • only what has been assigned to you. If you are assigned Smart Ideas Concept Proposals, only assess the excellence criteria. If you are assigned Smart Ideas Full Proposals, assess what you have been asked to assess. This may be excellence or impact, or both. • the proposal against the scoring guide by assigning a score and providing a constructive comment that justifies your score • each proposal, and score it relative to the scoring guide NOT relative to other proposals. Your role does not include: • weighing up the relative merits of different proposals • balancing the portfolio of investments. You also need to enter the assessment outcomes in the IMS portal. In doing this, we need you to: • declare situation of direct or indirect conflict of interest • respect the confidentiality of the proposals that you are assessing • undertake your assessments in accordance with the guidance in these guidelines. You will be able to view the entire proposal but you will only be able to score what you have been asked to score (excellence, impact or both). For excellence only you will also need to rate the ‘scientific or technical risk’ of the proposed research. Using the descriptors below, the characterisation should be consistent with the comments you have given. Additional information is provided in the scoring guide. Table 1: Assessment of scientific or technical risk Descriptor What is the unmitigated scientific or technical risk? What is the mitigated scientific or technical risk? Do the proposed additional benefits justify the proposed mitigated risk?

High

Medium

Low

High

Medium

Low

Yes

No

For Vision Mātauranga ensure that in your scoring decisions due weight is given to any contribution made by the proposal to Vision Mātauranga. Your comments on the criterion you are addressing should include reference to Vision Mātauranga. More information on Vision Mātauranga is available in the scoring guides and in Appendices C and D. MBIE is also interested overall in how well Vision Mātauranga has been addressed. The table below provides a guide to help MBIE understand how well applicants have addressed Vision Mātauranga and whether you agree with them or not.

7

Table 2: Classifications of Vision Mātauranga – Excellence and Impact Key questions 1) Is Vision Mātauranga, in your opinion, relevant to the proposal? 2) Does the applicant recognise that Vision Mātauranga is relevant to the proposal? 3) If Vision Mātauranga is, in your opinion, relevant to the proposal, how well has the applicant addressed this? Assessor comment

Yes

No

Yes

No

Very well Moderately well Not well Not Relevant Where there is a divergence of view between your and the applicant’s view (i.e. you consider it is relevant but the applicant does not or vice versa) to whether or not Vision Mātauranga applies, you must give reasons for this. If Vision Mātauranga is not relevant to the proposal and this is consistent with the applicant’s view (i.e. the answer to questions 1 and 2 is “No”, then the only comment you need to make is “Vision Mātauranga is not relevant to that proposal”.

Lead Assessors If you are also identified as a Lead Assessor, you will • assess the proposals assigned to you • read all individual assessments for a proposal (when scoring is complete) • prepare a summary of the individual assessments, including those related to both scientific or technical risk and Vision Mātauranga set out above. Your summary should synthesise different assessors’ views and reflect any significant differences.

MBIE Our role is to support the assessment process and assessors by providing you with information and guidance on administrative matters, the assessment process and the interpretation of guidance and other documents. It is not appropriate for us to become involved in the assessment of specific proposals. That is why we have external expert assessors.

Assessor training You will be provided with material to help you navigate through assessments closer to the time. We will contact you when the training material is available.

8

The Application and Assessment Process The Smart Ideas Application Process The Smart Ideas investment mechanism uses a two-stage assessment process. In Stage 1, concept proposals are submitted for assessment of excellence. Based on the results of the assessment, the Science Board will determine which concept proposals justify inviting the applicant to submit a full proposal. MBIE will advise all applicants of the Science Board’s decision on which concept proposals will proceed to the full proposal stage and those that are declined. In Stage 2, full proposals will be assessed for excellence and impact concurrently. Figure 2: Smart Ideas Application Process

Key steps of the assessment process The process in Figure 3 and Figure 4 illustrate, at a high level, the steps in each stage. Figure 3: Assessment of Smart Ideas Concept Proposals

9

Figure 4: Assessment of Smart Ideas Full Proposals

Assessing and scoring proposals When assessing a proposal you will need to consider whether, and how: • the proposal is delivering excellence (for concept proposals only) and excellence and/or impact separately (for full proposals) • the proposal responds to the signals set out in the Investment Plan (Appendix A) and Dimensions of Impact (Appendix B). Do this by considering the proposal against the excellence and impact criteria that are described in Parts B and C. More detail on what you need to consider in the criteria is provided along with a scoring guide.

Assessment comments Below are some simple guidelines for you to work to in drafting your assessment comment. When reading the proposal it is useful if you can highlight the following questions in your scoring and commentary: • What were the strengths and highlights of the proposed research? • What were the deficiencies or weaknesses of the proposed research? • Were there any concerns, issues around the proposed research? You should assess the proposal and address: • Only the criterion / criteria you have been asked to assess, please check you have addressed sub-criteria and their attributes • Include comments on whether or not Vision Mātauranga opportunities have been identified and followed through, and if not whether they should have been. Further guidance is provided in Appendix D. Your comments should reflect your score, the rationale for your assessment, and guidelines. • Ensure that your comments align with your scores, so they are mutually supportive, and explain why the particular score was given. • Ensure, for the sub-criteria of excellence, you comment on ‘risk’ and ‘innovation’. • Identify the specific reasons for a deficiency, particularly where the proposal has a score of 4 or less.

10

Word limits are in place for the length of your commentary – see Table 3 below. Table 3: Comment lengths

Assessor/individual comments Lead Assessor Summary Comment

Smart Ideas Concept Proposals 140 words 140 words

Smart Ideas Full Proposals 140 words 280 words

An introduction to the Scoring Guide Detailed guidance on how to carry out assessments has been incorporated in the Smart Ideas Scoring Guides (Parts B and C). The purpose of the Scoring Guide is to assist assessors in making judgements on scores, and to promote consistency in assessment between assessors by providing common: • interpretations of terms • understanding of how much merit or quality a proposal should have to warrant each score.

Structure and content of the scoring guide Scoring of proposals follows a scale of 1 to 7, with 1 being low quality and 7 being high quality. The Scoring Guide shows • scores on the left hand column • descriptions of the assessment criteria and sub-criteria in the columns to the right • within each sub-criterion there are a number of ‘attributes’, drawn from the text in the Gazette Notice, which need to be considered in scoring a proposal. Each level of score contains a set of reference statements on how much merit, quality or scale a proposal should possess for each attribute, in order to warrant a particular score. As Vision Mātauranga is assessed within each attribute where relevant, for each level of score there is also a separate indication of the Vision Mātauranga practice you might see at that score. The contributing attributes of each sub-criteria on which the guide is based are listed in the section titled Interpreting Attributes (in Parts B and C).

Differentiating assessment between concept proposals and full proposals The Scoring Guides apply to the assessment of both concept proposals and full proposals with the only difference being the level of information required between concept proposals, which are relatively short by nature, compared to full proposals in which more fulsome information is expected. This should be recognised in assessment as follows: • For concept proposals, focus should be on the quality of the information provided. Information is likely to be more summarised than for full proposals and it is acceptable for there to be limited supporting information. The story being told should still be logical and convincing. • For full proposals it is expected that the information provided will be more detailed and that the supporting information in particular will be more comprehensive. For full proposals the extent and quality of the supporting information should be a consideration in scoring. 11

Using the scoring guide Consistency Your score should align with the reference statements in the Scoring Guide. This will assist to achieve consistency in your scoring across the proposals that you assess and in your assessments relative to other assessors.

Weighing up ‘attributes’ of varying merit In Part C of this document, you’ll find the Scoring Guide (Impact) which shows attributes of equivalent merit or quality are grouped together against particular scores. For example, to achieve a score of 7, every attribute would be considered outstanding. Individual attributes will often merit different scores. In this situation you need to consider the balance across all attributes in determining the overall score. You may need to consider whether there are some attributes that should particularly influence the score or should have a dominant influence. For example, a serious flaw in one attribute would justify a low overall score, even if some attributes justified higher scores.

Other scoring tips Ensure your scores and comments reflect and align with the scoring guide. Use key words and reference statements. Only assess information presented in the proposal. Applicants are obliged to present the relevant information. If information in the proposal has obvious gaps, reflect that in your scoring and record any significant issues in your comments. Be wary of ‘drift’. It is not unusual for scoring to change as you gain experience with the assessment process. If it has drifted you may want to review your scores.

12

Your obligations Conflicts of interest MBIE takes the issue of conflict of interest very seriously. We follow a rigorous process to maintain the credibility of the Science Board’s funding decisions and to assure applicants that their proposals are given fair and reasonable assessment. Conflicts of interest may occur on two different levels, as outlined below:

Direct Examples of this could be where an assessor: • is directly involved with the proposal (as a participant, manager, mentor or partner) • has a close personal relationship with the applicants, for example family • is a collaborator or is in some way involved with the applicant’s research programme.

Indirect Examples of this could be where an assessor: • is employed by an organisation involved in the proposal but is not involved directly in the applicant’s research programme • has a personal and/or professional relationship with one of the applicants, for example, an acquaintance • has an involvement with a proposal that is in direct competition with a proposal being assessed or where the impacts proposed by a proposal under discussion may compete with an assessor’s personal business interests.

Managing conflicts Where there is a clear and direct conflict you must declare the conflict of interest to MBIE and not proceed with the assessment of the proposal. We will reassign the proposal to another assessor. You must also declare all indirect or unclear conflicts in relation to any allocated proposal to MBIE. If we agree that the conflict is significant, we will reassign that proposal to another assessor. You should advise all direct or indirect conflicts to MBIE by emailing [email protected].

Confidentiality As mentioned on page 6, we cannot overly stress the importance of information security and confidentiality. Please ensure the safe-keeping of all proposals and related documents (for example, workbooks and notes) and destroy all documentation after the assessment process is complete. You should not discuss any aspect of the assessment process with parties other than those involved in the assessment process. Under no circumstances should you discuss a proposal or potential proposal with any applicant. If an applicant contacts you, please direct them to MBIE at [email protected] and inform MBIE of the contact. You will be required to indicate your agreement to the confidentiality terms and conditions when you first log into to the IMS portal.

13

Communications with MBIE Assessors should always communicate with MBIE via the email address [email protected] in the first instance. This enables MBIE to ensure that questions are answered by the right person and that the responses are copied to other assessors if appropriate. Please do not phone or email specific individuals in MBIE.

Official Information Act 1982 MBIE is required to protect and release information in accordance with the Official Information Act 1982. Proposals, assessments and other information that is held by MBIE, including online content stored in the IMS Portal, is subject to the provisions of the Act and may be required to be released in full or in part if requested.

Using the IMS portal Portal Guidelines for Assessors will be available on MBIE’s webpage soon. We will also send an email to you advising they are available. .

14

PART B: EXCELLENCE ASSESSORS

15

Assessing Excellence Your assessment of Excellence in Smart Ideas proposals must be based on the assessment criteria which are taken from the Gazette Notice, outlined in Table 3 below. Points to note: Excellence should be considered in the context of: • Research horizons: early stage research may pose higher scientific or technical risk than later stage research. Both approaches are valid. • Areas of research: excellent research should be appropriate to the relevant discipline. Excellence includes two sub-criteria: Science and Team. For full proposals, science is given more weight (50%) than the team (15%) in assessment of Excellence, and that should be reflected in the assessment. All proposals must explain whether or not the proposed research is relevant to the Vision Mātauranga policy. How they might do this is discussed on Appendix D of this document. Beneath each sub-criteria sit attributes (Figure 5). Further details of the attributes can be found in Tables 5 and 6, pages 19-22. Table 4: Gazetted excellence criteria Criteria 2a. Excellence i. Science (weighted 50%) Research should be well-designed, well-performed, and leverage additional value from wider research. Assessment must have particular regard to whether the proposed research, science or technology or related activities: a. progress and disseminate new knowledge; b. possess high scientific risk, technical risk, novelty or innovative approaches; c. are well-positioned in the domestic and international research context; and d. have a well-managed research plan and credible approach to risk management.

ii. Team (weighted 15%) The proposed team should have the mix of complementary skills, knowledge and resources to deliver the proposed research, science or technology or related activities and to manage risk.

16

Figure 5: Layers of criteria, sub-criteria and attributes – Smart Ideas Excellence

Sub-criteria

Excellence

Attributes Science Research Positioning and new knowledge

Risk versus additional benefit

Team

Innovation or novelty

Fit for purpose

Research Plan

Skills, knowledge and human resource mix

Risk

We are looking to invest in research which has scientific or technical risk or stretch because it may have the potential to deliver greater value or impact for New Zealand than results of less risky research. You need to consider the: • level of scientific and technical risk and the rationale for the research approach. That is, does the potential proposed benefit justify the level of risk? • plans to ensure that scientific and technical risk, along with the programme risk, is well managed. There could be a large number of possible combinations to consider in deciding overall scores for assessment criteria. For practical reasons the scoring guide only gives some of the possible combinations. You should be flexible in deciding what combination applies to the proposal you are assessing, and how that might affect the score. For example, you might score proposed research with high technical risk that is well managed quite highly, yet score another piece of proposed research with a similar level of technical risk quite lowly because the risk does not appear to be well managed. Similarly, you might score proposed research with high technical risk with the potential to deliver sizable benefits to New Zealand quite highly, yet score another piece of proposed research with a similar level of technical risk quite lowly because the level of proposed benefit does not justify that risk.

Area of research Each discipline of science should be assessed within the context of what excellent science means for that discipline For example, the nature of the scientific or technical risk or innovation for social science is likely to be different from that for high value manufacturing, or environmental research. The key issue is whether the research is fit for purpose for the proposed research goals and benefits, and is the degree of risk (or stretch) or innovation well managed and providing additional value.

17

Team There is only one attribute for Team but it incorporates two elements – the mix of skills involved (are they appropriate to the research proposed?) and the level of skill (does the team overall have the level of experience and other attributes which would give confidence in their ability to deliver the research?). Emphasis should be given to the skill mix compared to the experience of team members as a relatively high level of risk is acceptable for Smart Ideas.

18

Interpreting Criteria, Sub-criteria and Attributes Table 5: Excellence, Sub-criteria Science Criteria a: Excellence Sub-criteria i: Science “Research should be well-designed, well performed and leverage additional value from wider research. Assessment must have particular regard to whether the proposed research, science or related activities…” Attribute …progress and disseminate new knowledge

Explanation ‘Dissemination’ means making the research results available for potential end users (or next users), so that impacts or benefits can be achieved. The means of dissemination may vary according to the situation and should not be confined to publication in the peer reviewed scientific literature.

…possess high scientific risk, technical risk, novelty or innovative approaches

Scientific risk has its source in the scientific basis of a proposal. This may include the assumption of a scientific hypothesis which steps beyond that which can be confidently asserted based on current knowledge and scientific principles; or the application of experimental or other scientific techniques in a context which is unproven or speculative. Technical risk has its source in new technology which needs to be developed to enable the research to be done, and the availability and suitability of experimental or supporting facilities and equipment. Table 5.1 explains the different categories of risk used in the Scoring Guide. ‘Innovative’ means bringing in new methods or ideas and there are degrees of innovation possible. The threshold for ‘novelty’ is higher – to be novel a method or idea must be of a new kind or nature which was previously unknown. So an idea or method is either novel or not. However, novelty can exist in a range of states which vary from something with only minor impact to something which represents a ground breaking advance, and elements of a proposal may be novel rather than the whole proposal. The Investment Plan encourages applicants to take risks, or embody ‘stretch’ in their proposals, in order to increase the potential scientific benefits from the proposed research. Smart Ideas proposals are intended to be the testing ground for new ideas and approaches so greater stretch is desirable. This is reflected in the relatively short durations and funding levels for Smart Ideas proposals. 19

Criteria a: Excellence Sub-criteria i: Science “Research should be well-designed, well performed and leverage additional value from wider research. Assessment must have particular regard to whether the proposed research, science or related activities…” Attribute

Explanation The risks need to be managed through risk mitigation strategies and/or contingency plans, and the level of risk resulting needs to be considered against the potential additional value. ‘Additional’ here means value that would not have been possible if the risks were not taken. However, for Smart Ideas it is expected that the balance will be in favour of risk. This is reflected in the reference statements in the scoring guide. Risk and additional value need to be considered together. However, risk and additional value can exist in many combinations, so the reference statements in the guide are only examples of where some combinations should sit in the scoring range. Assessors need to use their judgement in deciding where other combinations might most appropriately sit.

…are well-positioned in the domestic and international research context

‘Well positioned’ means that the proposed research takes account of existing knowledge and research, either by avoiding redundancy or overlap, or using existing knowledge/research as a platform for achieving more significant advances in knowledge than would otherwise be the case.

…have a well-managed research plan and credible approach to risk management

A ‘credible research plan’ is one which contains all of the expected elements set out in a way which is scientifically and managerially competent and can be effectively implemented. The expected elements include: the research methodology, methods, the research design and proposed outputs, a risk management and mitigation plan, and provision for access to and use of the facilities and equipment for carrying out the research. Table 5.1 sets out definitions of the descriptors used to describe different levels of risk in the Scoring Guide. These definitions also apply to the characterisation of full proposals which Lead Assessors are asked to undertake as an additional task to assessment.

20

Criteria a: Excellence Sub-criteria i: Science “Research should be well-designed, well performed and leverage additional value from wider research. Assessment must have particular regard to whether the proposed research, science or related activities…” Attribute

Explanation Table 5.1: Descriptor What is the unmitigated scientific or technical risk? What is the mitigated scientific or technical risk? Do the proposed additional benefits justify the proposed mitigated risk?

21

High

Medium

Low

High

Medium

Low

Yes

No

Table 6: Excellence, Sub-criterion Team Criteria a : Excellence Sub-criteria ii: Team “The proposed team should have the mix of complementary skills, knowledge and resources to deliver the proposed research, science or technology related activities to manage risk…” Attribute The extent to which the proposed team has the mix and level of complementary skills to deliver the proposed activities and to manage risk.

Explanation Both the team mix and the level of skill are important, e.g. a team may have the appropriate mix of scientific, engagement and risk management and mitigation skills but there may be important variations in the level of knowledge possessed and in experience in applying their skills and knowledge. Applicants are not required to provide track record information for Smart Ideas proposals. Assessment is therefore based only on the information in the proposal on skills and level of experience.

22

Scoring guide for Smart Ideas – Excellence Score

1 (Low quality)

Key words

Negligible Not present Not plausible Inadequate Not credible

Reference statements for attributes Criteria a. Excellence

Sub-criteria i. Science Weighting: 50% Research positioning and new knowledge Takes no account of related research or existing information. There is no new knowledge created and no consideration has been given to disseminating the research results.

Sub-criteria ii. Team Weighting: 15%

Skills mix The mix and level of skills, knowledge and resources in the team are inadequate in all respects needed to deliver the proposed research, science or technology and foresee or manage risks.

Risk versus additional benefit Scientific and/or technical risk is negligible; Scientific and/or technical risk is moderate to high but there is no additional value to justify this; Innovation or novelty Not innovative or novel. Fit for purpose The proposed research is not fit for purpose in any respect. Research plan The research plan is inadequate and does not contain any of the expected elements.

Vision Mātauranga

Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring.

Indications for identifying Vision Mātauranga at this scoring level

The research proposed does not unlock any Māori innovation. Vision Mātauranga is negligible or not present in the research plan.

23

The implementation pathway has no credible engagement with Māori. Vision Mātauranga is not present.

Score

2

Key words

Low Little present Limited Insufficient Lacking Poor credibility

Reference statements for attributes Criteria a. Excellence

Sub-criteria i. Science Weighting: 50% Research positioning and new knowledge There is little recognition of related research or existing information, and little evidence that this has influenced the research design. There is limited new knowledge created which does no more than amplify or further explain what is already known. The means of dissemination proposed are not well matched to the type of knowledge involved.

Sub-criteria ii. Team Weighting: 15%

Skills mix The mix and level of skills, knowledge and resources in the team lacks most, but not all, of what is needed to deliver the proposed research, science or technology. The team presents insufficient or poorly credible evidence to be confident they can capably foresee or manage risks.

Risk versus additional benefit Eg Scientific and/or technical risk is low but so is the additional value that could be potentially achieved. Innovation or novelty Has little innovation or very few elements of novelty. Fit for purpose The proposed research is fit for purpose in very few respects.

Vision Mātauranga

3

Research plan The research plan contains some of the expected elements, but the information provided is insufficient for the plan to be plausible. Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring.

Indications for identifying Vision Mātauranga at this scoring level

The research approach is a poor match for unlocking Māori innovation potential through the science proposed. Insufficient Vision Mātauranga elements in the research plan.

The team lacks the capability and/or capacity to navigate and engage with Māori stakeholders and their interests.

Low to moderate

Research positioning and new knowledge There is some recognition of related research or existing information, but this has only partly influenced the research design. The new knowledge created is incremental in character, i.e. is no more than a logical extension of what is already known. The means of dissemination proposed are workable but unlikely to be very effective.

Skills mix The mix and level of skills, knowledge and resources in the team has some significant gaps in what is needed to deliver the proposed research, science or technology or to manage risk.

Some Doubtful effectiveness Some aspects inadequate

24

Score

Key words

Significant gaps

Reference statements for attributes Criteria a. Excellence

Sub-criteria i. Science Weighting: 50% Risk versus additional benefit Scientific and/or technical risk is low and is largely justified by the additional potential value that could be achieved.

Sub-criteria ii. Team Weighting: 15%

Innovation or novelty Contains some elements of innovation or contains some elements that are novel. Fit for purpose The proposed research is fit for purpose in some respects but the connection to purpose is not well demonstrated. Research plan The research plan contains most of the expected elements, but the details are limited and some elements are inadequately developed.

4

Vision Mātauranga

Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring.

Indications for identifying Vision Mātauranga at this scoring level

The research approach is a doubtful match for unlocking Māori innovation potential through the science proposed. There are Vision Mātauranga gaps in the research plan which suggest low effectiveness delivering new knowledge or IP. Research positioning and new knowledge There is reasonable recognition of related research and existing knowledge, and this is leveraged in research design but with some gaps. The knowledge created is new but is largely an extension of existing knowledge rather than breaking significant new ground. The means of dissemination proposed are appropriate but conventional.

Moderate Significant in some aspects Reasonable Largely appropriate Largely satisfactory but with gaps or deficiencies Partly

Risk versus additional benefit Scientific and/or technical risk is moderate and is justified by the additional potential value that could be achieved. Innovation or novelty

25

The team has significant gaps in capability and capacity to navigate and engage with Māori stakeholders or identify associated risks. Skills mix The mix and level of skills, knowledge and resources in team is generally well matched to those required to carry out the proposed research, science or technology, but with some question marks. The mix and level of skills, knowledge and human resource in the team are generally well matched to achieve a reasonable level of risk management and mitigation. There are no significant gaps.

Score

Key words

Reference statements for attributes Criteria a. Excellence

Sub-criteria i. Science Weighting: 50% There is significant innovation in some aspects of the proposal or many elements of the proposal are novel.

Sub-criteria ii. Team Weighting: 15%

Fit for purpose The proposed research is largely fit for purpose but with some connections that need further explanation. Research plan The research plan contains all of the expected elements, but the level of detail is not of a satisfactory standard in some cases.

5

Vision Mātauranga

Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring.

Indications for identifying Vision Mātauranga at this scoring level

The research approach is a reasonable match for unlocking Māori innovation potential through the science proposed. Largely appropriate Vision Mātauranga elements appear in the research plan however there may be some gaps or deficiencies. New knowledge and IP of value to Māori interests is delivered.

The team has reasonable capability and capacity for engagement with obvious Māori stakeholders in familiar settings and can take on board their expectations with a reasonable level of risk management however there may be some gaps or deficiencies.

Moderate to high

Research positioning and new knowledge There is comprehensive recognition of related research and existing knowledge and this is competently leveraged in research design. The new knowledge created, breaks new ground rather than being incremental in character, and the means of dissemination proposes is appropriate and likely to be effective.

Skills mix The mix and level of skills, knowledge and human resource in the team are well matched to: • those required for carrying out the type of research involved • achieve satisfactory delivery of the research • achieve moderate to high level of risk management and mitigation.

Significant Very Comprehensive Appropriate Meets good practice standards Good/effective

Risk versus additional benefit Eg Scientific and/or technical risk is moderate and is more than justified by the additional potential value that could be achieved. Innovation or novelty The proposal is very innovative or the proposal contains ideas and/or approaches which are significantly novel.

26

There are no significant gaps.

Score

Key words Sub-criteria i. Science Weighting: 50%

Reference statements for attributes Criteria a. Excellence

Fit for purpose The proposed research is fully fit for purpose.

Sub-criteria ii. Team Weighting: 15%

Research plan The research plan contains all of the expected elements meets expected standards of good practice throughout, and should be delivered as stated.

6

Vision Mātauranga

Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring.

Indications for identifying Vision Mātauranga at this scoring level

The research approach is a very good match with unlocking Māori innovation potential through the science proposed. Credible Vision Mātauranga elements are central to the research plan. New knowledge and IP of significant strategic value to Māori interests is delivered.

The team has a moderate to high level of capability and capacity to navigate and engage effectively with Māori stakeholders, networks, interest groups in various setting. They can respond to expectations and opportunities with positive exchanges. They engage effectively. And can provide an appropriate level of risk management.

High/highly

Research positioning and new knowledge There is comprehensive and insightful recognition of related research and existing information and this is extremely well leveraged in research design. The new knowledge created is potentially significant beyond the immediate field of research, and the means of dissemination proposed are very good and likely to be very effective.

Skills mix The mix and level of skills, knowledge and human resource in the team are very well matched to achieve credible delivery of the type of research involved. This is a very good team with the ability to leverage significant affiliations.

Markedly Comprehensive Insightful Very good/effective Very large Very credible Significant

Risk versus additional benefit Scientific and/or technical risk is high and there is good potential additional value from taking that risk. Innovation or novelty Is highly innovative or contains ideas and/or approaches which are markedly novel, ie in either case likely to attract attention from other researchers and potential end users.

27

The mix and level of skills, knowledge and resource in the team are very well matched to achieve a high level of risk management and mitigation.

Score

Key words Sub-criteria i. Science Weighting: 50%

Reference statements for attributes Criteria a. Excellence

Fit for purpose The proposed research is fully fit for purpose.

Sub-criteria ii. Team Weighting: 15%

Research plan The research plan contains all of the expected elements at a level of competence and detail which exceeds expected standards of good practice, and gives confidence that the plan will be delivered as stated.

7 (High quality)

Vision Mātauranga

Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring.

Indications for identifying Vision Mātauranga at this scoring level

The research approach is a very good match for unlocking Māori innovation potential though the science proposed. Very credible and comprehensive Vision Mātauranga elements are central to the research plan. New knowledge and IP of significant strategic value to Māori interests is delivered.

The team have high capability and high capacity to navigate Māori stakeholders, networks, interest groups and their expectations. Engagement is a high value exchange. They are capable of very effective management of Māori knowledge of various types.

Very High

Research positioning and new knowledge The recognition of related research and existing information is of outstanding calibre, and is carried through into a research design which takes full advantage of the opportunities for leverage thus provided. The new knowledge created is potentially of international significance and likely to attract interest accordingly, and the means for dissemination proposed are wholly appropriate and likely to be extremely effective.

Skills, knowledge and human resource mix The mix and level of skills in the team are exceptionally well matched to achieving the excellent delivery of the research, science or technology.

Risk versus additional benefit Eg Scientific and/or technical risk is high and there is excellent potential to achieve additional value from taking that risk.

This is an outstanding team with the ability to leverage very significant affiliations.

Excellent Outstanding Exemplary Best international standard Wholly appropriate Very/extremely large Wholly credible Extremely significant

Innovation or novelty Is outstandingly innovative or contains ground breaking novelty, which in either case is likely to attract international attention.

28

The mix and level of skills, knowledge and resource in the team are exceptionally well matched to achieve a very high level of risk management.

Score

Key words Sub-criteria i. Science Weighting: 50%

Reference statements for attributes Criteria a. Excellence

Fit for purpose The proposed research is fully fit for purpose.

Sub-criteria ii. Team Weighting: 15%

Research plan The research plan not only contains all of the expected elements, but the approach and detail are exemplary and give high confidence that the research will be delivered excellently and as planned. Vision Mātauranga

Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring.

Indications for identifying Vision Mātauranga at this scoring level

The research approach selected is the best match with unlocking Māori innovation potential in regard to the science proposed. Wholly credible Vision Mātauranga elements are central to the research plan to a very high standard. New knowledge and IP of very significant strategic value to Māori interests is delivered on a large scale.

29

The team has excellent capability and capacity is to expertly navigate significant and diverse Māori stakeholders, networks, interest groups and their expectations. Engagement is a very high value exchange. They are very capable of excellent management of Māori knowledge of various types.

PART C: IMPACT ASSESSORS

30

Assessing Impact Your assessment of impact in Smart Ideas proposals must be based on the assessment criteria which are taken from the Gazette Notice, outlined in Table 6 below. Impact includes the sub-criteria Benefit to New Zealand and Implementation Pathway(s). For full proposals, Benefit to New Zealand if given more weight (25%) than Implementation Pathway(s) (10%) in the assessment of impact, and that should be reflected in the attention given to each aspect by you (see Table 7) All proposals must explain whether or not the research is relevant to the Vision Mātauranga policy. How they might do this is discussed in Appendix D of this document. Table 7: Gazetted impact criteria Criteria 3a. Impact i. Benefit to New Zealand (weighted 25%) Research should have direct and indirect benefits or effect on individuals, communities or society as a whole, including broad benefits to New Zealand’s economic, social, human or natural capital. Assessment must have particular regard to: a. the credibility of the need for scale and extent of potential benefits from the proposed research, science or technology or related activities; and b. the relevance and additional value they deliver to New Zealand.

ii. Implementation Pathway(s) (weighted 10%) The credibility of indicative implementation pathway(s) to deliver public benefit to New Zealand, not limited to a single firm or end-user, and which may be uncertain in nature.

Beneath each sub-criteria sit attributes (Figure 6). Further details of the attributes can be found in Tables 8 and 9, pages 34 - 36.

31

Figure 6: Layers of Criteria, sub-criteria and attributes – Smart Ideas Impact

Criteria

Impact

Benefit to New Zealand

Sub-criteria

Attributes

Credibility of the need, scale and extent of potential benefits

Relevance and additional value

Implementation Pathway(s)

Credibility of implementation pathway(s)

Benefit to New Zealand Descriptions of the benefits we are looking to invest in are included in the investment signals in the MBIE Contestable Research Fund Investment Plan (Appendix A). Your overall assessment of benefit should be driven by the combined size, breadth/extent, and credibility of the benefits. Credibility includes whether or not there are potential end-users, next users, or beneficiaries, and the quality of the estimates of benefit. In assessing these aspects, the relevance and additional value provided by proposals also needs to be considered. You should also consider the breadth/extent of the proposed benefits, which may include aspects that go beyond the direct benefits associated with the output of the research. In addition to those referred to in Appendix B these can include: • Benefits across multiple sectors • Faster uptake of results in multiple areas • Improved state of the environment • Potential to scale up regional initiatives to nation-wide implementation • Consistency of standards or approaches for regulators • Improved social well-being • Better use of resources • Preservation or enhancement of cultural heritage and values • More efficient processes • Upskilling industry • Job creation • Development of a cluster of businesses • Multinational business attraction to or retention in New Zealand • Protecting existing markets, or impact on New Zealand’s reputation • Diversification of the economy. If a sector is new or emerging, or only affects a narrow range of companies, then a vision of how the research will contribute to building a sector that will transform New Zealand in the future, or how the technology may affect multiple sectors in the future is needed. 32

Implementation Pathway(s) A detailed description of implementation pathways is expected in proposals. You should look for some indication that pathways have been given serious thought and that planned implementation is not limited to a very generic ‘one size fits all’.

33

Interpreting Criteria, Sub-criteria and Attributes Table 8: Impact, Sub-criterion Benefit to New Zealand Criteria b: Impact Sub-criteria i: Benefit to New Zealand “Research should have direct and indirect benefits or effect on individuals, communities or society as a whole, including broad benefits to New Zealand’s economic, social, human or natural capital. Assessment must have particular regard to…” Attribute …the credibility of the need for scale and extent of potential benefits from the proposed research, science or technology or related activities

Explanation Scale’ means size but this should be expressed in a way which sensibly reflects the end use area. For example an economic development project may be able to express scale in $$ terms or degree of penetration of markets etc. Social and environmental projects may use scalars of a different type, for example the level of impact on or significance for, reducing environmental effects, resolving social issues, developing more effective policies etc. In assessing estimates of scale it is important to apply the principle of additionality. ‘Additionality’ in this context means: • Value over and above that which would be expected to occur anyway through routine research investment by existing, scientifically competent businesses or user organisations; • Value which exceeds the cost of doing the research (or value for money). ‘Extent’ means the coverage of the benefits, that is, irrespective of scale whether benefits are concentrated in a narrow area (eg individual organisations) or are of widespread potential impact. Given a particular scale of impact, proposals which are of widespread coverage should score more highly than those which are of narrow impact. For Smart Ideas proposals there is not expected to be a high level of supporting information on impact that can be used to judge credibility. It should be sufficient for there to be evidence that the estimates of benefits have some basis and are not just presented with no justification. There should be sufficient information to indicate whether there are plausible types of end users or beneficiaries for the research.

…the relevance and additional value they deliver to New Zealand

Investment signals are set out on pages 18 and 19 of the Investment Plan. In assessing impact, assessors should consider the extent to which proposals will enable: • potential impact for New Zealand • more investment in research with higher (impact) risk and longer term horizons to impact. (Impact risk needs to be considered 34

Criteria b: Impact Sub-criteria i: Benefit to New Zealand “Research should have direct and indirect benefits or effect on individuals, communities or society as a whole, including broad benefits to New Zealand’s economic, social, human or natural capital. Assessment must have particular regard to…” Attribute

Explanation separately from scientific/technical risk in the research, which is included in assessing excellence) • better leveraging of wider existing investment and knowledge in New Zealand and overseas • greater effect to be given to Vision Mātauranga. Areas of strategic importance are set out on p19 of the Investment Plan and described as ‘future directions of investment by objective area.’ Alignment with those areas can be taken to also mean relevance to New Zealand’s current and future needs. Detailed directions are set out under separate headings for each objective: • Economic • Environment • Society; and • Vision Mātauranga across all 3 of the above objectives Proposed impacts should be aligned with one or more of the detailed objective areas, but the number of areas to which they are aligned should not be a consideration (some proposals will be naturally more sharply focused than others). However, the strength of the alignment is an appropriate consideration.

35

Table 9: Impact, Sub-criterion Implementation Pathway(s) Criteria b: Impact Sub-criteria ii: Implementation Pathway(s) “The credibility of indicative implementation pathway(s) to deliver public benefit to New Zealand, not limited to a single firm or end-user, and which may be uncertain in nature…” Attribute … the credibility of indicative implementation pathway(s) to deliver public benefit to New Zealand, not limited to a single firm or end-user, and which may be uncertain in nature

Explanation A ‘credible’ implementation pathway is one which contains sufficient end use-specific information to confirm that the analysis takes account of the characteristics of the end use area and is not simply a generic, ‘one size fits all’ description. Credible within the context of the horizon and area of research, and span offered by the proposal. The information sought in concepts is indicative only and it is likely that at this early stage of investigation, information on implementation may be uncertain. There does not have to be detailed supporting analysis and the information can be at a relatively high level, for example types of end user, beneficiary or stakeholder rather than specific end users etc, and couched in relatively broad terms which reflect uncertainty about the precise implementation path. That said, research positioned within the new ideas horizon should identify a next user to take the research outcome to the next stage of development (e.g. developing an emerging idea).

36

Scoring guide for Smart Ideas 2017 - Impact Score

1 (Low)

Key words

Negligible Not present Not plausible Inadequate Not credible

2

Sub-criteria i. Benefit to New Zealand Weighting: 25%

Reference statements for attributes Criteria b: Impact

Credibility, scale and extent of benefits The potential direct and indirect benefits are negligible and/or are not credible at any level. Potential benefits are almost completely captured by a small area of limited impact for New Zealand.

Sub-criteria ii. Implementation pathways Weighting: 10% Implementation pathways There are no credible implementation pathways and there is no supporting information.

Relevance and additional value Not relevant to areas of strategic importance to New Zealand as set out in the Investment Plan. Provides no additional value through alignment with any of the future directions of investment in the Investment Plan.

Vision Mātauranga

Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring.

Indications for identifying Vision Mātauranga at this scoring level

The benefits of the proposed research are not distinctly relevant to Māori innovation requirements.

The implementation pathway has no credible engagement with Māori. Vision Mātauranga is not present.

Low

Credibility, scale and extent of benefits The potential benefits are very low and not significant within a sector. The estimates of benefits are not very credible or well explained, including the links to stakeholders, next users, end users and beneficiaries able to achieve uptake. Potential benefits are captured by a small area with limited spread of benefits.

Implementation pathways Implementation pathways are set out but are of poor credibility. The supporting information provided is very limited and largely unsatisfactory.

Little present Limited Insufficient Lacking Poor credibility

Relevance and additional value Marginally relevant and of little potential significance to one or more areas of strategic importance to New Zealand. There is little additional value in areas that are aligned with one or more future directions of investment.

37

Score

Key words Sub-criteria i. Benefit to New Zealand Weighting: 25% Vision Mātauranga Indications for identifying Vision Mātauranga at this scoring level

3

Low to moderate Some Doubtful effectiveness Some aspects inadequate Significant gaps

Vision Mātauranga Indications for identifying Vision Mātauranga at this scoring level

Reference statements for attributes Criteria b: Impact

Sub-criteria ii. Implementation pathways Weighting: 10%

Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring. The benefits of the proposed research have a limited amount of relevance to Māori Innovation requirements.

The implementation pathway has very limited arrangements for engagement with Māori stakeholders and is not sufficient.

Credibility, scale and extent of benefits The potential direct and indirect benefits are low and are not significantly spread across a sector. The estimates of benefits are not very credible or well explained, including the links to stakeholders, next users, end users and beneficiaries able to achieve uptake. Potential benefits are captured by a moderately sized area with some spread of benefits.

Implementation pathways The implementation pathways set out are of limited credibility. There is some supporting information but it is unconvincing.

Relevance and additional value Some relevance but of very limited potential significance to one or more areas of strategic importance to New Zealand. There is some additional value in areas that are aligned with one or more future directions of investment. Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring. The benefits of the proposed research have low to moderate relevance to Māori. It’s unclear if any economic and/or social and/or environmental benefits will be realised by any Māori interests or if Māori contributions to the science are identified and valued. There are gaps.

38

The implementation pathway has some significant gaps in arrangements for engagement with and responding to Māori stakeholders. The Vision Mātauranga related aspects of the implementation pathway appear somewhat inadequate.

Score

4

Key words

Moderate Significant in some aspects Reasonable Largely appropriate Largely satisfactory but with gaps or deficiencies Partly

5

Sub-criteria i. Benefit to New Zealand Weighting: 25%

Reference statements for attributes Criteria b: Impact

Credibility, scale and extent of benefits The potential direct and indirect benefits are moderate and may have some impacts within a sector or some impacts across several sectors. The estimates of benefits have some credibility but the supporting explanation is not completely convincing, including the links to stakeholders, next users, end users and beneficiaries able to achieve uptake. Potential benefits are captured by diverse areas with some spread of benefits.

Sub-criteria ii. Implementation pathways Weighting: 10% Implementation pathways The implementation pathways set out are partly credible, but there are gaps in the supporting information that leave key questions unanswered.

Relevance and additional value Largely relevant and of some potential significance to one or more areas of strategic importance to New Zealand. There is moderate additional value in areas that are aligned with one or more future directions of investment.

Vision Mātauranga

Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring.

Indications for identifying Vision Mātauranga at this scoring level

The benefits of the proposed research are somewhat relevant to Māori achieving their goals and links have been satisfactorily described. Moderate economic and/or social and/or environmental benefits are likely to be realised to some Māori interests. Māori contributions to the science have been identified and are valued.

The implementation pathway has satisfactory arrangements for engagement with obvious Māori stakeholders. Some distinct risk and sensitivity management is in place. Satisfactory Māori participation in research is evident. Obviously relevant tikanga Māori has been satisfactorily gauged and described throughout the pathway. The IP agreements in place response to any indigenous knowledge management issues/requirements if raised.

Moderate to high

Credibility, scale and extent of benefits The potential direct and indirect benefits are large and will have significant impacts within a sector or some impacts across several sectors. The estimates of benefits are credible and well explained, including the links to stakeholders, next users, end users and beneficiaries able to achieve uptake. Potential benefits are captured by many areas with significant spread of benefits. 39

Implementation pathways The implementation pathways set out are credible, fit for purpose and the indicative supporting information is satisfactory in scope and competent.

Significant Very Comprehensive Appropriate

Score

Key words

Meets good practice standards Good/effective

6

Reference statements for attributes Criteria b: Impact

Sub-criteria i. Benefit to New Zealand Weighting: 25% Relevance and additional value Very relevant and of substantial potential significance to one or more areas of strategic importance to New Zealand. There is substantial additional value in areas that are aligned with one or more future directions of investment.

Sub-criteria ii. Implementation pathways Weighting: 10%

Vision Mātauranga

Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring.

Indications for identifying Vision Mātauranga at this scoring level

The benefits of the proposed research are very relevant to Māori achieving large or significant economic and/or social and/or environmental benefits and are presented as such. Māori contributions are valued and are central to the delivery of benefits.

The implementation pathway is informed by comprehensive engagement with key Māori stakeholders who are participating in and contributing to the research. Distinct risks and sensitivities are well mapped and managed. The role of tikanga Māori throughout the pathway has been identified. Credible IP agreements are in place and respond to identified indigenous knowledge management requirements.

High/highly

Credibility, scale and extent of benefits The potential direct and indirect benefits are very large and will have substantial impacts within a sector and/or significant impact across several sectors. The estimates of benefits are very credible and very thoroughly explained, including the links to stakeholders, next users, end users and beneficiaries able to achieve uptake. Potential benefits are captured by many and diverse areas with very significant spread of benefits.

Implementation pathways The implementation pathways set out are of a high standard and very credible, fit for purpose and the supporting information is satisfactory in scope and competent.

Markedly Comprehensive Insightful Very good/effective Very large Very credible Significant

Relevance and additional value Very relevant and of very substantial potential significance to one or more areas of strategic importance to New Zealand. There is very substantial additional value in areas that are aligned with one or more future directions of investment.

40

Score

Key words

Vision Mātauranga

7 (Very High)

Reference statements for attributes Criteria b: Impact

Sub-criteria i. Sub-criteria ii. Benefit to New Zealand Implementation pathways Weighting: 25% Weighting: 10% Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring.

Indications for identifying Vision Mātauranga at this scoring level

The benefits of the proposed research are highly relevant to Māori achieving their strategic aspirations and presented in this context. Very large or significant economic and/or social and/or environmental benefits are likely to be realised across large natural groupings or sectors. Māori contributions are highly valued and are fundamental to the delivery of benefits.

The implementation pathway is informed by very credible engagement with diverse Māori stakeholders which results in meaningful Māori participation in, and contributions to, the research. Distinct risks and sensitivities are effectively mapped and managed. Appropriate tikanga has been identified throughout the pathway and responsibilities allocated. IP agreements align strongly to indigenous knowledge management best practice approaches.

Very High

Credibility, scale and extent of benefits The potential direct and indirect benefits are extremely large and will have impacts that are nationally significant. The estimates of benefits are highly credible and convincingly explained, including the links to stakeholders, next users, end users and beneficiaries able to achieve uptake. Potential benefits are captured across multiple New Zealand sectors with outstanding spread of benefits.

Implementation pathways The implementation pathways set out are of an extremely high standard, completely credible, fit for purpose and the supporting information is satisfactory in scope and competent.

Excellent Outstanding Exemplary Best international standard Wholly appropriate Very/extremely large

Relevance and additional value Extremely relevant and of potentially transformative significance to one or more areas of strategic importance to New Zealand. There is very large additional value in areas that are aligned with one or more future directions of investment.

Wholly credible Extremely significant

41

Score

Key words Sub-criteria i. Benefit to New Zealand Weighting: 25%

Reference statements for attributes Criteria b: Impact

Sub-criteria ii. Implementation pathways Weighting: 10%

Vision Mātauranga

Assess how satisfactorily the proposal identifies and addresses appropriate Māori-relevant aspects within all sub-criteria attributes above. The following indicates practices (indicative not absolute) you might see at this level of scoring.

Indications for identifying Vision Mātauranga at this scoring level

The benefits of this research are extremely relevant to Māori achieving their strategic aspirations and presented in this context. Extremely large or significant and potentially transformative economic and/or social and/or environmental benefits are likely to be realised across large natural groupings. Māori contributions are very highly valued fundamental to the delivery of benefits.

42

The implementation pathway is informed by exemplary engagement with diverse Māori stakeholders which optimises significant and meaningful Māori participation in, and contributions to, the research. Distinct risks and sensitivities are well mapped and managed. Appropriate tikanga has been identified throughout the pathway and responsibilities allocated. Exemplary IP agreements which model international indigenous knowledge management best practice are in place.

Appendix A Investment Signals for 2016-19 Source Document: MBIE Contestable Research Fund Investment Plan 2016-19

43

44

Appendix B Dimensions of Impact Source Document: MBIE Contestable Research Fund Investment Plan 2016-18

45

Appendix C Some guidance to help you strengthen your approach to Vision Mātauranga Source Document: Guidelines for Completing Proposals 2016 Science Investment Round The Vision Mātauranga policy (Vision Mātauranga) aims to unlock the science and innovation potential within Māori knowledge, resources, and people to assist New Zealanders to create a better future. Giving effect to this policy is a priority across the fund’s economic, environmental and societal objectives; therefore Vision Mātauranga must be addressed in your proposal. This guidance is to help you consider Vision Mātauranga in planning your proposal given this should shape how you develop the proposal, who should be involved, and therefore what content should be included in each of the application sections. Your proposal should show how you have considered, identified and responded to opportunities presented by relevant Māori knowledge, resources or people. This could include, for example, explaining and evidencing: • the steps you have taken to identify research opportunities relevant to Māori interests (collectives, businesses and communities), • which Māori interests will be involved, in what capacity, and the rationale for their selection • what contributions or innovations you will draw from them and how those contributions are integrated in the proposal • how you propose to respond to the distinctive issues, needs and requirements of those Māori interests • which of the four Vision Mātauranga outcome benefits outlined in the Investment Plan you propose to address and how. Please consider if you have provided sufficient information to show: • your analysis of Māori needs, opportunities or resources linked to government and other national strategies, or specific strategies developed by relevant Māori interests • how your proposal responds to relevant values, histories, relationships, rights, aspirations and interests held by related Māori interests • where your research is taking a generic approach, a Māori-centric approach, kaupapa Māori research approach, or a mix across the proposal, and the rationale for this • appropriate Māori voices and expertise relevant to the design of the proposal including the methodology, methods, or applications especially if it includes kaupapa Māori research • the agreed engagement methods or principles specific to the proposal, especially if you are proposing work at the interface between knowledge systems • specific and agreed Māori roles and responsibilities that could include Māori as: researchers, funders, knowledge contributors, participants, end-users, partners, leaders, advisors, or governance members • the specific commitments between your team and Māori eg decision-making, ownership of IP • appropriate use of Māori characterisation • that your processes, solutions, tools, frameworks and metrics in the implementation pathway are relevant to Māori world views, knowledge and context • the international contribution your research will make to the interface between knowledge systems • how the above is being resourced and supported • what risks are associated with the above and how they might be managed and mitigated. If you think Vision Mātauranga is not relevant to your research, you should test this assumption with independent advisors with relevant strategic Vision Mātauranga experience. You will need to provide evidence to explain why you consider Vision Mātauranga is not applicable.

46

Appendix D Assessing contributions to Vision Mātauranga Vision Mātauranga is the science investment policy that applies to all our contestable funds. It seeks to achieve benefits to New Zealand from the innovation potential of Māori knowledge, resources and people. The outcomes being sought through the policy appear in the investment signals section on pages 18-19 of the Investment Plan (Appendix A). Consistent with the goal of a more flexible and responsive science system, in time we are looking at increasingly flexible and dynamic approaches across the research being undertaken to accelerate unlocking Māori innovation potential. The base proposition in our investment approach is that Māori have an interest in all research unless the applicants provide a rationale to show that Vision Mātauranga is not applicable. We expect to see conscious consideration of meaningful and evolving practice in how Māori-held knowledge, resources and people are involved in research and its uptake to deliver excellence and impact. A proposal must show how the applicant has considered, identified and responded to opportunities presented by Māori knowledge, resources and people relevant to the research being undertaken. The critical word here is ‘relevance’ since the Māori interests in, potential contributions to, and outcomes sought from each proposal will be different. Vision Mātauranga-relevant content, its significance in the various components of a proposal, and how you assess its contribution to the overall scoring within each of the sub-criteria will vary for each proposal. Vision Mātauranga is assessed within all of the sub-criteria. It is not a separate or additional assessment. As you assess proposals it is helpful to keep prompting yourself about the Vision Mātauranga elements at each point. For example • when assessing to what extent a proposal recognises related research or existing information (within Research Proposals, Criteria 1 Science) ask yourself how well related Māori research or knowledge has been recognised as part of the whole body of related research or existing information • when assessing to what extent a proposal has strength of relationships with relevant end users, beneficiaries and stakeholders (within Research Proposals, the Implementation Pathway sub-criteria) ask yourself which Māori end users, beneficiaries and stakeholders have been identified, if relevant engagement approaches have been described, if they look credible for Māori, if Māori stakeholders have been fully and correctly identified etc. If a proposal does not address the Vision Mātauranga elements that might reasonably be expected at a particular scoring level, the proposal should not be scored at that level. That is to say, unless the proposal presents a sound case for not including Vision Mātauranga, reasonable Vision Mātauranga elements relative to each scoring level must be evident.

Understanding what we’ve asked applicants Applicants should have undertaken an analysis of Māori needs and opportunities, and have their thinking shaped by that process, before developing the research proposal. In doing so, they can propose that Vision Mātauranga is not relevant. Reflecting this in a statement such as: ‘Vision Mātauranga is not applicable’ is insufficient and provides no evidence about the rationale for this

47

premise. A more credible approach will identify how this premise has been established, for example: “Vision Mātauranga is not relevant to our research because there is not a distinctive Māori need being addressed, or distinctive knowledge base being drawn from. There are Māori interests in uptake of the research and we have tested options with xxx identified on the basis of xxx who confirm they could be interested once xxx.” Where you consider the applicants have NOT provided a credible rationale for discounting Vision Mātauranga, you should identify that in your comments. If you can see opportunities to address Māori needs or draw on Māori knowledge to provide greater value to what has been proposed, then also capture these in your comments. For each criteria, you will need to consider the specific Māorirelevant components within the overall factors considered in each criteria as outlined in the Scoring Guide.

Understanding relevance There are two approaches that can assist in determining what Māori-relevant content is important in a proposal. Firstly, applicants were also asked to consider the material outlined in Appendix C in developing their proposals. You can look to where this material is reflected and evidenced in the proposal, using your judgement to identify how important these elements are and how convincingly they have been addressed. A good proposal will typically reflect clear, specific and relevant Vision Mātauranga components that enrich the overall approach being taken. Secondly, research approaches relevant to Māori interests are broad and varied. They can be characterised as shown in Table 10 or can contain elements of each category. If the research type is not specified, you will get a feel for it from the proposal. The centrality of Māori knowledge grows as you move from left to right in Table 10 as does the importance of Māori characterisation, voice, methodology, and methods. Māori expectations from, and investment in, shaping the research approach and proposal content also generally increases left to right as does the influence of Māoridefined processes. However, each research approach serves in different ways and each proposal should therefore contain those elements of specific Māori-relevance that support the type of research, its focus and the outcomes it proposes to deliver. We expect excellence and impact in every proposal. The approaches outlined in Table 10 help to identify where relevant Vision Mātauranga content is likely to appear in different assessment criteria and the type of influence it may have. Shallow, unsubstantiated, or misdirected attempts to address Vision Mātauranga can tend to reflect activity rather than advancement, irrespective of which category the research can be defined by. As an assessor, you need to be confident that you can navigate the difference.

What to look for The validity of the Vision Mātauranga content in a proposal is context specific. It depends on the issue being addressed, the contribution of Māori-held knowledge, the processes and principles described, and the outcomes sought. Your assessment will involve looking for clear identification, inclusion and evidence of relevant Māori-held knowledge or resources in each proposal and how they contribute to its excellence and impact. Applicant responses should be specific, reflect integrity and be fit for purpose given the focus of the research and its potential impact(s), as should the broader characteristics of the proposal. 48

The assessor training will include further consideration of Vision Mātauranga with examples. These two questions will guide your assessment of Vision Mātauranga in Smart Ideas proposals, within the excellence assessment criteria: 1. Are there opportunities for Māori to contribute to or benefit from the research (ie, is the Vision Mātauranga policy relevant to the impact of the proposal?) 2. Does this proposal propose a credible approach to addressing Vision Mātauranga?

• •

•

•

•

• •

•

•

THINGS TO CONSIDER WHEN ASSESSING VISION MĀTAURANGA RELEVANCE Responses that are credible for some research may not be credible for others depending on the specific research programme and the context of the research being proposed. Proposals should always contain reference to Vision Mātauranga no matter what the level of relevance, ie there needs to be sufficient indication that it has been considered, if only to state it is not relevant and provide rationale to support this assumption. Proposals are more convincing when they contain credible rationale and specific details. For example which Māori organisations should be involved and why; what distinctive knowledge or opportunity is being addressed etc. Generalised and vague references may suggest poorer understanding of how Vision Mātauranga is to be successfully implemented and, typically, the less equipped the team will be to deliver. Do not assume that a researcher, who happens to be of Māori descent, knows about Vision Mātauranga or that a non-Māori researcher won’t. A strong proposal will specify the kind of skills, knowledge, and roles of the team members who will progress the Māori relevant aspects of the proposal and how that capability will be utilised. Do not accept that just mentioning Māori in a proposal is a credible response to the Vision Mātauranga policy. Consider whether what is being stated or proposed is credible, sufficient, and likely to be effective for delivering the full value of the science. If Māori have been identified as partners, end users or beneficiaries of the research, sufficient detail must be provided to show appropriate engagement in delivery of the impact. In considering Vision Mātauranga relevance, intellectual property associated with research involving indigenous species or other matters is often overlooked. We encourage you to identify how intellectual property has been addressed in the context of the proposal. It is highly likely that research involving any aspect of natural resources (ie any proposal addressing the Fund’s environmental objective) will be relevant to Māori given the role of Māori as environmental stewards (Kaitiaki) and decision-making partners under New Zealand Law. The extent and nature of that relevance is a matter of judgement. We encourage you to provide sufficient detail in your comments to support your assessment of how well a proposal has addressed Vision Mātauranga. For example…“the proposal appears to have failed to recognise relevance to Māori and intellectual property implications arising from use of indigenous species for commercial application” rather than …“Vision Mātauranga is not well addressed”.

49

Table 10: New Zealand spectrum of research approaches relating to unlocking Māori potential and the likely implications for assessing proposals Research defined categories Characteristics

Research not specifically involving and not specifically relevant to Māori

Research benefits NZ generally with no additional/ specific impact on Māori and distinctive Māori participation has not been sought. Māori may be involved as participants, researchers, investors, stakeholders on the same basis as their non-Māori counterparts.

Implications for assessment

Research specifically relevant to Māori Research results may contribute to unlocking the innovation potential of Māori and some Māori participation may contribute to maximising the benefit for New Zealand. Māori may: • provide advisory input • provide specific uptake potential as stakeholders • be participants.

Research Involving Māori

Māori-centred research

Research results will specifically contribute to unlocking the innovation potential of Māori.

Research results contribute specifically to unlocking the Māori innovation potential or the research addresses an issue distinct to Māori knowledge, people or resources.

Māori end user involvement is substantive from ideas inception to outcome delivery to a credible pathway shaped through that involvement.

Māori stakeholder involvement is substantive from idea inception to outcome delivery or a credible pathway shaped through that involvement.

Māori may: • provide advisory input or governance • be members of the research team • be participants or subjects whose data may be sought and analysed.

The research focus is more likely to meet some expectations and quality standards set by Māori. Through a variety of roles, Māori are likely to: • shape the research design and analysis • be significant participants • be involved as senior members of research teams • direct the research • open access to taonga (treasures) • provide substantive knowledge and end user involvement that may involve access to networks and communities • be significant implementation agents • be significant and direct beneficiaries of new knowledge and IP.

Kaupapa Māori Research

Research directly contributes to unlocking the innovation potential of Māori knowledge, people and resources. It addresses a distinct Māori issue and the analytical framework may be shaped by a Māori world view. Māori research methodologies are used. Māori end user involvement is substantive from idea inception to outcome delivery or a clear pathway is planned to specifically unlock Māori innovation potential. Core to this approach is: • Māori voice and characterisation • Tikanga Māori and Māori values guide and permeate the research • research that is very targeted to expectations and quality standards set by Māori • creation of new Māori knowledge and IP.

Māori could: • initiate the proposal, define the research problem and design the proposal • identify the outcomes and accountabilities • lead the research team and may constitute the entirety of it • be exclusive participants. Assessment is on a proposal-by-proposal basis, but specific Māori-relevant content is likely to influence as follows: No specific focus is expected on specific Māori issues, knowledge or resources although there may be elements of Māori participation or investment.

Specific Māori-relevant content is more likely to be important in assessing the impact criteria than the excellence criteria.

Specific Māori-relevant content is more likely to influence the research approach in the Science subcriteria and the impact assessment criteria. It may influence Team assessment (mix and skills) but is less likely to be significant that for Māori-centred research.

50

Distinctive Māori input is likely to have shaped the research, may be substantive, and may therefore be important in assessing both the excellence and impact criteria.

Distinctive Māori content is likely to be critical in both the excellence and impact criteria. Scoring criteria should be considered through science and relevant Māori perspectives.