Data Science Tutorials

Slides designed by Cesar Zamudio – www.cesarzamudio.com

Adding Logic to Qualtrics= questions

César Zamudio Assistant Professor of Marketing & Entrepreneurship Kent State University

Slides designed by Cesar Zamudio – www.cesarzamudio.com

Your survey, or parts of it, may not be designed for everyone. For example: • You may want only certain respondents to answer your survey (e.g., females; college students; individuals over 21 years old; parents; etc.). In this case, you need to filter out people that do not satisfy the criteria specified at the very beginning of the survey. This means these respondents will skip THE ENTIRE SURVEY.

• You may want each respondent that sees a certain question to see the items in a different order (e.g., respondent 1 will see Bose, Beats, Sennheiser; respondent 2 will see Beats, Sennheiser, Bose…). In this case you need to randomize the order of the items in certain questions. This means all individuals will answer all questions, but some of these will feature content IN DIFFERENT ORDER.

By adding Logic to Qualtrics survey questions you’ll be able to do all of the above!

A/B Tests

• You may want half of the respondents to answer a question when exposed to a certain stimulus (let’s call it Stimulus A) and the other half to answer the same question when exposed to Stimulus B (e.g., do consumers express more favorability for Logo A or Logo B?; do consumers report to like Advertising A or Advertising B more?). In this case, you need to set up an A/B test with respondents by randomly assigning to them either Stimulus A or Stimulus B. This means all individuals will answer all questions, but HALF will be exposed to Stimulus A and half to Stimulus B.

Randomize

• You may want only certain respondents to answer certain parts of your survey (e.g., only those people who have tried Fanta should answer questions 5 to 12). In this case, you need to redirect people that do not satisfy the criteria further into the survey. This means these individuals will skip SOME questions.

Filter and Redirect

What’s the deal with “Logic” in Qualtrics?

Slides designed by Cesar Zamudio – www.cesarzamudio.com

When to use: When you want only certain respondents to be able to answer the whole survey. • Step 1: Create a filter question. This should be the first question that respondents should see, immediately after the greeting. This question should allow you to identify consumers that you want to include/exclude from your survey, e.g.

2

Filter and Redirect

1. Filtering respondents

3 Randomize

• Step 2: Click the Gear Button next to the filter question. A window with options will appear. • Step 3: Select “Add Skip Logic” from the options window. A grey menu will appear at the bottom of that question.

4

5

• Step 4: Modify the contents of the grey banner to set up your filtering. You need to decide:

• IS statement: Will you filter out those who selected the choice in the IF statement, or those who did NOT? In this example we’ll filter those respondents who selected that they do not smoke. • SKIP TO statement: Where will you send the respondents that do not qualify? Because you want to filter them out of the survey, you can send them to the End of Survey. Alternatively you can send them to the Debriefing block. • Step 5: If you want to delete the Skip Logic created, simply click the “X” button in the grey menu.

A/B Tests

• IF statement: Who should be filtered out? Decide using the first dropdown. In this example we’ll filter those who do not smoke.

Slides designed by Cesar Zamudio – www.cesarzamudio.com

Survey intended for college freshmen. Those who are NOT freshmen will be filtered out.

Filter and Redirect

1. Filtering respondents (Examples)

Randomize Survey intended for respondents without a car loan. Those HAVE a car loan will be filtered out.

A/B Tests

Note: If you need an IRB Consent Form (e.g., for graduate students conducting research) you should place a Block 0 that includes such Consent Form. Then, those who do not accept the Consent should be FILTERED just as in these examples.

Slides designed by Cesar Zamudio – www.cesarzamudio.com

When to use: When you want only certain respondents to be able to answer a particular part of the survey. The strategy: It’s simple. You need to have a redirecting question that will allow you to determine who should answer a certain set of Items, and who should not. For example, suppose you’re interested in knowing more about respondents’ experiences with AXE deodorants. Naturally, you would want only those respondents who have used those deodorants recently to tell you about their experience. Here’s an example of how your survey should look like BEFORE ADDING ANY LOGIC:

• BLOCK 2: Questions about product category  Item 2: Do you shower at least once a week?  Item 3: How often do you purchase products from the health and beauty aisle at your grocery store?  Item 4: Please indicate your agreement with the following statement: I purchase health and beauty products on impulse. • BLOCK 3: Questions about AXE  Item 5 REDIRECTING QUESTION: Have you purchased AXE deodorants in the last three months?  Item 6: How satisfied are you with AXE deodorants?  Item 7: Please list the first three brands that you would consider to purchase instead of AXE on a given visit to a grocery store.  Item 8: How likely is it that your next deodorant purchase will be an AXE deodorant?

• BLOCK 5: Debriefing  Item 12: Debriefing text

Here, only those respondents who reported purchasing AXE deodorants in the last three months should answer Items 6-8. Afterwards, EVERYONE will answer the rest of the questions!

A/B Tests

• BLOCK 4: Demographics questions  Item 9: What is your gender?  Item 10: What is your age?  Item 11: Please indicate your current income.

Randomize

• BLOCK 1: Greeting  Item 1: Greeting text

Filter and Redirect

2. Redirecting respondents (What is it?)

Slides designed by Cesar Zamudio – www.cesarzamudio.com

• Step 1: Create a redirecting question. This question should be immediately above the questions you want only some respondents to answer. This question should allow you to identify consumers that you want to answer a certain portion of the survey, e.g.

2

Filter and Redirect

2. Redirecting respondents

3

• Step 3: Select “Add Skip Logic” from the options window. A grey menu will appear at the bottom of that question.

4

5

Randomize

• Step 2: Click the Gear Button next to the redirecting question. A window with options will appear.

• Step 4: Modify the contents of the grey banner to set up your redirecting. You need to decide:

• IS statement: Will you redirect those who selected the choice in the IF statement, or those who did NOT? In this example we’ll redirect those respondents who selected that they have NOT used AXE products in the last three months. • SKIP TO statement: Where will you redirect the respondents that do not qualify? Because you want to redirect them past the rest of the questions in the block, send them to End of Block. • Step 5: If you want to delete the Skip Logic created, simply click the “X” button in the grey menu.

A/B Tests

• IF statement: Who should be redirected? Decide using the first dropdown. In this example we’ll filter those who have not used an AXE product in the last three months. Those people should NOT answer questions about experiences with AXE products.

Slides designed by Cesar Zamudio – www.cesarzamudio.com

In this redirection, those who HAVE used the new app will be redirected. Thus, for those who HAVE NOT used the app yet, questions about the reason why the app hasn’t been used yet can be included.

Filter and Redirect

2. Redirecting respondents (Examples)

Randomize

In this redirection, those who are NOT business majors will skip the remaining questions in the Block.

A/B Tests

Slides designed by Cesar Zamudio – www.cesarzamudio.com

When to use: When you want each respondents to see choices in a different order. • Step 1: Create any Item that contains multiple choices, for example:

2

Filter and Redirect

3. Randomizing choices within an Item

Randomize

3 • Step 2: Click the Gear Button next to the question. A window with options will appear.

• Step 3: Select “Randomization…” from the options window. The

question, as well as a new window. • Step 4: Select the option “Randomize the order of all choices”. This will make every respondent see the six choices above in a different order.

5

• Step 5: Click “Close”.

A/B Tests

4

symbol will appear next to your

Slides designed by Cesar Zamudio – www.cesarzamudio.com

Filter and Redirect

3. Randomizing choices within an Item (Example of view)

Randomize A/B Tests

Slides designed by Cesar Zamudio – www.cesarzamudio.com

When to use: When you want to test which of two stimuli respondents prefer, or report a higher attitude towards. For example, your company may be deciding between releasing two logos, or two advertising executions. A/B tests will allow you to test whether these are the same or different in respondents’ eyes. The strategy: You need some pieces of information before beginning an A/B test: Stimuli: You need two stimuli (A and B). They may be two competing logos, two ads, etc.



Dependent variable: This is the key marketing metric that the stimuli are supposed to improve. For example, if you’re testing two logos, likelihood of purchase or liking of the logos could be dependent variables.



Think of A and B as two buttons. When you press A, your dependent variable moves. When you press B, your dependent variable moves too. Which button BEST MOVES (“lifts”) THE NEEDLE?

25% 20% 15%

A/B Tests

• And yes, sometimes you may find out that even unexpected stimuli can yield lifts:

30%

% of website visitors who bought

• In the example to the right a website was redesigned. Half of respondents saw the old design and half saw the new one. The research question is which of the two designs • (old or new) generates more purchases? The results clearly show that the new design is superior in terms of the dependent variable, % of website visitors who purchased.

Lift in % of buyers after redesign (N=35,432)

Randomize



Filter and Redirect

4. A/B Tests (What are they?)

10% 5% 0% Old design

New design

Slides designed by Cesar Zamudio – www.cesarzamudio.com

You need two blocks to run an A/B test. We’ll call the the A block and the B block. For example, suppose we are deciding whether we want to make a logo for a new venture red or blue on the basis of (1) how professional respondents judge the logo to be and (2) how much the color reminds them of a pleasant experience in the past. Diagrammatically, then, your survey should look something like this, with a “front end”, the A/B test itself, and then the “back end”:

Filter and Redirect

4. A/B Tests: Block structure

Front-end of the survey (Everyone responds)

A/B Tests

Back-end of the survey (Everyone responds)

Test of Stimulus B “Red Label” (Half of respondents)

Randomize

Test of Stimulus A “Blue Label” (Half of respondents)

Slides designed by Cesar Zamudio – www.cesarzamudio.com

You need two blocks to run an A/B test. We’ll call the the A block and the B block. For example, suppose we are deciding whether we want to make a logo for a new venture red or blue on the basis of (1) how professional respondents judge the logo to be and (2) how much the color reminds them of a pleasant experience in the past. Diagrammatically, then, your survey should look something like this, with a “front end”, the A/B test itself, and then the “back end” (Example of blocks):

Filter and Redirect

4. A/B Tests: Block structure

• BLOCK 1: Greeting  Item 1: Greeting text

• BLOCK 3: A/B TEST BLOCK A: RED  (Show picture of Red Logo)  Item 2: How professional do you think this logo is?  Item 3: Please indicate your agreement to the following statement: “The color blue reminds me of a past pleasant experience”.

• BLOCK 5: Debriefing  Item 7: Debriefing text KEY IDEA: The ONLY THING THAT CHANGES between Block A and Block B is the picture of the logo and the wording of the questions! We will exploit this to construct our A/B test by (1) creating Block A, then (2) copying it and replacing stimulus A with stimulus B.

A/B Tests

• BLOCK 4: Demographics questions  Item 4: What is your gender?  Item 5: What is your age?  Item 6: Please indicate your current income.

Randomize

• BLOCK 2: A/B TEST BLOCK A: BLUE  (Show picture of Blue Logo)  Item 2: How professional do you think this logo is?  Item 3: Please indicate your agreement to the following statement: “The color blue reminds me of a past pleasant experience”.

Slides designed by Cesar Zamudio – www.cesarzamudio.com

• Step 1: Create the COMPLETE BLOCK STRUCTURE of the survey, EXCEPT FOR BLOCK B: Yes, there is NO BLOCK 3 right now!

• Step 2: Add a stimulus (here, a picture) to your Block A. • Step 3: Add the questions to your Block A.

A/B Tests

3

Randomize

2

Filter and Redirect

4. A/B Tests: Block structure

Slides designed by Cesar Zamudio – www.cesarzamudio.com

• Step 4: Collapse the block and, in “Block Options”, select “Copy Block”. Name your block accordingly.

• Look at the new block structure. Note that the new block is at the BOTTOM of the block structure. The B block needs to be put next to the A block.

Randomize

4

Filter and Redirect

4. A/B Tests: Block structure

• Step 5: Go to “Block Options” and select “Move Block Up”. Repeat until Block is in place.

A/B Tests

5

Slides designed by Cesar Zamudio – www.cesarzamudio.com

ONLY THIS IS DIFFERENT

Filter and Redirect

4. A/B Tests: How your blocks should look

Randomize A/B Tests

Slides designed by Cesar Zamudio – www.cesarzamudio.com

We’re almost done! The last step is to tell Qualtrics to randomly assign each respondent to one of two groups: EITHER the A group (Blue) or the B group (Red)! To do so:

1

Filter and Redirect

4. A/B Tests (Randomizer)

• Step 1: In the Survey Options toolbar, click on “Survey Flow”. You will see a new window with your Blocks arranged. • Step 2: Click the block immediately on top of Block A, then click “Add Below”.

Randomize

2

A/B Tests

3

• Step 3: Select Randomizer from the yellow menu that appears.

Slides designed by Cesar Zamudio – www.cesarzamudio.com

• Step 4: Click “Add a New Element Here”. Add a Block, specifically, Block A. Repeat for Block B.

Filter and Redirect

4. A/B Tests (Randomizer)

Randomize

4 A/B Tests

Slides designed by Cesar Zamudio – www.cesarzamudio.com

Your Survey Flow up to this point should look EXACTLY like this:

Filter and Redirect

4. A/B Tests (Randomizer)

Randomize

5 A/B Tests

• Step 5: Notice that Blocks 3 and 4 are now repeated. You don’t need that! Delete both A/B Blocks from the second part of the survey as indicated.

Slides designed by Cesar Zamudio – www.cesarzamudio.com

Your FINAL SURVEY FLOW should look EXACTLY like this:

6

Filter and Redirect

4. A/B Tests (Randomizer)

Randomize

• LAST STEP: In the Randomizer settings, set it to 1. You’re done! Still with us?

A/B Tests

EXTREMELY IMPORTANT: The Randomizer must only show ONE of the two Blocks it contains (either the A or the B Block). As you probably figured out by now, the Randomizer will assign each respondent with one A/B Block at random – either A or B. To make sure this is the case:

Randomize

Questions? = [email protected] www.cesarzamudio.com

Filter and Redirect

Data Science Tutorials

Slides designed by Cesar Zamudio – www.cesarzamudio.com

A/B Tests