Test Creation

The iterations building from a 0→1 survey builder experience for mainly non research professionals. These iterations brought more guardrails, repeatable behaviors, and ability for leveling up package offerings.

User Stories • Permissions • Wireframes • Visual Design • User Research

Templates was the first step on repeating processes and creating a self service version of the test creation experience. The project was started before I joined the company and first I needed to instill trust in the company to have them gain trust with the UX process.

Prior to templates there was a singular text box with what the client wanted to know about their customers, and a test type (such as user discovery) to help our researchers craft a test for them. This led to difficulties since clients wanted faster ability to make tests and wanted to know what questions were being asked before they were returned test results.

Quick suggestions

Since the developers were mostly done with the first iteration, I just made a few quick suggestions to:

Update wording from View a template to Use a template, to make the wording more clear that they were not just viewing templates but could use and apply them.
Making the popover double the width (it was still limited to ⅔ the page due to page architecture and time, but was an improvement from the 300px width it was originally).
Make the fill in the blank part of the template look more visibly like it was the only editable part (as opposed to all being inside a text box), but there was not allocated dev time for this initial release.

UX Testing

After this initial release we ran a UX test on 3 customers. Funny enough we were a feedback platform looking to promote customers getting rapid feedback from the market, and there had never been a usability test in our own platform. From the findings we made improvements to the UI, error validation, and added clarity to what the test participant would see.

The following are solutions are improvements from the UX testing.

Updated fill-in-the-blank UI

While being in brackets and a different color, the editable text did not look visually distinct enough from the non-editable template text. B pulling everything out of the input box and putting a line underneath the fill in the blank to make it appear more like a paper mad lib. This was not an ideal solution but it was a good first step to adding some visual distinction, knowing that there were more iterations to quickly follow.

Improved choice path between template and custom test

The call to action to use a template was unnoticeable and hard to find. If the users weren’t asked to use a template in the test they may have not seen it.

Instead we separated the flows at the beginning and had users choose a path between template and create your own, vs the path being chosen by the test type (which meant little or nothing to most users - they just clicked the first option for speed). Templates were surfaced more quickly which helped us get closer to the company goal of 15% tests being created through templates; we managed to increase it from 0% in 3 months to 8% after this release.

Improved error validation

Template required items were sometimes missed and audiences were at the bottom of the screen (often easy to scroll past) that when clicking submit users would get an “Error occured” message but not know or understand what that error was. We improved error validation by highlighting missing elements and adding extra messaging near the submit/save as draft buttons.

Improved visibility into test participant experience

Customers did not know what the description (a question type) meant and who it was for. Similarly for concept test and feature prioritization templates, users wanted to know where and how those fields would be visible to test takers.

I created a few options for these fields, where they could live and how the could populate automatically through the template (after the user entered). After testing the options through the various templates and talking through it with the development team, we settled on the option that was parallel to a you see what you get (what clients type in, is the same way that participants see it). Clients really loved the auto-population of features/concepts/description and it was much more clear to them what test takers actually saw and what was for the Feedback Loop research consulting team to see. Was developed with the next iteration.

Iteration 2

Template improvements and audience creation

The purpose of this release was to improve some of the outstanding UX issues from the first round of testing, add features to templates, and add a customer facing audience creation experience.

Added the ability to edit or remove template questions

From the beta release customers quickly wanted the ability ot be able to edit questions, if the grammar was slightly off. Many customers also liked how they were not starting on a blank slate with questions and would use the prewritten questions as jumping off points, but wanted to make edits from there.

Since the pre edited was vetted by Feedback loops research team we wanted to signal that editing the question means its on them if they write a bad question (while hoping to get more guardrails in the future). With that once edited a question became a “custom” question in our backend. Because of time and developer constraints, questions cannot be saved individually nor can they have places where people add their own variables to make their own template fill in the blank questions. Given that this was a more advanced feature and could mess up the validity of the test we first only gave it to users who were under the persona of a researcher, or those who asked for it knowing the potential consequences.

Additionally, we added the ability to remove most of the questions from the template or reorder the questions. You could not reorder or delete the description/intro questions because those set up the scenario for test participants.

Added to ability to add up to 3 custom questions

Our first step towards a self service survey builder was being able to also add custom questions (if you had the editing permission). You could only add up to 3 custom questions, while there were only 10 questions per test max. We had to go through a few iterations to find the right way to represent these limits to customers.

In this first iteration we also only added a few question types to see what customers did. We wanted to make sure we didn’t have customers write bad questions as we were trying to improve our relationship with our survey participant provider. This took coordination between our data science team, who was monitoring sourcing and dropoffs, to figure out what the best questions types were. We settled on Rate, Single Select and Multi Select.

Made audience demographics visible to customers, along with the ability to create their own audience

Audiences were frustrating since you had to create an audience before creating a test. Users could only choose from audiences that were made by the Feedback Loop research consultants and used on other tests. Additionally to “create an audience” customers could only type a description of the audience and had no control creating the demographics/behaviors.

Added an additional step into test creation where there was an audience library where customers could see audience demographics or create a new audience on the fly.

Improved template selection process

Added a template library for quick flipping through/browsing and descriptions of the purpose of each template. This was a good first step for the finding/selection of templates, however through UXR people felt it was a lot of text and expressed a desire for bullets and keywords. However, they liked being able to flip through for question inspiration. This has led to future initiatives on template organization and choosing guidance, as well as a question bank.

Added test title

To find the test easier and to separate the test title from the question they are trying to solve. Historically, they were the same text box, as the 0→1 version of Feedback Loop was intended to be a user inputting one question they had about something, and tests results would come from there. Seeing as it has expanded to people entering documents and long amounts of texts some of the titles became so long and made tests almost impossible to find. Creating a title and separating it from the test questions was the first step for intentional places for information to go (the next desired step was having a place for adding test stimuli and test questions).

UX Testing Round 2

Easy to lose things if you click out or switch templates in the template library. However, people loved the easy switching and visibility.
Test title was a new behavior and easily missed, until they got an error during error messaging. Additionally it was weird to be able to title the test in every step (got confusing with audience name). However, once users got used to it - about 2 weeks of usage - they only mentioned the location weirdness, and enjoyed having the separate field to find tests easier in the ecosystem.
People would love a preview of tests before it was sent out to participants.
Save as draft was not used, and people did not understand the differences between submit and save. Because save was a new behavior people typically did not click it, and therefore would have tests run when they only intended them to be saved.
Easy to click out of test creation modal and lose everything.
Sometimes the person who created the audience and test questions were two different people. Typically a researcher was creating the audience and a product manager/business user was editing the questions.
Errors were confusing if they were in different steps of the test creation process.
Users couldn’t switch templates or choose custom once saved or submitted.

As the components of creating a test became more complex it became too much to do in one modal and in one flow, so we took the time to reevaluate and build a new draft state of a test, creating a more intentional test creation/editing experience.

Separated the audience flows and test creation flows

Giving the ability for different design and development teams to work separately. It also removed the confusion between audience name and test title. Furthermore this allowed for people to be able to reset sections without having to create a new test.

This went through many iterations.

The first was trying to figure. out if audiences or questions should come first. We started with test questions first, since in the legacy platform that is what. we had. After UXR, all the participants said they wanted to know audiences first so that they could frame their questions around that.

Second, we were trying to figure out how to surface the test question paths one could take, templates or custom. We started with testing three paths: templates, I have a script, and I have questions or ideas. Many of our internal stakeholders were worried about breaking up the two custom paths and wanted to re-evaluate that later on, so we returned to templates vs create your own. In the last iteration we surfaced all the templates, trying to push that behavior more. asa company, and had an option at the bottom to create your own (which led customers down the custom test option). This caused a much higher usage of templates, but surfaced the need for better organization and descriptions of them to be explored later.

Editing and creating tests were the same experience

By condensing the editing and creating of tests into the same experience, it was less instances for the developers to keep track of as well as easier for customers to understand the purpose of a draft state. Customers used to be able to edit in multiple states but with this change they had to return to the draft state to make edits to the test. This led to a lot less bugs and led to less accidental submitting queue submissions before they were ready.

Iteration 4

Self service vs enterprise custom test

Introduced a beta self service test builder

We finally launched a self service beta of the platform, which would be a cheaper (or trial) version of the test experience. These tests would return results faster because automatically launching (aka no human intervention from Feedback Loop admins). For a simple solution to this experience we just created a blank version of a test with up to 10 custom question options. Our goal was to release the beta and see where customer pain points were, or where guardrails were needed for badly written tests.

Some things we quickly discovered were that customers were still nervous to submit a test because they didn’t understand what that meant. Also, we saw that people didn’t know that we already asked demographic screener questions, so we needed to find a way to surface that to customers - this further proved a need for previewing tests before launching.

Added more question types and abilities

We also added the matrix question type, and provided the functionality for rate/matrix labels to be a 0-10 (or any numeric variation of that) endpoint numeric scale or 5 point fully labeled scale.

We looked at competitors (such as Qualtrics and Zappi) for what they did for this, and many allowed a completely free for all option for typing in labels or choosing between suggested and custom labels. Some problems to solve that these didn’t address were:

Bad labels that were not consistent in a completely custom option
Guardrails for mobile view (if a matrix had 10 options of fully labelled scale what would that look like, and would there be a character limit)
Best practices or scales to jump off of for non research professionals

Being inspired by this we decided to find a best of both worlds by providing the already allowed functionality of end point labels. Also we surfacing a library of 5pt fully labelled scales they could choose from that were the most commonly used by our research consultants or followed best practices.

Improved the enterprise custom test input experience

With this new self service model we also had to improve our consultant aided offering for enterprise to help with reducing back and forth between the team and customers.

Problems to solve were:

There were no specific inputs for places. Being free for all upload or one large text box, each test case was a snowflake case depending on the client. There were no specific inputs for instructions/notes for our consultant team, stimuli uploads, questions, etc.
Depending on the client, some clients came with a full script and wanted little to no edits (only checking for spelling errors or things that would not operate well on our participant experience), where others wanted us to fully script a test for them.
Unclear where to put comments for Feedback Loop or teammates, vs what would be customer facing.

After aggregating some of the quantitative usage patterns, we conducted qualitative interviews with our internal stakeholders to better contextualize what was happening within each of these testing use-cases and highlight the pain points & areas of inefficiencies.

With the current testing experience, it took anywhere between 4 to 14 steps for a Research Consultant to execute a single test. There was an opportunity to remove up to 6 steps from this 14 step process with updated flows by simply being clearer with expectations. It took anywhere between 3 to 19 steps for a Customer Success Manager to execute a single test. There was an opportunity to remove up to 12 steps from this 19 step process by simply being clearer with expectations

We went through many iterations and three rounds of UXR to test the wording and order of this experience. What we wanted to know were sentiments and expectations around the level of help (high-medium-low, yes help/no help, max/min guidance, etc) as well as the level of granularity shown with those offerings.

Additionally we tested the order of the selection flow for level of help, adding an asset, and questions. For the first test after internal interviews we had level it in the order of questions, stimuli, and then help. After interviews and A/B testing, we landed on the exact opposite order with help coming first (and selected with guidance on default), then had stimulus, then questions.

Furthermore we wanted to set expectations better on what type of stimuli we supported, and what type of results they may expect after those uploading those stimuli.

At this point I moved off the project as an individual contributor into a manager, passing it off to one of my direct reports.