More topics

Setting up A/B testing

Last updated on November 22, 2024

You can set up and test multiple versions of a form, known as A/B testing.

On this page:

Overview

A/B tests are a common way to see if a change to a form improves performance compared to what's is already being used.

An A/B test - basically providing two versions of a form - can also be used to test a form change with users before final release. For example, if you normally receive submissions by email and want to shift to receiving them through an API, you could "soft launch" the new API form and test how well it works before committing to it completely.

For additional information, see:

Before you begin

There is no automated way to present an "A" or "B" version of a form to a user at random, so you'll need to offer both forms to the user and let them choose which one they want to use.

  • The easiest way to do this is by placing an informational alert on each form page in CMS Lite, advising of the test and providing a link to the other version of the form.
  • If you're using a form library, you can mention the test there, or provide a button link to both versions in content pages.

Ask for assistance and advice

If you want to conduct an A/B test, let us know about it. We can help you in deciding what approach will work best for your scenario and assist in any special configurations or transitions needed.

Setting up an A/B test

How you set up an A/B test basically comes down to whether:

  • You have an existing published form, or
  • You are working with two new forms

Using an existing form as "A"

If you have an existing published form, use it as the "A" option.

  • DO NOT rename the form

Create a "B" version by:

  1. Duplicating the existing form
  2. Adding "-ALT" to the end of the Form Name
  3. Making the change you want to test
  4. Publishing the form

Set up the "B" version for trial by:

  1. Cloning the original form page in CMS Lite and:
    • Removing the excess " - clone" markings
    • Adding "-ALT" to the Nav Title and Page Path
    • Updating the Keywords and Description (not cloned)
  2. Adding an informational alert about the A/B test and linking to the original form
  3. Saving the page
  4. Publishing the page, if you're ready

Set up the "A" version for trial by:

  1. Opening the form page in CMS Lite
  2. Adding an informational alert about the A/B test and inviting users to try the "B" version with a link to it
  3. Saving the page
  4. Publishing the page

Using new forms for "A" and "B"

If you are creating a new form and want to test two versions, pick one design to be the "A" version and treat it like it already existed.

  • FRM-23 ("A" version)
  • FRM-23-ALT ("B" version)

Create an "A" version by:

Create a "B" version by:

  1. Duplicating the "A" form
  2. Adding "-ALT" to the end of the Form Name
  3. Making the change you want to test
  4. Publishing the form

Set up the "B" version for trial by:

  1. Cloning the original form page in CMS Lite and:
    • Removing the excess " - clone" markings
    • Adding "-ALT" to the Nav Title and Page Path
    • Updating the Keywords and Description (not cloned)
  2. Adding an informational alert about the A/B test and linking to the "A" form
  3. Saving the page
  4. Publishing the page, if you're ready

Set up the "A" version for trial by:

  1. Opening the form page in CMS Lite
  2. Adding an informational alert about the A/B test and inviting users to try the "B" version with a link to it
  3. Saving the page
  4. Publishing the page

Collecting feedback

If you're testing a "soft launch" of something, such as sending data to a new application, you may not need to collect feedback from citizens. If you're testing a new design, such as a wizard view versus a normal one, you probably will want citizen feedback.

There are basically 3 approaches you could use:

  1. Collect feedback in-form (good)
    Add a section to the bottom of your form to ask for feedback. You can make one or more of the questions required if you like, and send the data to a separate email address. The data collected would be included in the XML copy of the data, but the XML copy is optional for non-API submissions, plus you can easily ignore or discard that data from the XML if you're sending to an API.
  2. Collect feedback through a link in email (better)
    You can include a link to a feedback form in the email you send to the citizen. This ensures that the feedback data is fully excluded from the form submission, but may be less likely to be provided by the citizen as it's a separate activity and some time may have passed before they provide it.
  3. Collect feedback post-submission (best)
    You can direct citizens to a feedback form after submitting the form, requesting feedback immediately after their experience. Depending on the form though, this approach could impact their experience or post-submission actions, such as downloading the PDF.

Contacting us about your A/B test ahead of time can help with choosing the best approach for your needs.

Releasing the "winner"

You had 2 forms used in your A/B test:

  • FRM-23 ("A" version)
  • FRM-23-ALT ("B" version)

You've concluded your test and decided that the "A" version will be replaced by the "B" version.

Depending on what the change/difference was, you can:

  • Make the change to FRM-23 directly
    • Save, publish and release the form
  • Replace the definition of FRM-23 with an edited version of FRM-23-ALT

If the difference involved a custom configuration, we'll need to adapt the "ALT" configuration to the original form for you. In this case, it may be easier for us to manage the form update/replacement for you as well, since a configuration and form release often need to be coordinated.