Product:
Microsoft Dynamics 365 CXP - customer experience management system
User Problem:
Customer experience specialist often is not sure which type of content and through which channel would bring better end result for the designed customer journey. CX analyst would like to feel more confident about what content or channel to choose. To make an informed decision CX specialist often sets up a Version test.
Constraints:
• The feature was designed without fully defined persona.
• The feature was designed while overarching product and parent feature were still being defined.
• Time pressure.
• Limited ability to collaborate due to Covid-19 pandemic
• Because of the time pressure and limited dev resources feature had to be heavily scoped with a plan for expansion post-private preview.
My Role:
• Feature owner
• Collaborating with PM and development team to define feasible scope and ensure quality
• Collaborating with design team to make sure the design of the feature is aligned with overall vision of the new product
Scoping down
At the start we wanted to give users a feature that would help them test very complex scenarios against each other, but the more we worked on this problem the more we understood that if we want to give user statistically significant winner over the reasonable amount of time feature has to be simplified.
Design
Creating initial user flow
Iteration
As we as a team were defining how CXP should look and work I tried to adapt A/B testing feature to frequently changed design of the product.
Testing
Two rounds of testing was done on this feature.
First round major findings:
Majority of users tried to add a content first and then add a version to it.
Major blocker in the flow were Done/ Cancel buttons which were not necessary
Users in general had hard time stopping the test
Second round major findings:
More users were trying to add A/B test module first and specify content later
Users had an easier time stopping the test but stated that would love it to be even easier
A number of usability issues coming from a parent feature was found