Designing the Experience of a Productive AI Agent
It is August 2023 and the Mozilla Innovation Studio expands the team exploring an AI agent focused on knowledge work. When the UX team were onboarded to Project Alfred, we learned of its primary goal; to save people time while preserving their agency.
Whether it is planning a vacation, a large event or finding the perfect product, we tend to traverse multiple sources of information, researching and collating our findings into some form of plan. This knowledge work can be time consuming, spanning multiple sessions over time.
Could an AI agent perform this work for you, an agent which would research, access live information and compile the findings into a neat package? Could it be done in a transparent way, in which the agent's methods and "working out" could be observed? This was Project Alfred.
To scope the project, the team agreed to focus on the problem of vacation planning. This felt like a meaningful problem to solve, one we had each experienced our own share of pain points with.
At the end of August, the UX team presented a strategy document. The document was designed to achieve a few things:
- Confirm our understanding of the project and its goals.
- Assess our target audience's mental models and expectations.
- Present our summarisation of the value proposition and opportunities.
An opportunity which gathered momentum was something we called the "Scrapbook" concept. Emerging from a conversation within the design team, we discovered that certain stakeholders would create a plan for themselves which would be updated and customised throughout their vacation. Eventually this plan would be decorated with supplementary notes and pictures documenting the experiences of the trip. The crux of the idea being personalisation, customisation and the feeling of ownership.
We needed a method to communicate the scrapbook concept and the user's desire to use it. In early September an interactive storyboard was created, the storyboard documented a proto-persona, Sophie, as she utilised many of Project Alfred's features to assist with her trip to Rome. The storyboard was delivered to the wider organisation in a live call with narration.
This coincided with a comparative assessment, an exploration into other services using AI agents to assist with vacation planning and itinerary creation. With this information we could assess the expectations our users may bring along with helping us refine our competitive differentiators.
By the end of September a number of mobile-first prototypes had been created, each iteration exploring a value proposition of Project Alfred and gauging engineering effort. An area we were keen to test and gather feedback on was the comparison between two form input methods, the new LLM natural language prompt versus typical web form controls.
Through cycles of collecting stakeholder feedback it became clear that the experience of creating a plan came second to the experience of the plan document itself. With AI generated itineraries, functional packing checklists and live travel advisories, we needed to clearly communicate to the user the wealth of useful information they effortlessly receive.
In mid-October we had created a desktop prototype which simulated much of the user journey for Project Alfred. The interactive prototype placed most emphasis on the plan, an editable, multi-page document pre-populated with information related to the user's trip needs. With everyone aligned on where the value of the project lies, it was time to test.
A test plan was created, a document which turned our assumptions into value and usability questions we sought answers for. An unmoderated remote usability test was launched, by the end of November we had just under 3 hours of participant test footage translating to 63 pages of notes and observations.
After five prototypes, various strategy and flow documents, multiple stakeholder meetings and one usability test later, what did we learn? Quite a lot actually.
We learned that despite the freedom of natural language inputs granted by LLMs, clear interface elements and form controls still commanded a strong pull.
We learned that there was no one clear answer for where Project Alfred's value lies, though "Speed to results" narrowly clinched the top spot. Generally however, the vast majority of participants understood what Project Alfred brought as a product, with all but a single participant rating its value as high to extremely high.
This test was a limited prototype however and did not offer participants significant freedom to explore on their own terms, therefore our next test will have more scope for exploration, focusing on observation rather than direct questioning.
So what’s next? It's clear that the project will take shape in various directions as we explore a multitude of potential opportunities. The Innovation Studio continues to design, build and iterate over what it means to be a good, trustworthy AI agent. We have taken our findings throughout the last four months and drafted the next leg of the journey. Watch this space!