Samsung Bixby Apps.
Introducing the design of Bixby voice assistant apps tailored for the Brazilian market, showcasing a variety of local products and services to boost consumer adoption of Bixby
Voice assistants offer a convenient way to interact with mobile and home devices or services, particularly excelling at tasks requiring quick access to information or actions. For instance, they can control smart home devices, check the weather, call a cab, order food, and more.
This case study showcases a project with Samsung Brazil aimed at discovering, designing, and presenting new possibilities for client prospects. The goals were to secure development agreements for releasing their products and services on Samsung's Bixby platform and to provide new experiences and usage scenarios through a voice-first interface. The case study also highlights the challenges of designing and developing Bixby applications, known as "capsules," and aligning these ideas with the business goals of client prospects.
Product Designer (UI/UX)
Lead the project discovery, UX/UI Design and Prototype concepts
The project was managed by SiDi research and development in partnership with Samsung Brazil.
Voice assistants represent innovative channels for delivering consumption and entertainment experiences, showcasing products and services across various segments. The primary goal of this project was to design and propose new methods for companies to market their products and services on the Bixby platform. We aimed to demonstrate Bixby's capabilities and highlight scenarios where a voice assistant can be more effective than traditional touch-based applications.
After releasing Bixby in Portuguese, Samsung needed to enhance its voice recognition and text-to-speech capabilities. This improvement could be achieved by collecting usage data.
The goal was to make Bixby communicate more naturally, using appropriate words and phrases based on various contexts and interactions, making it feel more like conversing with a real person rather than an AI.
In the initial release of Bixby for the Brazilian market, most applications were developed by Samsung. At the time, Samsung Brazil aimed to expand Bixby's offerings by adding more apps and services, with the goal of attracting new audiences, increasing the user base, enhancing retention, and boosting market relevance.
An overview of the Bixby apps proposed for the Brazilian market, designed during the project.
From a design perspective, the most significant challenge was creating innovative and valuable ways for users consume, discover and interact with products and services through a new distribution channel featuring a voice-first interface.
From a business perspective, the challenges involved discoverability and return on investment. How could we ensure users would discover and engage with these services? And how could we persuade business owners to invest in developing APIs for Bixby applications?
Samsung aimed to evolve Bixby beyond just an entertainment platform into an ecosystem where users could access most of their preferred products and services. This represents the ultimate challenge—achieving the goals of both the platform owner and the companies providing their services.
My initial step was to learn about the current Bixby ecosystem and technology, by studying official design documentation and evaluating other capsules that were being developed at the time for the US and European markets. To understand the current landscape of voice assistants, another essential task was to do benchmark research about key competitors like Amazon Alexa and Google assistant, and identify riveting market segments and usage scenarios.
The next step was to plan a design workflow focusing on speed and creativity, to create concepts and capsule proposals in short periods of time, which we would present for potential clients as initial demonstrations, to settle development partnership deals. The idea was to build thRewrite the following in a more fluent way:e most interactive and complete prototype as possible, featuring relevant business cases for each client and users.
During the project, we collaborated with Samsung's business development team to prospect clients from various market segments. We presented the Bixby platform, showcasing how it could serve as a relevant channel for offering their products and services.
To illustrate our ideas, we presented usage scenarios and concepts tailored to both their business needs and the Bixby ecosystem. In some cases, we co-created with prospective clients and users, developing ideas that better aligned with their goals. We reached out to companies in segments such as social media, electronics, food delivery, retail, entertainment, and more, always focusing on the Brazilian market.
For the proposals, I created high-fidelity design prototypes in Sketch. In some instances, the team developed these concepts into interactive voice prototypes using Bixby Studio, allowing for testing and validation. These prototypes demonstrated how Bixby application development works, the speed at which new products and services could be designed and launched on the platform, and the API development and support needed to bring each project to fruition.
We developed everything from proof-of-concept prototypes to fully functional capsules for the companies listed below. The first capsules to be published by Samsung are featured in this design case.
The design process for the project focused on ideating and conceptualizing new Bixby capsules tailored to the unique needs of each business. This approach involved extensive market and product research to identify trends and opportunities for voice UI interactions.
The project phases were divided into two main parts: before the demo and after the demo. In the demo phase, we developed proof-of-concept Bixby apps to present to the business owners of each company. These functional prototypes showcased key features and usage scenarios for their products and services on the Bixby platform.
After validating the proposed concepts with both business owners and users, the project moved forward to the development phase, where we implemented all the proposed features for each Bixby capsule.
Diagram representing the project phases and cycles.
The ideation phase of this project was unique, as each Bixby app concept needed to highlight relevant usage scenarios and demonstrate the potential benefits of the Bixby platform for each business case and users.
The process began by identifying potential business partners and conducting an in-depth study of their products and services to pinpoint opportunities that aligned with Bixby's capabilities and their business objectives. I was responsible for creating and specifying the usage scenarios based on these opportunities.
During this phase, the development team identified available APIs to achieve the proposed goals or requested new ones if necessary. This step was crucial to understanding what was feasible for the proof-of-concept apps.
The design and prototype phase was guided by insights from the ideation sessions and the business goals of each company.
My primary responsibility was to design the complete interaction flows, including visual concepts for new ideas and voice interactions with Bixby. I used Sketch as the main design tool, incorporating components from the Bixby design system as the foundation for the UI. New components were created as needed to meet the specific business goals and interaction proposals.
The voice interaction flows were designed using Samsung's proprietary VUI tool, which allowed for seamless integration with the Bixby code. Interaction flows created and exported from the tool could be implemented without additional coding or modifications.
We validated our designs with end users through usability tests, where we analyzed how users activated and interacted with each Bixby capsule. We also presented various concepts and ideas to gauge their acceptance. I was responsible for planning and organizing these tests, as well as moderating the sessions.
Validating with business owners required a different approach, which is detailed in the following demo presentation section.
The demo phase was a pivotal moment in the project, during which we presented the proof-of-concept prototypes to stakeholders from the involved companies. The goal was to showcase the most relevant usage scenarios for each company and demonstrate how Bixby could effectively feature their products and services.
These demo presentations also served as a kickoff for the project, particularly in cases where the companies decided to proceed with developing the full Bixby capsule and the necessary APIs.
The developed capsules were launched in the Bixby marketplace, where users could access them by prompting Bixby with specific keywords defined for each capsule's goals. This eliminated the need to download a Bixby app beforehand, as activation was instantaneous upon saying the keywords.
Each Bixby goal is triggered by specific keywords and phrases, making it essential to monitor what users prompted when interacting with each capsule. This allowed us to enhance the experience by adding new keywords and goals based on user requests. Further monitoring and analysis after the launch focused on overall usage metrics, user demographics, and spontaneous feedback.
Bixby is a voice assistant that allows users to interact with their Samsung devices more intuitively. It can be activated by saying "Hi Bixby" or pressing a dedicated button. Once activated, Bixby uses automatic speech recognition (ASR) to convert spoken words into text and natural language processing (NLP) to understand the user's intent and context.
Bixby can perform a wide range of tasks such as adjusting device settings, launching apps, making calls, sending texts, retrieving information, and managing calendar events. It also learns from user interactions to provide personalized responses and suggestions. The different abilities to interact with distinct applications are called Bixby Capsules, that are essentially applications developed for Bixby that provide the needed code and API integrations to allow such interaction.
Illustration depicting the basic prompt interaction with Bixby
When designing for a voice-first interface, it is crucial to identify usage scenarios where voice commands are more practical than traditional visual interfaces that rely on clicks and gestures.
Throughout the project, I found that a voice UI is particularly beneficial for multitasking or performing specific, objective tasks that require quick feedback from particular features. Examples include searching for topics and word definitions, checking the weather, managing appointments, getting information about movie times, and more. Prompting Bixby to run features from other applications, such as calling a number, booking a cab, or ordering food, are also valuable use cases.
Bixby features a voice-first interface, however, it also supports various visual components, including cards, images, and buttons. These elements, part of the default Bixby Views library, can be used as is or customized for unique layouts. In this project, we utilized both standard and custom elements, particularly for capsules showcasing rich visual content like product images and movie posters.
Additionally, the use of text-to-speech technology offered significant benefits. For applications involving lengthy text content, such as news articles, Bixby could read the content aloud fluently and clearly. This feature provided a convenient option for users to access such content while multitasking, like driving or working out.
Example of a VUI interaction flow of the Samsung's device repair service Bixby app
Example of Bixby's base UI components library
The main goal of the Samsung Store capsule was to offer a new way to discover and browse the complete lineup of Samsung products available in Brazil, including phones, tablets, TVs, and more. Users could interact with Bixby to ask about product specifications and features, search for specific products, discover new releases and sale deals, and receive product recommendations based on their requests.
The capsule was integrated with the Samsung Store API, allowing users to start a purchase flow by selecting a product and its variations, such as size or color, and adding it to the shopping cart on the Samsung Store website, where the purchase flow could be completed.
Some examples of the capsule's capabilities are shown below.
Samples of the Samsung Shop Bixby capsule
The Health Info capsule was created to provide an alternative way to get trustworthy and updated information about the Covid-19 pandemic in Brazil. Designed and developed during the early days of the pandemic in 2020, it was connected to public health system APIs to deliver official and up-to-date statistics and figures.
Users could ask Bixby about the pandemic and related topics such as medical support, recommendations, symptoms, news, and tips to cope with the stressful situation. The recommendations were based on official guidelines from the Health Ministry, and the news was curated from media partners like Globo, Extra, and Twitter.
Examples of the capsule's features are shown below.
Samples of the Health info capsule, featuring key information about the covid-19 pandemic
As a result of the collaborative partnership with Samsung and nine other client prospects, I designed and prototyped 15 new capsules targeting Brazilian consumers. These capsules introduced innovative ways for products and services to be discovered and experienced through the Bixby platform. In April 2020, we released a Bixby capsule providing essential information about the Covid-19 pandemic, which quickly gained traction, reaching over 100,000 users in its first week. This capsule provided statistical data about the pandemic in Brazil and was integrated with the official Brazilian healthcare system.
Capsules developed for Samsung services, such as the Repair store finder and Store, were launched and continue to be used by local Samsung consumers across their main lineup of products, including smartphones, TVs, and home appliances. Unfortunately, due to an NDA, specific usage figures and metrics cannot be disclosed.
The concepts and prototypes produced during this project successfully experimented with and proposed new use cases for Bixby, representing a significant research and development effort for SiDi in the voice assistant domain.
Reflecting on the challenges, difficulties, and achievements of the project, these are the key insights and lessons I have gained:
All interactions with Bixby were crafted using the VUI tool, which outputs the conversation trees for each goal. This tool functioned as a flow-chart and diagramming utility, providing a highly visual interface that didn't require coding skills. As a result, it was possible to create fully functional prototypes for validation before moving into the development phase.
The development of the proposed concepts and ideas relied heavily on specific APIs to integrate the services with Bixby. In most cases, these APIs were either incomplete or nonexistent, necessitating substantial development efforts from the companies involved. This often led to unexpected delays and technical limitations.
The promotion of products and services typically relies on rich visual presentations, which was true for most Bixby capsules developed during the project. The services that benefited most from voice-only interactions were from specific segments, such as news and banking.
