Go Summarize

ML Kit x Material Design: Design Patterns for Mobile Machine Learning (Google I/O'19)

Google Design2019-05-10
type: Conference Talk (Full production);#pr_pr: Google I/O#purpose: Educate
7K views|5 years ago
💫 Short Summary

The video segments explore the integration of machine learning into design systems for enhanced user experiences, emphasizing object detection, tracking, and barcode scanning using ML Kit APIs. Design guidelines focus on user interaction, search patterns, and seamless transitions, with an emphasis on clear communication, error handling, and customization. Collaboration examples with Ikea, Adidas, and Flutter showcase real product flows. The importance of anticipating issues, testing boundaries, and considering environmental conditions in design is highlighted throughout the segments. Overall, the videos aim to optimize user experience through innovative design elements and technological advancements.

✨ Highlights
📊 Transcript
Integration of machine learning into design systems.
Machine learning can make predictions and solve problems without specific instructions.
Well-designed interfaces are important for user understanding of technology benefits.
Machine learning can be harnessed for various product experiences.
Fluid and unpredictable design solutions are needed when machine learning does not go as planned.
Object detection and tracking using ML Kit APIs for visual search patterns.
Design guidelines include familiar and new elements like the reticle and object markers.
Emphasis on the importance of object detection in visual search journeys, allowing users to search by image instead of text.
Technical process involves cropping the camera frame to the detected object and sending it to a machine learning model for classification.
User experience is categorized into sensing, recognizing, and communicating phases.
The sensing phase involves users opening the visual search feature in the app to identify objects like plants or shoes.
First-time users need an explanation of how the feature works, emphasizing a light and fast onboarding experience.
The app uses animation to guide users on how to identify objects with their device.
The app focuses on detecting the largest object in the camera's center using ML Kits API.
To help users troubleshoot detection issues, a detection time is set, and error messages may be displayed.
Best practices for object detection in AR apps.
Avoid triggering a search for every object seen to save costs and meet user expectations.
Use a light signal to confirm user interest, like asking them to hold the camera still over an object briefly.
Set a preset time before initiating a search and pause the live camera feed with a progress indicator.
Establish a minimum size for detected objects and display a thumbnail above results for easy comparison.
Advantages of modal bottom sheets include separating the sheet from the app UI with a scrim, emphasizing results, and providing a way to return to the camera.
Layout for results should be based on app needs, considering content, number of results, and confidence level.
Customize layout based on result type - list/grid for multiple results, tailored layout for single/high-confidence results.
Address low-confidence results by informing users to search again and setting confidence thresholds.
Customize live object detection flow to match app design like Shrine's minimal aesthetic with angled corners and light pink color.
Integration of object detection in static images for user analysis on their device.
ML Kit used for object detection and classification with a custom model in the technical flow process.
Flow divided into input, recognize, and communicate phases with user guidance through onboarding screens.
Emphasis on explaining features to users and providing a seamless experience within the application.
Importance of user interactions in app design process.
Phases like Input, Recognize, and Communicate are crucial for guiding users to desired information.
Strategies include using native selection screens, indeterminate loaders, and object markers for detected objects.
Potential issues like poor image quality and network connection loss are addressed with solutions like error banners and help sections.
Optimizing user experience through clear communication and user-friendly design elements is key.
Enhancing user experience through design elements and themed customization.
Motion transition, tooltips, and markers are used to improve user experience.
Result confidence and error handling in search results are emphasized.
Shrine's visual language is used as an example of themed customization.
Introduction to barcode scanning using ML Kit's barcode scanning API for recognizing barcodes without a separate application.
Highlights of the Barcode Scanning API:
The API can detect common one-dimensional and two-dimensional barcode formats and read the values.
It can automatically parse structured data formats like contact information or event details.
The API simplifies the user experience by providing a prominent area for scanning barcodes and adjusting the aspect ratio for different applications.
For complex formats like PDF 417, a higher quality image is recommended for accurate reading.
The API offers guidance on setting a minimum detection size and provides visual cues for users to improve scanning accuracy.
Importance of communicating loading times when processing barcode information.
The barcode scanning API can read structured and unstructured data.
Structured data is represented using text fields in key-value pairs, while custom information requires layout customization.
Different tasks, like scanning multiple items or completing forms, call for different display methods, such as mobile bottom sheets or modal dialogs.
The goal is to streamline user experience by automating tasks and providing editing options before saving and exiting the feature.
Core design principles for implementing new patterns into products.
Emphasizing minimal and meaningful UI for the camera, keeping users informed at every moment, and anticipating issues by designing for fringe use cases.
Importance of explaining steps to users, organizing flows, and using various communication methods.
Need to test boundaries, understand the model's capabilities and limitations, and consider environmental conditions during design.
Importance of considering user environments in visual search apps.
Common issues applicable to all users regardless of technical familiarity, and the need to adapt design based on result confidence.
Design principles highlighted include minimal and meaningful camera UI, keeping users informed at every moment, and anticipating opportunities for issues.
Collaborations with Ikea, Adidas, and Flutter showcased real product flows and tailored details.
Resources for implementation include ML Kit, dev docs, and demo specific to material design available on ML Kit site and GitHub.