GenAI Evaluation KDD2024:

KEYNOTES

Next-Generation Intelligent Assistants for AR/VR Devices

Slides

Abstract: An intelligent assistant shall be an agent that knows you and the world, can receive your requests or predict your needs, and provide you the right services at the right time with your permission. As smart devices such as Amazon Alexa, Google Home, Meta Ray-ban Stories get popular, Intelligent Assistants are gradually playing an important role in people's lives. The Emergence of AR/VR devices brings more opportunities and calls for the next generation of Intelligent Assistants. In this talk, we discuss the many challenges and opportunities we face to grow intelligent assistants from voice-only to multi-modal, from context-agnostic to context-aware, from listening to the users' requests to predicting the user's needs, and from server-side to on-device. We expect these new challenges to open doors to new research areas and start a new chapter for providing personal assistance services.

Bio: Xin Luna Dong is a Principal Scientist at Meta Reality Labs, working on Intelligent Assistant. She has spent more than a decade building knowledge graphs, such as Amazon Product Graph and Google Knowledge Graph. She has co-authored books "Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases" and “Big Data Integration”. She was awarded VLDB Women in Database Research Award (2023) for "significant contributions to knowledge graph construction and data integration", ACM Distinguished Member (2018) for "significant contributions to data and knowledge integration", and VLDB Early Career Research Contribution Award (2016) for “advancing the state of the art of knowledge fusion”. She serves in the VLDB endowment and PVLDB advisory committee, and is a PC co-chair for KDD'2022 ADS track, WSDM'2022, VLDB'2021, and Sigmod'2018.

Graph-based Fusion for Multimodal Learning

Abstract: With the advances in data collection techniques, large amounts of multimodal data collected from multiple sources are becoming available. Such multimodal data can provide complementary information that can reveal fundamental characteristics of real-world subjects. Thus, multimodal machine learning has become an active research area. Extensive works have been developed to exploit multimodal interactions and integrate multi-source information. In this talk, we will discuss using a graph-based multimodal fusion approach to enable multimodal fusion of incomplete data within a heterogeneous graph structure. This approach develops a unique strategy for learning on incomplete multimodal data without data deletion or data imputation. Moreover, we will discuss a dynamic graph-based approach to support federated training over multimodal distributed data without assuming similar active sensors in all clients. The key idea is to employ a dynamic and multi-view graph structure to adaptively capture the correlations amongst multimodal client models.

Short Bio: Dr. Aidong Zhang is Thomas M. Linville Endowed Professor of Computer Science, with joint appointments at Data Science, and Biomedical Engineering at University of Virginia (UVA). Prof. Zhang’s research interests include machine learning, data science, bioinformatics, and health informatics. Prof. Zhang is a fellow of ACM, IEEE, and AIMBE.

Aidong Zhang
Thomas M. Linville Professor of Computer Science
University of Virginia

Call for Contributions

Link to the submission website: [link]

This workshop aims to serve as a pivotal platform for discussing the forefront of Generative AI trustworthiness and evaluation advancements. Generative AI models, such as Large Language Models (LLMs) and Diffusion Models have revolutionized various domains, underscoring the critical need for reliable Generative AI technologies. As these models increasingly influence decision-making processes, establishing robust evaluation metrics and methods becomes paramount. Our objective is to delve into diverse evaluation strategies to enhance Generative AI models reliability across applications. The workshop topics include, but are not limited to:

• Holistic Evaluation: Covering datasets, metrics, and methodologies
• Trustworthiness in Generative AI Models:
• Truthfulness: Counteracting misinformation, hallucination, inconsistency, sycophancy in responses, adversarial factuality.
• Ensuring Safety and Security: privacy concerns, preventing harmful and toxicity content.
• Addressing Bias and Fairness.
• Ethical Considerations: social norm alignment, compliance with values, regulations and laws.
• Privacy: privacy awareness and privacy leakage.
• Enhancing misuse resistance, explainability, and robustness.
• User-Centric Assessment.
• Multi-perspective Evaluation: Emphasizing logical reasoning, knowledge depth, problem-solving, and user alignment.
• Cross-Modal Evaluation: Integrating text, image, audio, etc.

The workshop is designed to convene researchers from the realms of machine learning, data mining, and beyond, fostering the interdisciplinary exploration into Generative AI trustworthiness and evaluation. By featuring a blend of invited talks, presentations of peer-reviewed papers, and panel discussions, this workshop aims to facilitate exchanges of insights and foster collaborations across research and industry sectors. Participants from diverse fields such as Data Mining, Machine Learning, Natural Language Processing (NLP), and Information Retrieval are encouraged to share knowl- edge, debate challenges, and explore synergies, thereby advancing the state of the art in Generative AI technologies.

Submission Guidelines

• Paper submissions are limited to 9 pages, excluding references, must be in PDF and use ACM Conference Proceeding templates (two column format).
• Additional supplemental material focused on reproducibility can be provided. Proofs, pseudo-code, and code may also be included in the supplement, which has no explicit page limit. The supplement format could be either single column or double column. The paper should be self-contained, since reviewers are not required to read the supplement.
• The Word template guideline can be found here: [link]
• The Latex/overleaf template guideline can be found here: [link]
• The submissions will be judged for quality and relevance through single-blind reviewing.
• A paper should be submitted in PDF format through EasyChair at the following link: [link]

Organizers - Amazon Teams