Hackathon

This project evaluates the reasoning capabilities of Large Language Models (LLMs) on long-form meeting understanding. Using the NOTSOFAR-1 dataset, we compare reasoning based on the reference transcript with E2E MT-ASR transcripts, as well as reasoning from state-of-the-art audio-understanding models. Our evaluation encompasses multiple reasoning tasks including meeting summarization and information retrieval/question-answering, assessing the feasibility of true end-to-end meeting understanding and evaluating whether existing metrics adequately capture faithful, speaker-aware outputs that reflect who said what.

Our project is a real-time speech and LLM-powered co-pilot that addresses the challenge of missed non-verbal cues in remote patient check-ins. It performs parallel analyses of the transcript (what is said) and the prosody (how it's said). When our system detects a word-tone mismatch, it provides the Community Health Worker (CHW) with a discreet, real-time alert and an LLM-generated, context-aware suggestion. This tool moves beyond simple transcription to provide an "empathy-aware" layer, empowering CHWs to catch hidden distress and improve patient outcomes.

This project proposes an Accent-Aware Emotion Understanding Assistant that leverages Large Audio-Language Models (LALMs) to perform direct speech-to-understanding without relying solely on intermediate transcription. Our system captures both linguistic content and paralinguistic cues (tone, pauses, intensity, and rhythm) and produces empathetic reflections. This approach promotes fairness and inclusivity for accented and multilingual speakers by interpreting affective and contextual meaning directly from acoustic signals. The prototype will demonstrate how LALMs can power culturally sensitive, emotionally intelligent spoken interfaces.