A methodology for generating grammars for multimodal languages

Abstract

Human communication is naturally multimodal. People normally interact through several communication channels, such as gesture, drawing, handwriting, facial expressions, gaze in combination with speech or speech only, which is the prevalent modality. This synergistic use of multiple interaction channels makes human communication flexible, natural and robust. In the last years several efforts have been made to endow computer interface with similar flexibility, naturalness and robustness. The research presented in this thesis represents one of this effort. The main contributions of this thesis are twofold. First of all, it provides a methodology for multimodal language definition that is general enough to be applicable for whatever modalities and in whichever domains. Secondly, it provides an efficient incremental learning algorithm that, following an approach "by example", allows to generate the production rules of the defined grammar starting from the acceptable multimodal sentences.