Dealing with multimodal languages ambiguities : a classification and solution method

Abstract

Starting from discussing the problem of ambiguity and its pervasiveness on communication processes, this thesis dissertation faces problems of classifying and solving ambiguities for multimodal languages. This thesis gives an overview of the works proposed in literature about ambiguities in natural language and visual languages and discusses some existing proposals on multimodal ambiguities. An original classification of multimodal ambiguities has been defined using a linguistic perspective, introducing the notions of multimodal grammar, multimodal sentence and multimodal language. An overview of methods that the literature proposes for avoiding and detecting ambiguities has been done. These methods are grouped into: prevention of ambiguities, a-posterior resolution and approximation resolution methods. The analysis of these methods has underlined the suitability of Hidden Markov Models (HMMs) for disambiguation processes. However, due to the complexity of ambiguities for multimodal interaction, this thesis uses the Hierarchical Hidden Markov Models to manage the semantic and syntactic classes of ambiguities for multimodal sentences; this choice permits to operate at different levels going from the terminal elements to the multimodal sentence. The proposed methods for classifying and solving multimodal ambiguities have been used to design and implement two software modules. The experimental results of these modules have underlined a good level of accuracy during the classification and solution processes of multimodal ambiguities.