Abstract

Multimedia communications over IP are booming as they offer higher flexibility and more features than traditional voice and video services. IP telephony known as Voice over IP (VoIP) is one of the commercially most important emerging trends in multimedia communications over IP. Due to the flexibility and descriptive power, the Session Initiation Protocol (SIP) is becoming the root of many sessions-based applications such as VoIP and media streaming that are used by a growing number of users and organizations. The increase of the availability and use of such applications calls for careful attention to the possibility of transferring malformed, incorrect, or malicious SIP messages as they can cause problems ranging from relatively innocuous disturbances to full blown attacks and frauds. Given this scenario, a deep knowledge of the normal behavior of the network and users is essential to problem diagnosis and security protection of IP Telephony. Moreover, analysis tools taking into account service semantics and troubleshooting VoIP systems based on SIP are of paramount importance for network administrators. However, efficient design and deployment of robust and high performance security controlling systems remain a high challenge, in particular due to the open architecture of the Internet, heterogeneous environment and real time communication constraint. This thesis deals with the analysis and protection of services based on the SIP protocol with a special focus on SIP based VoIP applications. The first part of the work is dedicated to the conformance and security analysis of SIP based VoIP services. To this end, our first endeavor is to define a formal conceptual model of VoIP threat domain with the aim to exchange a common vocabulary about the security related information of the domain. We have introduced an ontology defined as “VoIP-Onto" that provides a formal representation of a comprehensive taxonomy of VoIP attacks followed by specific security recommendations and guidelines for protecting the underlying infrastructure from these attacks. The use of “VoIP-Onto" is not only limited to as a general vocabulary and extensible dictionary for sharing domain knowledge about VoIP security, but also can be employed in a real environment for testing or intrusion detection purposes. We have also concentrated on designing synthetic traffic generators considering the difficulties and challenges of collecting real-world VoIP traffic for the purpose of testing monitoring and security controlling tools. To this end, we have introduced “VoIPTG", a generic synthetic traffic generator, that provides flexibility and efficiency in generation of large amount of synthetic VoIP traffic by imitating the realistic behavior profiles for users and attackers. We have also implemented “SIP-Msg-Gen", a SIP fuzzer, capable to generate both the well-formed and fuzzed SIP messages with ease. Then, we focus on designing an on-line filter able to examine the stream of incoming SIP messages and classifies them as “good" or “bad" depending on whether their structure and content are deemed acceptable or not. Because of the different structure, contents and timing of the SIP “bad" messages, their filtering is best carried out by a multistage classifier consisting of deterministic lexical analyzer and supervised machine learning classifiers. The performance and efficiency of our proposed multi-stage filtering system is tested with a large set of SIP based VoIP traffic including both the real and synthetic traces. The experimental result of the filtering system is very promising with high accuracy providing fast attack detection. Next, the focus is shifted on the understanding and modeling the social interaction patterns of users of the VoIP domain. The notion of “social networks" is applied in the context of SIP based VoIP network, where “social networks" of VoIP users are built based on their telephone records. Then, Social Network Analysis (SNA) techniques are applied on these “social networks" of VoIP users to explore their social behavioral patterns. A prototype of filtering system for SIP based VoIP services is also implemented to demonstrate that the knowledge about the social behavior of the VoIP users is helpful in problem diagnosis, intruders detection, and security protection. The filtering system is trained with the normal behavioral patterns of the users. The machine, thus trained, is capable of identifying “malicious" users.