Downloaded 151 times
Uploaded byDilum Bandara
PPTX, PDF21,568 views
Message and Stream Oriented Communication
The document discusses communication methods in distributed systems, highlighting message-oriented and stream-oriented communication. It defines persistent vs. transient and asynchronous vs. synchronous communication, detailing their types and applications such as message queuing and pub/sub networks. Additionally, it addresses the architectures and challenges associated with these communication paradigms, including quality of service in stream-based communication.
Related topics:
Recommended
Message and Stream Oriented Communication
- 1.Message & StreamOrientedCommunicationCS4262 Distributed SystemsDilum BandaraDilum.Bandara@uom.lkSome slides extracted from Dr. Srinath Perera & Dr. Rajkumar Buyya’sPresentation Deck
- 2.Outline Message orientedcommunication Event queues Pub/sub networks MPI Stream-based communication Multicast communication2
- 3.Definitions – Persistentvs.Transient Communication Persistent communication Message submitted for transmission is stored by communicationsystem for as long as it takes to deliver it to receiver e.g., e-mail, SMS Not necessary for sender to continue execution after submittinga message Not necessary for receiver to be executing at the time messagesubmission Transient communication Message is stored by communication system only as long assending & receiving applications are executing e.g., transport-level communication services (store-and-forwardrouter) Receiver needs to be there when a message is received3
- 4.Definitions – Asynchronousvs.Synchronous Communication Asynchronous communication Sender continues immediately after it has submittedits message for transmission Message may be stored, in a local buffer at sendinghost or at an intermediate communication server Synchronous communication Sender is blocked until its message is stored in alocal buffer at receiving host or actually delivered toreceiver Strongest form – Sender blocked until receiver hasprocessed message4
- 5.Types of Communication5a)Persistent asynchronous communication (e.g., e-mail)b) Persistent synchronous communicationSource: A.S. Tanenbaum & M.V. Steen, Distributed Systems: Principles and Paradigms
- 6.Types of Communication(Cont.)6c) Transient asynchronous communication (e.g., UDP)d) Receipt-based transient synchronous communicationSource: A.S. Tanenbaum & M.V. Steen, Distributed Systems: Principles and Paradigms
- 7.Types of Communication(Cont.)7e) Delivery-based transient synchronous communicationf) Response-based transient synchronous communicationSource: A.S. Tanenbaum & M.V. Steen, Distributed Systems: Principles and Paradigms
- 8.Types of Communication(Cont.)1. Persistent, asynchronous communication Messages are persistently stored in local host buffer,or at an intermediate communication server e.g., e-mail2. Persistent, synchronous communication Messages can be persistently stored only at receivinghost Weaker form of synchronous communication It isn’t necessary for receiving application to be executing8
- 9.Types of Communication(Cont.)3. Transient, asynchronous communication Messages is temporarily stored in a local buffer &sender immediately continues e.g., UDP, RPC fire & forget4. Transient, synchronous communicationI. Weakest form Receipt-based, transient, synchronous communication Sender is blocked until message is stored in a local buffer ofreceiving host e.g., Asynchronous RPC delivery part (send with Ack)9
- 10.Types of Communication(Cont.)II. Weaker form Delivery-based, transient, synchronous communication e.g., Asynchronous RPC delivery part (Send with Ack)III. Strongest form Response-based, transient, synchronous communication e.g., RPC & RMI10
- 11.Message-Oriented Communication Message-orientedtransient communication Transport-level sockets Message-Passing Interface (MPI) Message transfer latency milliseconds to seconds Message-oriented persistent communication Message-queuing systems or Message-OrientedMiddleware (MOM) Provide intermediate-term storage capacity formessages Doesn’t requiring either sender or receiver to be active duringmessage transmission Message transfer latency seconds to minutes 11
- 12.Message-Queuing Model Applicationscommunicate by inserting messages into aseries of queues Loosely-coupled communication Sender is given guarantee that its message will eventually beinserted in recipient’s queue No guarantee on timing, or message will actually be read12Source: http://msdn.microsoft.com/en-us/library/windows/desktop/ms699870%28v=vs.85%29.aspx
- 13.Message-Queuing ModelCombinations13Loosely-coupled communicationsusing Queues.Sender & receiver can execute completely independent of each other.Source: A.S. Tanenbaum & M.V. Steen, Distributed Systems: Principles and Paradigms
- 14.Message-Queuing Interface Basicinterface to a queue in a message-queuing system14Primitive MeaningPut Append a message to a specified queueGet Block until the specified queue is nonempty, & remove firstmessagePoll Check a specified queue for messages, & remove firstmessage. Never blockNotify Install a handler to be called when a message is put intospecified queue
- 15.Architecture of aTypical Message-Queuing System With Routers15Source: http://csis.pace.edu/~marchese/SE765/L7/L7.htm
- 16.Message Queue ApplicationsAmazon Simple Queue Service (Amazon SQS) Decouple components of a cloud application Can transmit any volume of data, at any level ofthroughput, without losing messages or requiringother services to be always available16Source: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/as-using-sqs-queue.html
- 17.Message Queue Applications(Cont.) Java Message Service (JMS) queues Based on Java Enterprise Edition (JEE) Loosely coupled, reliable, & asynchronous Other applications E-mail Workflow & Groupware Batch processing Job queues Stream/complex-event processing17
- 18.Pub/Sub Networks Publisherspublish messages Usually to a topic Subscribers may expressinterest for a subset ofmessages Pub/Sub system makes sureinterested parties getcorresponding messages 80-90% implementations aretopic based Content based is hard18Source: http://msdn.microsoft.com/en-us/library/ff649664.aspx
- 19.
- 20.Eventing in Pub/SubDecouple Time – both parties need not be online same time Space – don’t know each other’s addresses Synchronization – don’t have to wait for each other Models Event producer Event consumer Event producer Broker Event consumer Actually it’s Event producer Event Bus Notifier Event bus can be 1+ nodes Decoupling makes it is hard to debug20
- 21.Pub/Sub Brokers RSSfeeds Apache ActiveMQ, Qpid OGCE WS-Messenger For web services WSO2 Event Server Microsoft BizTalk Server Distributed brokers Narada Broker (www.naradabrokering.org) Whihdum (http://code.google.com/p/wihidum/)21
- 22.Message Brokers Applicationsneed to understand messages theyreceive Options Standard message formats Not suitable as message-queuing applications typicallyoperate at a higher level of abstraction Convert messages using a Message Broker Convert incoming messages to a format that can beunderstood by destination application22
- 23.
- 24.Message Broker Architecture24Source:A.S. Tanenbaum & M.V. Steen, Distributed Systems: Principles and Paradigms
- 25.Message Bus A.k.a.Enterprise Service Bus (ESB) Tasks Monitor & control routing of message exchange between services Resolve contention between communicating service components Control deployment & versioning of services Marshal use of redundant services25Source: http://msdn.microsoft.com/en-us/library/ff647328.aspx
- 26.Complex Event ProcessingSystem26
- 27.WSO2 Siddhi CEPArchitecture27Source: S. Suhothayan et al., “Siddhi: A Second Look at Complex Event Processing Architectures”, Nov. 2011
- 28.Siddhi Pipeline Architecture28Source:S. Suhothayan et al., “Siddhi: A Second Look at Complex Event Processing Architectures”, Nov. 2011
- 29.Message-Passing Interface (MPI)Designed for communication among parallelapplications Primarily used in HPC systems with high-speedinterconnection networks Provides an interface with advanced featuressuch as different forms of buffering &synchronization Provides hardware independence Supports many types/forms of communication Algorithm/application specific performanceoptimization 29
- 30.MPI Operations30Source: http://www.broadinstitute.org/gatk/about/#high-performanceSource:http://mpitutorial.com/mpi-reduce-and-allreduce/
- 31.MPI_Allreduce31Global sum followedbydistribution of resultSource: Peter Pacheco, "An Introductionto Parallel Programming"
- 32.Butterfly-Structured Global Sum32Source:Peter Pacheco, "An Introduction to Parallel Programming"
- 33.MPI Primitives33Primitive MeaningMPI_bsendAppend outgoing message to local send buffer.MPI_send Send a message & wait until copied to local or remotebuffer.MPI_ssend Send a message & wait until receipt starts.MPI_sendrecv Send a message & wait for reply.MPI_isend Pass reference to outgoing message, and continue.MPI_issend Pass reference to outgoing message, & wait until receiptstarts.MPI_recv Receive a message; block if there is none.MPI_irecv Check if there is an incoming message, but do not block.
- 34.Stream Oriented CommunicationContinuous streams of data e.g., real media stream Modes Asynchronous – no time limit Synchronous – max time limit Isochronous – both max & lower limit Simple stream – One type of streams Complex stream – Many streams e.g., movie with video, 2 audio, & subtitles QoS – bit rate, delay, jitter, etc. Enforcing QoS is a main challenge 34
- 35.Streams (Cont.) EnforcingQOS Mark packets as high priority Use buffers to reduce jitter (play from buffer)35Source: T. Banka et al., “An architecture and a programming interface for application-aware datadissemination using overlay networks,” COMSWARE 2007
- 36.Streams (Cont.) Streamsynchronization Read alternatively Control interface to control rates Distribution – merge at sender36
- 37.Multicast Communication Networklevel – IP multicast Very efficient within LAN No global routing support Application level Main challenge is to setup a path Options Tree based Mesh based Can recover from failures Often used in parallel computing clusters Group communication Ordered reliable multicast 37
- 38.Tree-Push & Mesh-Pull38Source:J.Liu et al., "Opportunities and challenges of peer-to-peerinternet video broadcast,” 2008.X. Zhang et al., "CoolStreaming/DONet: a data-driven overlaynetwork for efficient live media streaming," INFOCOM 2005.