Movatterモバイル変換

Tutorial on RDF StreamProcessing 2016M.I. Ali, J-P Calbimonte, D. Dell'Aglio,E. Della Valle, and A. Maurihttp://streamreasoning.org/events/rsp2016RDF Stream ProcessingImplementationsJean-Paul Calbimontejean-paul.calbimonte@hevs.ch http://jeanpi.org @jpcik

http://streamreasoning.org/events/rsp2016Share, Remix, Reuse — Legally This work is licensed under the Creative CommonsAttribution 3.0 Unported License. You are free:• to Share — to copy, distribute and transmit the work• to Remix — to adapt the work Under the following conditions• Attribution — You must attribute the work by inserting– “[source http://streamreasoning.org/rsp2014]” at the end ofeach reused slide– a credits slide stating- These slides are partially based on “RDF Stream Processing 2014”by M. Balduini, J-P Calbimonte, O. Corcho, D. Dell'Aglio, E. DellaValle http://streamreasoning.org/rsp2014 To view a copy of this license, visithttp://creativecommons.org/licenses/by/3.0/2

http://streamreasoning.org/events/rsp2016RSP for developers• RDF Streams in practice• RSP Query Engines• Developing with an RSP Engine• Handling Results• RSP Services3

http://streamreasoning.org/events/rsp2016RDF Streams in Practice4

http://streamreasoning.org/events/rsp2016RSP: Keep the data moving5Process data in-streamNot required to storeActive processing modelinput streamsRSPqueries/rules output streams/eventsRDF Streams

http://streamreasoning.org/events/rsp2016 6RDF Stream…GiGi+1Gi+2…Gi+n…unboundedsequenceGi {(s1,p1,o1),(s2,p2,o2),…} [ti]1+ triplesimplicit/explicittimestamp/intervalRDF streams in theoryHow do I code this?Use Web standards?

http://streamreasoning.org/events/rsp2016 7Linked Data on the WebWeb of DataLinked DataW3C Standards: RDF, SPARQL, etc.

http://streamreasoning.org/events/rsp2016 8Linked Data principles for RDF streams?e.g. publish sensor data as RDF/Linked Data?URIs as names ofthingsHTTP URIsuseful information when URIis dereferencedLink to otherURIsusersapplicationsWEBUse RDF model to continuously query real-timedata streams?static vs. streamsone-off vs.continuous

http://streamreasoning.org/events/rsp2016 9(Sensor) Data Streams on the Web9http://mesowest.utah.edu/http://earthquake.usgs.gov/earthquakes/feed/v1.0/http://swiss-experiment.ch• Monitoring• Alerts• Notifications• Hourly/daily update• Myriad of Formats• Ad-hoc access points• Informal description• Convention-semantics• Uneven use of standards• Manual exploration

http://streamreasoning.org/events/rsp2016 10RDF Streams before RDF Streamshttp://richard.cyganiak.de/2007/10/lod/2011Linked Sensor DataMetOfficeAEMET

http://streamreasoning.org/events/rsp2016 11Sensor Data & Linked Data11Zip FilesNumber of TriplesExample: Nevada dataset-7.86GB in n-triples format-248MB zippedAn example: Linked Sensor Datahttp://wiki.knoesis.org/index.php/LinkedSensorData

http://streamreasoning.org/events/rsp2016 12Sensor Data & Linked Data12<http://knoesis.wright.edu/ssw/MeasureData_Precipitation_4UT01_2003_3_31_5_10_00><http://www.w3.org/1999/02/22-rdf-syntax-ns#type><http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#MeasureData> .<http://knoesis.wright.edu/ssw/MeasureData_Precipitation_4UT01_2003_3_31_5_10_00><http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#floatValue>"30.0"^^<http://www.w3.org/2001/XMLSchema#float> .<http://knoesis.wright.edu/ssw/MeasureData_Precipitation_4UT01_2003_3_31_5_10_00><http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#uom><http://knoesis.wright.edu/ssw/ont/weather.owl#centimeters> .<http://knoesis.wright.edu/ssw/Observation_Precipitation_4UT01_2003_3_31_5_10_00><http://www.w3.org/1999/02/22-rdf-syntax-ns#type><http://knoesis.wright.edu/ssw/ont/weather.owl#PrecipitationObservation> .<http://knoesis.wright.edu/ssw/Observation_Precipitation_4UT01_2003_3_31_5_10_00><http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#observedProperty><http://knoesis.wright.edu/ssw/ont/weather.owl#_Precipitation> .<http://knoesis.wright.edu/ssw/Observation_Precipitation_4UT01_2003_3_31_5_10_00><http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#procedure><http://knoesis.wright.edu/ssw/System_4UT01> .<http://knoesis.wright.edu/ssw/Observation_Precipitation_4UT01_2003_3_31_5_10_00><http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#samplingTime><http://knoesis.wright.edu/ssw/Instant_2003_3_31_5_10_00> .<http://knoesis.wright.edu/ssw/Instant_2003_3_31_5_10_00><http://www.w3.org/1999/02/22-rdf-syntax-ns#type><http://www.w3.org/2006/time#Instant> .<http://knoesis.wright.edu/ssw/Instant_2003_3_31_5_10_00><http://www.w3.org/2006/time#inXSDDateTime>"2003-03-31T05:10:00-07:00^^http://www.w3.org/2001/XMLSchema#dateTime" .What do we get in these datasets?Nice triplesWhat is measuredMeasurementUnitSensorWhen is it measured

http://streamreasoning.org/events/rsp2016 13RDF Streams before RDF Streamsi.e. just use RDF:observation1 rdf:type om-owl:Observation .:observation1 om-owl:observedProperty weather:_AirTemperature .:observation1 om-owl:procedure :sensor1 .:observation1 om-owl:result :obsresult1 .:observation1 om-owl:resultTime "2015-01-01T10:00:01":obsresult1 om-owl:floatValue 35.4 .Plain triplesWhere is thetimestamp?:observation2 rdf:type om-owl:Observation .:observation2 om-owl:observedProperty weather:_AirTemperature .:observation2 om-owl:procedure :sensor1 .:observation2 om-owl:result :obsresult2 .:observation2 om-owl:resultTime "2015-01-01T10:00:02":obsresult2 om-owl:floatValue 36.4 .What is the orderin the RDF graph?Appended to a file?Or to some RDF dataset?How to store it?

http://streamreasoning.org/events/rsp2016 14Feed an RDF Stream to a RSP engineAd-hocConversion toRDFLive Non-RDF StreamsRDFRDF datasetsRSPAdd (internal)timestampon insertionWhat is currently done in most RSPsContinuousadditionsRDF +timestamps

http://streamreasoning.org/events/rsp2016 15Feed an RDF Stream to C-SPARQLpublic class SensorsStreamer extends RdfStream implements Runnable {public void run() {..while(true){...RdfQuadruple q=new RdfQuadruple(subject,predicate,object,System.currentTimeMillis());this.put(q);}}}somethingto run ona threadtimestampedtriplethe stream is“observable”Data structure, executionand callbacks are mixedObserver patternTightly coupled listenerAdded timestamp

http://streamreasoning.org/events/rsp2016 16Actor ModelActor1Actor2m No shared mutable stateAvoid blocking operatorsLightweight objectsLoose couplingcommunicatethrough messagesmailboxstatebehaviornon-blocking responsesend: fire-forgetImplementations: e.g. Akka for Java/Scala

http://streamreasoning.org/events/rsp2016 17RDF Streamobject DemoStreams {...def streamTriples={Iterator.from(1) map{i=>...new Triple(subject,predicate,object)}}Data structureInfinitetripleiteratorExecutionval f=Future(DemoStreams.streamTriples)f.map{a=>a.foreach{triple=>//do something}}Asynchronous iterationMessage passingf.map{a=>a.foreach{triple=>someSink ! triple}}send triple toactorImmutable RDF stream avoid shared mutablestate avoid concurrent writes unbounded sequenceIdeas using akka actorsFutures non blocking composition concurrent computations work with not-yet-computed resultsActors message-based share-nothing async distributable

http://streamreasoning.org/events/rsp2016 18RDF Stream… other issues:Graph implementation?Timestamps: application vs system?Serialization? Loose coupling Immutable data streams Asynchronous message passing Well defined input/output

http://streamreasoning.org/events/rsp2016 19Data stream characteristics19Data regularity• Raw data typically collected as time series• Very regular structure.• Patterns can be exploitedE.g. mobile NO2 sensor readings29-02-2016T16:41:24,47,369,46.52104,6.6357929-02-2016T16:41:34,47,358,46.52344,6.6359529-02-2016T16:41:44,47,354,46.52632,6.6363429-02-2016T16:41:54,47,355,46.52684,6.63729...Data order• Order of data is crucial• Time is the key attribute for establishing an order among the data items.• Important for indexing• Enables efficient time-based selection, filtering and windowingTimestamp Sensor ObservedValueCoordinates

http://streamreasoning.org/events/rsp2016 20Feed an RDF Stream to a RSP engineConversion toRDFLive Non-RDF StreamsRDFRDF datasetsRSPAdd (internal)timestampon insertionAdding mappings to the data flowContinuousadditionsRDF +timestamps

http://streamreasoning.org/events/rsp2016 21R2RML Mappings21:ObsValueMaprr:subjectMap [rr:template "http://opensense.epfl.ch/data/ObsResult_NO2_{sensor}_{time}"];rr:predicateObjectMap [rr:predicate qu:numericalValue;rr:objectMap [ rr:column "no2"; rr:datatype xsd:float; ]];rr:predicateObjectMap [rr:predicate obs:uom;rr:objectMap [ rr:parentTriplesMap :UnitMap; ]].:ObservationMaprr:subjectMap [rr:template "http://opensense.epfl.ch/data/Obs_NO2_{sensor}_{time}"];rr:predicateObjectMap [rr:predicate ssn:observedProperty;rr:objectMap [ rr:constant opensense:NO2]];URI of subjectURI of predicateObject: colum nameColumn names in a templateCan be used for mapping both databases, CSVs, JSON, etc

http://streamreasoning.org/events/rsp2016 22An example: TripleWaveRunning modesSources

http://streamreasoning.org/events/rsp2016 23RDF Streams in W3C RSP:g1 {:axel :isIn :RedRoom. :darko :isIn :RedRoom}{:g1, prov:generatedAtTime, "2015-06-18T12:00:00Z"^^xsd:dateTime}:g2 {:axel :isIn :BlueRoom. }{:g2, prov:generatedAtTime, "2015-06-18T12:00:35"^^xsd:dateTime}:g3 {:minh :isIn :RedRoom. }{:g3, prov:generatedAtTime, "2015-06-18T12:02:07"^^xsd:dateTime}...https://www.w3.org/community/rsp/http://streamreasoning.github.io/RSP-QL/RSP_Requirements_Design_Document/Graph-basedFlexible timepropertyRDF-friendlyFlexiblemetadata:g_1 :startsAt "2015-06-18T12:00:00"^^xsd:dateTime:g_1 :endsAt "2015-06-18T13:00:00"^^xsd:dateTime:g_2 :validBetween [:startsAt "2015-06-18T12:00:00"^^xsd:dateTime;:endsAt "2015-06-18T13:00:00"^^xsd:dateTime]Intervals

http://streamreasoning.org/events/rsp2016 24RSP Engine Implementations

http://streamreasoning.org/events/rsp2016Existing RSP systems (oversimplified!) C-SPARQL: RDF Store + Stream processor• Combined architecture CQELS: Implemented from scratch. Focus onperformance• Native + adaptive joins for static-data and streamingdata25RDF StoreStreamprocessorC-SPARQLquerycontinuousresultsNative RSPCQELSquerycontinuousresultstranslator

http://streamreasoning.org/events/rsp2016Existing RSP systems (oversimplified!) SPARQLstream: Ontology-based stream query answering• Virtual RDF views, using R2RML mappings• SPARQL stream queries over the original data streams. EP-SPARQL: Complex-event detection• SEQ, EQUALS operators Instans: RETE-based evaluation26DSMS/CEPSPARQLStreamquerycontinuousresultsrewriterR2RML mappingsPrologengineEP-SPARQLquerycontinuousresultstranslator

http://streamreasoning.org/events/rsp2016Classification of existing systemsModelContinuousexecutionUnion,Join,Optional,FilterAggregatesTimewindowTriplewindowR2SoperatorSequence,Co-ocurrenceTA-SPARQLTA-RDF ✗ ✔ Limited ✗ ✗ ✗ ✗tSPARQL tRDF ✗ ✔ ✗ ✗ ✗ ✗ ✗StreamingSPARQLRDFStream✔ ✔ ✗ ✔ ✔ ✗ ✗C-SPARQLRDFStream✔ ✔ ✔ ✔ ✔ RstreamonlytimefunctionCQELSRDFStream✔ ✔ ✔ ✔ ✔ Istreamonly✗SPARQLStream(Virtual)RDFStream✔ ✔ ✔ ✔ ✗ ✔ ✗EP-SPARQLRDFStream✔ ✔ ✔ ✗ ✗ ✗ ✔Instans RDF ✔ ✔ ✔ ✗ ✗ ✗ ✗27Disclaimer: other features may be missing

http://streamreasoning.org/events/rsp2016 28C-SPARQL

http://streamreasoning.org/events/rsp2016 29A Reminder of SPARQL

http://streamreasoning.org/events/rsp2016 30Where C-SPARQL Extends SPARQL

http://streamreasoning.org/events/rsp2016 31C-SPARQL LanguageQuery and Stream Registration

http://streamreasoning.org/events/rsp2016 32C-SPARQL LanguageQuery and Stream Registration All C-SPARQL queries over RDF streams are continuous• Registered through the REGISTER statement The output of queries is in the form of• Instantaneous tables of variable bindings• Instantaneous RDF graphs• RDF stream Only queries in the CONSTRUCT form can be registered asgenerators of RDF streams Composability:• Query results registered as streams can feed other registeredqueries just like every other RDF stream32

http://streamreasoning.org/events/rsp2016C-SPARQL LanguageQuery registration - Example Using the social stream fb, Who is where?REGISTER QUERY QWhoIsWhereOnFb ASPREFIX : <http://…/sr4ld2014-onto#>SELECT ?room ?personFROM STREAM <http://…/fb> [RANGE 1m STEP 10s]WHERE {?person1 :posts [ :who ?person ; :where ?room ] .} The resulting variable bindings has to be interpreted as aninstantaneous. It expires as soon as the query isrecomputed33

http://streamreasoning.org/events/rsp2016C-SPARQL LanguageStream registration - Example Results of a C-SPARQL query can be stream out for downstream queriesREGISTER STREAM SWhoIsWhereOnFb ASPREFIX : <http://…/sr4ld2014-onto#>CONSTRUCT { ?person :isIn ?room }FROM STREAM <http://…/fb> [RANGE 1m STEP 10s]WHERE {?person1 :posts [ :who ?person ; :where ?room ] .} The resulting RDF triples are streamed out on an RDFstream• More details in the C-SPARQL Engine hands-on session34

http://streamreasoning.org/events/rsp2016C-SPARQL LanguageStream Registration - Notes The output is constructed in the format of an RDF stream. Every query execution may produce from a minimum ofzero triples to a maximum of an entire graph. The timestamp is always dependent on the queryexecution time only, and is not taken from the triples thatmatch the patterns in the WHERE clause.35

http://streamreasoning.org/events/rsp2016 36C-SPARQL LanguageFROM STREAM Clause

http://streamreasoning.org/events/rsp2016C-SPARQL LanguageFROM STREAM Clause FROM STREAM clauses are similar to SPARQL datasets• They identify RDF stream data sources• They represent windows over a RDF stream They define the RDF triples available for querying andfiltering.37

http://streamreasoning.org/events/rsp2016C-SPARQL LanguageFROM STREAM Clause - windows physical: a given number of triples logical: a variable number of triples which occur during agiven time interval (e.g., 1 hour)• Sliding: they are progressively advanced ofa given STEP (e.g., 5 minutes)• Tumbling: they are advanced of exactly their time interval38

http://streamreasoning.org/events/rsp2016C-SPARQL LanguageFROM STREAM Clause - Example Using the social stream fb, how many people are in thesame room? Count on a window of 1 minute that slidesevery 10 secondsREGISTER QUERY HowManyPoepleAreInTheSameRoom ASPREFIX : <http://…/sr4ld2014-onto#>SELECT ?room (COUNT(DISTINCT ?s) as ?person)FROM STREAM <http://…/fb> [RANGE 1m STEP 10s]WHERE {?person1 :posts [ :who ?person ; :where ?room ] .}GROUP BY ?room40

http://streamreasoning.org/events/rsp2016 41C-SPARQL LanguageC-SPARQL reports only snapshotstt+10t+20t+30t+40t+50t+60t+70t+80d1d2d3d1 d1 d1 d1 d1d2 d2 d2 d2d3 d3IncomingtimestampedRDF triplesTime window [RANGE 40s STEP 10s] Windowcontentt+40d1d1, d2d1, d2d1, d2, d3d2, d3t+50 t+60 t+70 t+80

http://streamreasoning.org/events/rsp2016C-SPARQL LanguageMultiple FROM STREAM Clause - Example Using the social stream fb and fs, how many people arein the same room? Count on a window of 1 minute thatslides every 10 secondsREGISTER QUERY HowManyPoepleAreInTheSameRoom ASPREFIX : <http://…/rsp2014-onto#>SELECT ?room (COUNT(DISTINCT ?s) as ?person)FROM STREAM <http://…/fb> [RANGE 1m STEP 10s]FROM STREAM <http://…/fs> [RANGE 1m STEP 10s]WHERE {?person1 :posts [ :who ?person ; :where ?room ] .}GROUP BY ?room42

http://streamreasoning.org/events/rsp2016C-SPARQL LanguageQuery Chaining A C-SPARQL query Q1 registered using the STREAM clausestreams results on an RDF stream A down stream C-SPARQL query Q2 can open a window onthe RDF stream of Q1 using the FROM STREAM clause E.g.,43Is in on 4query4StreamfStreamIs with on fqueryIs In acrossf and 4queryStreamStream:Bob :posts [ :who :Bob ; :where :BlueRoom ] .:Carl :posts [ :who :Carl , :Bob ] .:Bob :isIn :BlueRoom .:Carl :isWith :Bob .:Carl :isIn :BlueRoom .

http://streamreasoning.org/events/rsp2016 44C-SPARQL LanguageTimeStamp Function

http://streamreasoning.org/events/rsp2016C-SPARQL LanguageTimeStamp Function – Syntax and Semantics The timestamp of a triple can be bound to a variable usinga timestamp() function Syntax• timestamp(variable|IRI|bn, variable|IRI, variable|IRI|bn|literal) Semantics45Triple Result of evalutaionIt is not in the window Type ErrorIt appears once in thewindowTimestamp of tripleIt appears multiple times inthe windowThe timestamp of the most recenttriple

http://streamreasoning.org/events/rsp2016C-SPARQL LanguageTimeStamp Function - Example Who is “following” whom?REGISTER QUERY FindFollowers ASPREFIX f: <http://larkc.eu/csparql/sparql/jena/ext#>PREFIX : <http://…/sr4ld2014-onto#>SELECT ?someOne ?someOneElse ?roomFROM STREAM <http://…/isIn> [RANGE 1m STEP 10s]WHERE {?someOne :isIn ?room .?someOneElse :isIn ?room .FILTER(?someOne!=?someOneElse )FILTER (f:timestamp(?someOne :isIn ?room) <f:timestamp(?someOneElse :isIn ?room)}46

http://streamreasoning.org/events/rsp2016C-SPARQL LanguageAccessing background Information C-SPARQL allows for asking the engine to issue the queryalso against RDF graphs using the FROM clauses. E.g., Where else can Alice go?REGISTER QUERY WhereElseCanAliceGo ASPREFIX : <http://…/sr4ld2014-onto#>SELECT ?roomFROM STREAM <http://…/isIn> [RANGE 10m STEP 10m]FROM <http://…/bgInfo>WHERE {?:Alice :isIn ?someRoom .?someRoom :isConnectedTo ?room .}47IRI identifying the graphcontaining thebackground information

http://streamreasoning.org/events/rsp2016C-SPARQL LanguageC-SPARQL queries and reasoning - example Memo• posts is a sub property of observes Data Query under RDFS entailment regimeREGISTER QUERY QueryUnderRDFSEntailmentRegime ASPREFIX : <http://…/sr4ld2014-onto#>SELECT ?x ?room ?personFROM STREAM <http://…/fs> [RANGE 1m STEP 10s]FROM STREAM <http://…/sensors> [RANGE 1m STEP 10s]WHERE { ?x :observes [ :who ?person ; :where ?room ] .} Results at t2 + 10s48RDF graph Time-stamp Stream:RedSensor :observes [ :who :Alice; :where :RedRoom ] . t1 sensors:Bob :posts [ :who :Bob ; :where :RedRoom] . t2 fs?x ?room ?person:RedSensor :RedRoom :Alice:Bob :RedRoom :Bob

http://streamreasoning.org/events/rsp2016 49IntroductionC-SPARQL Engine Architecture Simple, modulararchitecture It relies entirely onexisting technologies Integration of• DSMSs (Esper) and• SPARQL engines (Jena-ARQ)

http://streamreasoning.org/events/rsp2016IntroductionC-SPARQL Engine Features at a glance 1/3 In-memory RDF stream Processing• Continuous queries, filtering, aggregations, joins, sub-queriesvia C-SPARQL• Push based• Reactive C-SPARQL Engine 0.9.5 supports• SPARQL 1.1 (tested with http://www.w3.org/wiki/SRBench)• query chaining• background RDF graph access and update (via SPARQL 1.1Update)• naïve stream reasoning (via Jena Generic Rule Reasoner)• time aware matching via timestamp function50

http://streamreasoning.org/events/rsp2016IntroductionC-SPARQL Engine Features at a glance 2/3 Extensible Middleware• Runtime management of– RDF streams– C-SPARQL query– Result listerners• API driven Quick start available• C-SPARQL Engine– http://streamreasoning.org/download/csparqlreadytogopack Source code are released open source under Apache 2.0• C-SPARQL Engine– https://github.com/streamreasoning/CSPARQL-engine– https://github.com/streamreasoning/CSPARQL-ReadyToGoPack51

http://streamreasoning.org/events/rsp2016IntroductionC-SPARQL Engine Features at a glance 3/3 Known limitations• large background data and timestamp function can spoilperformance• no support for named graphs and named streams• no support for multiple windows on the same stream• triple based windows are buggy52

http://streamreasoning.org/events/rsp2016IntroductionC-SPARQL Engine as general RSP RSP-services proposes a unified interface for the RDFstream processors and offers Rest services to interact withthem. RSP-services-csparql represents the specificimplementation of the RSP-services for the C-SPARQLengine (more detailed information in the hands-on session) Quick start available• RDF Stream Processging RESTful Interface (RSP-service) forC-SPARQL Engine– http://streamreasoning.org/download/rsp-service4csparql Source code are released open source under Apache 2.0• RSP-services– https://github.com/streamreasoning/rsp-services-csparql– https://github.com/streamreasoning/rsp-services-api– https://github.com/streamreasoning/rsp-services-client-example53

http://streamreasoning.org/events/rsp2016Resources Read out more• C-SPARQL semantics– Davide Francesco Barbieri, Daniele Braga, Stefano Ceri, EmanueleDella Valle, Michael Grossniklaus: C-SPARQL: a Continuous QueryLanguage for RDF Data Streams. Int. J. Semantic Computing 4(1):3-25 (2010)• Most recent syntax– D. F. Barbieri, D. Braga, S. Ceri, E. Della Valle, M. Grossniklaus,Querying RDF streams with C-SPARQL, SIGMOD Record 39 (1)(2010) 20–26.• RSP-services– M Balduini,E Della Valle: A Restful Interface for RDF StreamProcessors. International Semantic Web Conference (Posters &Demos) 2013: 209-212 Downloads• http://streamreasoning.org/download/csparqlreadytogopack• http://streamreasoning.org/download/rsp-service4csparql Contact points• marco.balduini@polimi.it• emanuele.dellavalle@polimi.it54

http://streamreasoning.org/events/rsp2016 55SPARQL Stream &Morph-streams

http://streamreasoning.org/events/rsp2016Morph-streams: Overview56QueryrewritingQueryProcessingClientSPARQLStream[tuples][triples/bindings]AlgebraexpressionR2RMLMappingsMorph-streams procesing SPARQLStream queriesSELECT ?proximityFROM STREAM<http://streamreasoning.org/SensorReadings.srdf> [NOW–5 S]WHERE {?obs a ssn:ObservationValue;qudt:numericalValue ?proximity;FILTER (?proximity>10) }SELECT proxFROM sens.win:time(5 sec)WHERE prox >10π timed,proxωσprox>105 SecondssensDatatranslationSNEEEsperGSNCosmpull/pushhttps://github.com/jpcik/morph-streamsOther

http://streamreasoning.org/events/rsp2016SPARQLStream Language57FROM NAMED STREAMISTREAMDSTREAMRSTREAMWINDOWUnderlyingdata sourcerestrictions

http://streamreasoning.org/events/rsp2016SPARQLStream Language NamedStream  ‘FROM’ [‘NAMED’] ‘STREAM’ StreamIRI ‘[’ Window ‘]’ Window  ‘NOW-’ Integer TimeUnit [UpperBound] [Slide] UpperBound  ‘TO NOW-’ Integer TimeUnit Slide  ‘SLIDE’ Integer TimeUnit TimeUnit  ‘MS’ | ‘S’ | ‘MINUTES’| ‘HOURS’ | ‘DAY’ Select  ‘SELECT’ [Xstream] [Distinct | Reduced] … Xstream  ‘RSTREAM’ | ‘ISTREAM’ | ‘DSTREAM’58SELECT ISTREAM ?roomFROM NAMED STREAM <http://www.streamreasoning.org/streams/socialsensor.srdf> [NOW-10 S]WHERE {…

http://streamreasoning.org/events/rsp2016SPARQLStream: examples59PREFIX sr4ld: <http://www.streamreasoning.org/ontologies/socialsensor,owl#>SELECT ?roomFROM NAMED STREAM <http://www.streamreasoning.org/streams/socialsensor.srdf> [NOW-10 S]WHERE {?obs sr4ld:observedBy ?sensor.?obs sr4ld:where ?room.}SPARQLStreamAll rooms where something was observed in the last 10sPREFIX sr4ld: <http://www.streamreasoning.org/ontologies/socialsensor,owl#>SELECT (COUNT(?person) AS ?nmb) ?roomFROM NAMED STREAM <http://www.streamreasoning.org/streams/socialsensor.srdf> [NOW-10 S]WHERE {?obs sr4ld:who ?pers.?obs sr4ld:where ?room.}GROUP BY ?roomNumber of persons observed in each room in the last 10s

http://streamreasoning.org/events/rsp2016Underlying Query ProcessorsEsper• CEP/DSMS• EPL languageSNEE• DSMS/Sensor Network Query Evaluator• Compile queries to sensor codeGSN• Sensor middleware• REST APICosm/Xively• Sensor middleware• Open platform• REST API60SELECT prox FROM sensors [FROM NOW-5MINUTES TO NOW]WHERE prox >10SELECT prox FROM sensors.win:time(5 minute)WHERE prox >10http://montblanc.slf.ch:22001/multidata?vs[0]=sensors&field[0]=proximity_field&c_min[0]=10&from=15/05/2012+05:00:00&to=15/05/2012+10:00:00http://api.cosm.com/v2/feeds/14321/datastreams/4?start=2012-05-15T05:00:00Z&end=2012-05-15T10:00:00Z

http://streamreasoning.org/events/rsp2016Morph-streams: Overview61QueryrewritingQueryProcessingClientSPARQLStream[tuples][triples/bindings]AlgebraexpressionR2RMLMappingsMorph-streams procesing SPARQLStream queriesSELECT ?proximityFROM STREAM<http://streamreasoning.org/SensorReadings.srdf> [NOW–5S]WHERE {?obs a ssn:ObservationValue;qudt:numericalValue ?proximity;FILTER (?proximity>10) }SELECT proxFROM sens.win:time(5 sec)WHERE prox >10π timed,proxωσprox>105 SecondssensDatatranslationSNEEEsperGSNCosmpull/pushhttps://github.com/jpcik/morph-streamsOther

http://streamreasoning.org/events/rsp20163rd: Mapping the two models62ObservationSensorPersonRoomwherewhoobservessubClassOf(person, room,…)detectionsDefine mappings

http://streamreasoning.org/events/rsp2016R2RML – There is a recommendation!63We can use the W3Crecommendation

http://streamreasoning.org/events/rsp2016R2RML - Overview64

http://streamreasoning.org/events/rsp2016Encoding in R2RML65:triplesMap a rr:TriplesMap;rr:logicalTable [ rr:tableName ”sensors"; ]rr:subjectMap [rr:template "http://streamreasoning.org/data/Observation/{person}{timed}";rr:class sr4ld:Observation; rr:graph sr4ld:socialstream.srdf ];rr:predicateObjectMap [rr:predicate sr4ld:who ;rr:objectMap [ rr:template “http://streamreasoning.org/data/Person/{person}” ]];.the streamnamesubject URItriple predicate + objectMapping definitionstreamattributesthe object (a URI in this case)

http://streamreasoning.org/events/rsp2016Underlying Query Processors66SELECT ?proximityFROM STREAM <http://streamreasoning.org/SensorReadings.srdf>[NOW–5 S]WHERE {?obs a ssn:ObservationValue;qudt:numericalValue ?proximity;FILTER (?proximity>10) }SELECT prox FROM sensors [FROM NOW-5 MINUTES TO NOW]WHERE prox >10timed,proxπωσprox>105 SecondssensorsSELECT prox FROM sensors.win:time(5 minute)WHERE prox >10http://montblanc.slf.ch:22001/multidata?vs[0]=sensors&field[0]=proximity_field&c_min[0]=10&from=15/05/2012+05:00:00&to=15/05/2012+10:00:00http://api.cosm.com/v2/feeds/14321/datastreams/4?start=2012-05-15T05:00:00Z&end=2012-05-15T10:00:00ZQueryrewritingR2RMLSNEE (DSMS)Esper (CEP)GSN (middlwr)Cosm XivelySPARQLStream

http://streamreasoning.org/events/rsp2016Underlying query processorsFeatures Esper SNEE GSN Cosm/XivelyProjection ✔ ✔ ✔ FixedProj expression ✔ ✔ ✖ ✖Joins ✔ ✔✖ only window ✖ ✖Union ✖ ✔✖ not windows ✔ ✖Selection ✔ ✔ ✔ ✖✔ limitedAggregates ✔ ✔ ✔✖ ✖Time window ✔ ✔ ✔ ✔Tuple window ✔ ✔ ✔ ✖R2S ✔ ✔ ✖ ✖Conjunction, Disj ✔ ✖ ✖ ✖Repetition pattern ✔ ✖ ✖ ✖Sequence ✔ ✖ ✖ ✖67

http://streamreasoning.org/events/rsp2016Morph-streams: With reasoning!68QueryrewritingQueryProcessingClientSPARQLStream[tuples][triples/bindings]AlgebraexpressionR2RMLMappingsMorph-streams procesing SPARQLStream queriesDatatranslationSNEEEsperGSNCosmpull/pushhttps://github.com/jpcik/morph-streamsOtherOntologyTBoxRewrite taking into accountthe ontology TBox

http://streamreasoning.org/events/rsp2016Reasoning with Morph-streams Rewriting the SPARQLStream queries:69SELECT ?xFROM NAMED STREAM <http://linkeddata.es/s/obs.srdf> [NOW - 5 MINUTES]WHERE {?x ssn:observedBy ?y}SELECT ?xFROM NAMED STREAM <http://linkeddata.es/s/obs.srdf> [NOW - 5 MINUTES]WHERE {{?x ssn:observedBy ?y}UNION{?x a ssn:Observation}UNION{?x a aws:TemperatureObservation}UNION{?x a aws:HumidityObservation}}

http://streamreasoning.org/events/rsp2016Morph-streams: With reasoning!70QueryrewritingQueryProcessingClientSPARQLStream[tuples][triples/bindings]AlgebraexpressionR2RMLMappingsDatatranslationSNEEEsperGSNCosmpull/pushhttps://github.com/jpcik/morph-streamsOtherOntologyTBoxRewrite only happens onceQueryrewriting+translationThen continuous query is registered

http://streamreasoning.org/events/rsp2016Now some codeMorph-streams: Coded in Scala JAR bundle, use it from Scala or Java code Maven, Sbt Examples• One off query• Register continuous query• Pull data• Push• Basic REST https://github.com/jpcik/morph-streams https://github.com/jpcik/morph-web71

http://streamreasoning.org/events/rsp2016Code examples Parse SPARQLStreamval query= “PREFIX sr4ld: <…>. SELECT ?a …”val syntax= StreamQueryFactory.create(query); Execute One-off queryval query= “PREFIX sr4ld: <…>. SELECT ?a …”mapping=Mapping(new URI(mappings/social.ttl))val adapter:QueryEvaluator=Application.adapter(system)val results= adapter.executeQuery(query,mapping)72MappingBindings

http://streamreasoning.org/events/rsp2016Code examples Register and Pullval queryid= adapter.registerQuery(query,mapping)val results1=adapter.pull(queryid)val results2=adapter.pull(queryid) Register and Pushclass ExampleReceiver extends StreamReceiver{override def receiveData(s:SparqlResults):Unit=Logger.debug("got: "+res)}val receiver=new ExampleReceiverval queryid= adapter.listenToQuery(query,mapping,receiver)73Query identifierImplement receiverFor Java users: Exactly the same interface!

http://streamreasoning.org/events/rsp2016 74Querying RSPs in Practice

http://streamreasoning.org/events/rsp2016 75ExecContext context=new ExecContext(HOME, false);String queryString =" SELECT ?person ?loc …ContinuousSelect selQuery=context.registerSelect(queryString);selQuery.register(new ContinuousListener(){public void update(Mapping mapping){String result="";for(Iterator<Var> vars=mapping.vars();vars.hasNext();)result+=" "+ context.engine().decode(mapping.get(vars.next()));System.out.println(result);}});RSP QueryingExample with CQELS (code.google.com/p/cqels)CQELS continuous query:get result updatesaddinglistenerregisterquerySELECT ?person ?loc WHERE {STREAM <http://deri.org/streams/rfid> [RANGE 3s]{?person :detectedAt ?loc}}Tightly coupled listenersResults delivery: push & pull?

http://streamreasoning.org/events/rsp2016 76CQELS fed by a TripleWave WebSocketval conf = ConfigFactory.load.getConfig("experiments.rsp")val qq="""CONSTRUCT { ?s ?p ?o }WHERE {STREAM <ws://localhost:4040/primus> [RANGE 0ms]{?s ?p ?o}}"""val sys=new RspSystem("wstreams")val cqels=new CqelsEnginesys.startStream(Props(new WebSocketStream(cqels,"ws://localhost:4040/primus",conf)))cqels.registerQuery(qq, cqels.createListener(lissy))def lissy(triples:TripleList):Unit={println("tikki: "+triples)}

http://streamreasoning.org/events/rsp2016Similar models,similar (not equals!) query languages77SELECT ?sensorFROM NAMED STREAM <http://www.cwi.nl/SRBench/observations> [NOW-3 HOURS SLIDE 10 MINUTES]WHERE {?observation om-owl:procedure ?sensor ;om-owl:observedProperty weather:WindSpeed ;om-owl:result [ om-owl:floatValue ?value ] . }GROUP BY ?sensor HAVING ( AVG(?value) >= "74"^^xsd:float )SELECT ?sensorWHERE {STREAM <http://www.cwi.nl/SRBench/observations> [RANGE 10800s SLIDE 600s] {?observation om-owl:procedure ?sensor ;om-owl:observedProperty weather:WindSpeed ;om-owl:result [ om-owl:floatValue ?value ] .} }GROUP BY ?sensor HAVING ( AVG(?value) >= "74"^^xsd:float )SELECT ?sensorFROM STREAM <http://www.cwi.nl/SRBench/observations> [RANGE 1h STEP 10m]WHERE {?observation om-owl:procedure ?sensor ;om-owl:observedProperty weather:WindSpeed ;om-owl:result [ om-owl:floatValue ?value ] . }GROUP BY ?sensor HAVING ( AVG(?value) >= "74"^^xsd:float )SPARQLStreamCQELSC-SPARQL

http://streamreasoning.org/events/rsp2016 78Query using SQL on StreamsModelContinuousexecutionUnion,Join,Optional,FilterAggregatesTimewindowTriplewindowR2SoperatorSequence,Co-ocurrenceTimefunctionTA-SPARQL TA-RDF ✗ ✔ Limited✗ ✗ ✗ ✗ ✗tSPARQL tRDF ✗ ✔ ✗ ✗ ✗ ✗ ✗ ✗StreamingSPARQLRDFStream✔ ✔ ✗ ✔ ✔ ✗ ✗ ✗C-SPARQL RDFStream✔ ✔ ✔ ✔ ✔ ✗ ✗ ✔CQELS RDFStream✔ ✔ ✔ ✔ ✔ ✗ ✗ ✗SPARQLStream(Virtual)RDFStream✔ ✔ ✔ ✔ ✗ ✔ ✗ ✗EP-SPARQL RDFStream✔ ✔ ✔ ✗ ✗ ✗ ✔ ✗Instans RDF ✔ ✔ ✔ ✗ ✗ ✗ ✗ ✗W3C RSP review features in existingsystems agree on fundamentaloperators discuss on possiblesemanticshttps://www.w3.org/community/rsp/wiki/RSP_Query_FeaturesRSP is not always/only SPARQL-like queryingSPARQL protocol is not enoughRSP RESTful interfaces?Powerful languages forcontinuous query processing

http://streamreasoning.org/events/rsp2016 79W3C RSP-CG: RSP-QLPREFIX e: <http://somevocabulary.org/>PREFIX s: <http://someinvasivesensornetwork.org/streams#>PREFIX g: <http://somesocialnetwork.org/graphs#>PREFIX : <http://acrasycompany.org/rsp>REGISTER STREAM :GallehaultWasTheBarUNDER ENTAILMENT REGIME <http://www.w3.org/ns/entailment/RIF> AS CONSTRUCT ISTREAM {?poi rdf:type :Gallehault ;:count ?howmanycouples ;:for (?somebody ?someoneelse) }FROM NAMED WINDOW :veryLongWindow ON s:1 [RANGE PT4H STEP PT1H]FROM NAMED WINDOW :longWindow ON s:1 [FROM NOW-PT35M TO NOW-PT5M STEP PT5M]FROM NAMED WINDOW :shortWindow ON s:1 [RANGE PT10M STEP PT5M]FROM NAMED GRAPH g:SocialGraphFROM GRAPH g:POIsWHERE {?poi rdf:type e:bar .WINDOW :veryLongWindow {{?somebody e:enters ?poi} BEGIN AT ?t3{?someoneelse e:enters ?poi} BEGIN AT ?t4FILTER(?t3>?t4)}WINDOW :longWindow {{ ?somebody e:isCloseTo ?someoneelse MINUS {?somebody e:isCloseTo ?yetanotherone .FILTER (?yetanotherone != ?someoneelse) }} WITH DURATION ?durationFILTER (?duration>="PT30M"^^xsd:duration)}WINDOW :shortWindow {{ ?somebody e:exits ?bar} BEGIN AT ?t1{ ?someoneelse e:exits ?bar } BEGIN AT ?t2FILTER (abs(?t2-?t1)<"PT1M"^^xsd:duration )}GRAPH g:SocialGraph {FILTER NOT EXIST { ?somebody e:knows ?someoneelse }}FILTER (?somebody != ?someoneelse)}AGGREGATE { GROUP BY ?poi COUNT(?somebody) AS ?howmanycouples }Continuously look forbars where people arefalling in love (because oa book )RegisterstreamTime windowsSequencingDurationStored GraphsAggregatesAccess to timeReasoningThey entered the same barThey are close to each other,with no-one elseThey get out togetherDidn’t know each other

http://streamreasoning.org/events/rsp2016 80RSP Communication

http://streamreasoning.org/events/rsp2016 81RDF Stream ProcessingRSPEngineRDFgraphsinput RDF streams streams of resultsbackgroundknowledgecontinuous queriesstreamproducersRSPEngineproducersubscribenotifycont. query consumerpush resultssubscribestreamconsumerscontinuous queriesRSP Implementations

http://streamreasoning.org/events/rsp2016 82Reactive SystemsEvent-DrivenJonas Boner. Go Reactive: Event-Driven, Scalable, Resilient & ResponsiveSystems. 2013.Events:reacttoScalableLoad:ResilientFailure:ResponsiveUsers:

http://streamreasoning.org/events/rsp2016 83Actor ModelActor1Actor2m No shared mutable stateAvoid blocking operatorsLightweight objectsLoose couplingcommunicatethrough messagesmailboxstatebehaviornon-blocking responsesend: fire-forgetImplementations: e.g. Akka for Java/ScParentActor1SupervisionhierarchySupervisionActor2Actor4XActor2Actor1Actor2mActor3Actor4mmRemoting

http://streamreasoning.org/events/rsp2016 84RDF Streams: Actorsval sys=ActorSystem.create("system")val consumer=sys.actorOf(Props[RdfConsumer])class Streamer extends StreamRDF{override def triple(triple:Triple){consumer ! triple}}class RdfConsumer extends Actor{def receive= {case t:Triple =>if (t.predicateMatches(RDF.‘type‘))println(s"received triple $t")}RDF consumerActor receive methodImplements behaviorMessage-passing modelRDF producerAsync message passing

http://streamreasoning.org/events/rsp2016 85RSP Producer & ConsumerProcessorfaster producers >> slower processor/consumerProducerProducerProducerProducerRDF streams ConsumerConsumerConsumerunresponsiveoverloadOverload of the processor/receiverUnresponsiveness in stream processor

http://streamreasoning.org/events/rsp2016 87Dynamic Push-PullProducerConsumermdata flowdemand flowPush when consumer is fasterPull when producer is fasterDynamically switch modesCommunication is dynamicdepending on demand vs supplyProducer Consumermmm mmmmmmmpush

http://streamreasoning.org/events/rsp2016 88Evaluation: throughputBasic dynamic pull pushOn top of CQELSLimitations of Thread modelNot yet fully async

http://streamreasoning.org/events/rsp2016 89Reactive RSP workflowsMorphStreamsCSPARQLsEtalisTrOWLss CQELSDynamitesMinimal agreements: standards, serialization, interfacesFormal models for RSPs and reasoningWorking prototypes/systems!Event-driven message passAsync communicationImmutable streamsTransparent RemotingParallel and distributedSupervised Failure HandlingResponsive processingReactive RSPs

http://streamreasoning.org/events/rsp2016 90RSPs and the Linked Data Principles

http://streamreasoning.org/events/rsp2016 91URIs as Names of Thingshttp://mysensorweb.me/mytemperature/20151110Z10:00:00Different conceptshttp://mysensorweb.me/mytemperature/latesthttp://mysensorweb.me/mytemperature/lastMinutehttp://mysensorweb.me/mytemperature/lastMonthDifferent granularitiesDifferent useshttp://mysensorweb.me/mytemperature/avgLastMonthhttp://mysensorweb.me/mytemperature

http://streamreasoning.org/events/rsp2016 92HTTP URIshttp://mysensorweb.me/mytemperature/latestInternet of ThingsHow about XMPP, CoAP, MQTT?Websockets?

http://streamreasoning.org/events/rsp2016 93De-referenceable URIsGET http://mysensorweb.me/mytemperature/latest:Obs1 a my:TemperatureObservation;my:hasValue 33.5 ;my:hasUnit u:Celsius;my:atTime “20151110Z10:00:00”.GET http://mysensorweb.me/mytemperatureGet the whole stream?GET http://mysensorweb.me/mytemperature/lastMonthGet continuous updates?

http://streamreasoning.org/events/rsp2016 94Link to other URIs• Broken links?• Mix streaming and stored data• Persist or not persist?• Volatile links?http://mysensorweb.me/mytemperature/20151110Z10:00:00

http://streamreasoning.org/events/rsp2016 95For Java/Scala developers

http://streamreasoning.org/events/rsp2016 96Semantic Web DevsHave funLove challengesNeed cool toolsRDF4J

http://streamreasoning.org/events/rsp2016 97Scala: Functions and ObjectsJVM languageBoth object and functional orientedEasy Java-interopReuse Java librariesGrowing community

http://streamreasoning.org/events/rsp2016 98RDF in Jena: in ScalaString personURI = "http://somewhere/JohnSmith";Model model = ModelFactory.createDefaultModel();model.createResource(personURI).addProperty(VCARD.FN,"John Smith");TypeinferenceNot too useful ;and ()Terser & compact codeType-safe DSLCompiler takes careval personURI = "http://somewhere/JohnSmith"val model = ModelFactory.createDefaultModelmodel.createResource(personURI).addProperty(VCARD.FN,"John Smith")sw:JohnSmith“John Smith”vcard:FNval personURI = "http://somewhere/JohnSmith"implicit val model = createDefaultModeladd(personURI,VCARD.FN->"John Smith")boilerplateStringconverted toResource

http://streamreasoning.org/events/rsp2016 99Some more RDF99String personURI = "http://somewhere/JohnSmith";String givenName = "John";String familyName = "Smith";String fullName = givenName + " " + familyName;Model model = ModelFactory.createDefaultModel();model.createResource(personURI).addProperty(VCARD.FN,fullName).addProperty(VCARD.N,model.createResource().addProperty(VCARD.Given,givenName).addProperty(VCARD.Family,familyName));val personURI = "http://somewhere/JohnSmith"val givenName = "John"val familyName = "Smith"val fullName = s"$givenName $familyName"implicit val model = createDefaultModeladd(personURI,VCARD.FN->fullName,VCARD.N ->add(bnode,VCARD.Given -> givenName,VCARD.Family->familyName))sw:JohnSmith“John Smith”vcard:FN_:n“John”“Smith”vcard:Nvcard:Givenvcard:FamilyBlank nodeScala DSLs customizablePredicate-objects are pairs

http://streamreasoning.org/events/rsp2016 100Some more RDF in Jenaimplicit val m=createDefaultModelval ex="http://example.org/"val alice=iri(ex+"alice")val bob=iri(ex+"bob")val charlie=iri(ex+"charlie")alice+(RDF.`type`->FOAF.Person,FOAF.name->"Alice",FOAF.mbox->iri("mailto:alice@example.org"),FOAF.knows->bob,FOAF.knows->charlie, FOAF.knows->bnode)bob+ (FOAF.name->"Bob",FOAF.knows->charlie)charlie+(FOAF.name->"Charlie",FOAF.knows->alice)Still valid Jena RDFYou can do it even nicer

http://streamreasoning.org/events/rsp2016 101Exploring an RDF GraphArrayList<String> names=new ArrayList<String>();NodeIterator iter=model.listObjectsOfProperty(VCARD.N);while (iter.hasNext()){RDFNode obj=iter.next();if (obj.isResource())names.add(obj.asResource().getProperty(VCARD.Family).getObject().toString());else if (obj.isLiteral())names.add(obj.asLiteral().getString());}val names=model.listObjectsOfProperty(VCARD.N).map{case r:Resource=>r.getProperty(VCARD.Family).obj.toStringcase l:Literal=>l.getString}Imperative iteration of collectionsType-based conditional executionType castingCase typeMap applied to operators

http://streamreasoning.org/events/rsp2016 102

http://streamreasoning.org/events/rsp2016 103Query with SPARQLval queryStr = """select distinct ?Conceptwhere {[] a ?Concept} LIMIT 10"""val query = sparql(queryStr)query.serviceSelect("http://dbpedia.org/sparql").foreach{implicit qs=>println(res("Concept").getURI)}val f=Future(query.serviceSelect("http://es.dbpedia.org/sparql")).fallbackTo(Future(query.serviceSelect("http://dbpedia.org/sparql")))f.recover{case e=> println("Error "+e.getMessage)}f.map(_.foreach{implicit qs=>println(res("Concept").getValue)})Remote SPARQL endpoinSimplified access toQuery solutionsFutures: asnyc executionNon blocking codeFallback alternative execut

http://streamreasoning.org/events/rsp2016 104Muchas Gracias!

Movatterモバイル変換

Change Language

RDF Stream Processing Tutorial: RSP implementations

Embed presentation

Recommended

More Related Content

What's hot

Similar to RDF Stream Processing Tutorial: RSP implementations

More from Jean-Paul Calbimonte

Recently uploaded

RDF Stream Processing Tutorial: RSP implementations