Downloaded 14 times
![Step 2: “Connect” to Orion Context Broker§ POST orion/v2/subscriptions/§ Note notification url must be avalid url for Orion container.§ Note the inclusion ofdateModified§ Use the same insertion headers§ FIWARE-service§ FIWARE-servicepath49{"description": "Notify QuantumLeap on luminosity changes on any Lamp","subject": {"entities": [{"idPattern": "Lamp.*"}],"condition": {"attrs": ["luminosity"]}},"notification": {"http": {"url": "http://quantumleap:8668/v2/notify"},"attrs": ["luminosity"],"metadata": ["dateCreated", "dateModified"]},"throttling": 1}](/image.pl?url=https%3a%2f%2fimage.slidesharecdn.com%2fsession5-datapersistencydracocygnussthcometquantumleap-191127083746%2f75%2fData-persistency-draco-cygnus-sth-comet-quantum-leap-50-2048.jpg&f=jpg&w=240)
![Step 3: Get your data§ Check you are sending data toOrion§ Check Orion notifications did notfail§ GET orion/v2/subscriptions§ Get your data§ Nothing?§ Checkout QuantumLeap logs!50# REQUESTcurl -X GET 'http://localhost:8668/v2/entities/Lamp:001/attrs/luminosity?=3&limit=3' -H 'Accept: application/json' -H 'Fiware-Service: openiot' -H 'Fiware-ServicePath: /’# RESPONSE{"data": {"attrName": "luminosity","entityId": "Lamp:001","index": ["2018-10-29T14:27:26","2018-10-29T14:27:28","2018-10-29T14:27:29"],"values": [2000,1991,1998]}}](/image.pl?url=https%3a%2f%2fimage.slidesharecdn.com%2fsession5-datapersistencydracocygnussthcometquantumleap-191127083746%2f75%2fData-persistency-draco-cygnus-sth-comet-quantum-leap-51-2048.jpg&f=jpg&w=240)
Uploaded byFernando Lopez Aguilar
874 views
Data persistency (draco, cygnus, sth comet, quantum leap)
This document provides an overview of different FIWARE solutions for data persistency: Draco, Cygnus, STH Comet, and Quantum Leap. It describes what each solution is used for, their basic architectures, configurations, and APIs. Draco is for persisting context data to MongoDB using Apache NiFi. Cygnus uses Apache Flume to persist Orion context data to storages like HDFS, MySQL, MongoDB. STH Comet is a time series database built on MongoDB for storing historic context data. Quantum Leap uses CrateDB as a time series database to provide historical context data support for NGSIv2.
Related topics:
More Related Content
What's hot
Similar to Data persistency (draco, cygnus, sth comet, quantum leap)
More from Fernando Lopez Aguilar
Recently uploaded
Data persistency (draco, cygnus, sth comet, quantum leap)
- 1.Session 5 -Data Persistency (Draco, Cygnus, STH Comet,Quantum Leap)Fernando López, Cloud & Platform Senior Expertfernando.lopez@fiware.org@flopezaguilarFIWARE Foundation, e.V.
- 2.Learning Goals1● Whatis Data Persistrency?● What is a Time Series Database?● What is the different between Cygnus and Draco compares with Quantum Leapand STH-Comet?● Which main technology is behind is solution?
- 3.2Draco: Persisting ContextData to MongoDB
- 4.Draco GE▪ Ingestionand persistency of data▪ Easy to use, powerful, and reliable system for processing and distributing data▪ Based on Apache NiFi, dataflow system based on the concepts of flow-basedprogramming▪ Support powerful and scalable directed graphs of data routing, transformation, andsystem mediation logic▪ Built to automate the flow of data between systems3
- 5.Features● NGSIv2 Supportboth for ingestion and serialization to have full integration with the Orion Context Broker.● Several persistent backends :● MySQL, the well-know relaEonal database manager.● MongoDB, the NoSQL document-oriented database.● PostgreSQL, the well-know relaEonal database manager.● HDFS, Hadoop distributed file system.● Cassandra, Distributed database.● CartoDB, for geospaEal Data● Templates for some common scenarios● Rest API4
- 6.
- 7.Draco integration inthe FIWARE ecosystem6
- 8.
- 9.References8● FIWARE Catalogueohttps://www.fiware.org/developers/catalogue● FIWARE Dracoo https://fiware-draco.readthedocs.io/en/latest
- 10.
- 11.Cygnus● What isit for?● Cygnus is a connector in charge of persisting Orion context data in certain configured third-partystorages, creating a historical view of such data.● How does it receives context data from Orion Context Broker?● Cygnus uses the subscription/notification feature of Orion.● A subscription is made in Orion on behalf of Cygnus, detailing which entities we want to be notifiedwhen an update occurs on any of those entities attributes.10
- 12.
- 13.Cygnus● Internally, Cygnusis based on Apache Flume, data collection and persistence agents.● An agent is basically composed of● a listener or source in charge of receiving the data,● a channel where the source puts the data once it has been transformed into a Flume event,● a sink, which takes Flume events from the channel in order to persist the data within its body into athird-party storage.12
- 14.Cygnus Architecture● Cygnusruns Flume agents. Thus, Cygnus agents architecture is Flume agents one.13
- 15.Data sinks14§ NGSI-likecontext data in:§ HDFS, the Hadoop distributed file system.§ MySQL, the well-know relational database manager.§ CKAN, an Open Data platform.§ MongoDB, the NoSQL document-oriented database.§ STH Comet, a Short-Term Historic database built on top of MongoDB.§ Kafka, the publish-subscribe messaging broker.§ DynamoDB, a cloud-based NoSQL database by Amazon Web Services.§ PostgreSQL, the well-know relational database manager.§ CartoDB, the database specialized in geolocated data.§ Twitter data in:§ HDFS, the Hadoop distributed file system.
- 16.Cygnus events15§ ASource consumes Events having a specific format, and those Events are delivered to the Source by anexternal source like a web server.§ For example, an AvroSource can be used to receive Avro Events from clients or from other Flume agentsin the flow.§ When a Source receives an Event, it stores it into one or more Channels.§ The Channel is a passive store that holds the Event until that Event is consumed by a Sink.§ One type of Channel available in Flume is the FileChannel which uses the local filesystem as its backingstore.
- 17.Cygnus events16§ ASink is responsible for removing an Event from the Channel and puXng it into an external repositorylike HDFS (in the case of an HDFSEventSink) or forwarding it to the Source at the next hop of the flow.§ The Source and Sink within the given agent run asynchronously with the Events staged in the Channel.
- 18.Cygnus configuration example17§https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/cygnus-ngsi/installation_and_administration_guide/configuration_examples.mdJVMSource Channel Sink
- 19.Multiple persistence backends18JVMSourceChannel2 Sink2Channel1 Sink1Channel3 Sink3
- 20.Multiple agents19JVM2Source2 Channel2Sink2JVMnSourcen Channeln SinknJVM1Source1 Channel1 Sink1…§ One instance for each Agent.§ This add more capability to thesystem.
- 21.Connecting Orion ContextBroker and Cygnus20§ Subscription-notification mechanism of Orion Context Broker.§ Cygnus needs to be notified each time certain entity's attributes change.§ Therefore, Cygnus must subscribe to those entity's attribute changes.
- 22.Configure a basicCygnus agent (Introduction)21§ Edit /usr/cygnus/conf/agent_<id>.conf§ List of sources, channels and sinks:§ cygnusagent.sources = http-source§ cygnusagent.sinks = hdfs-sink§ cygnusagent.channels = hdfs-channel§ Channels configuration§ cygnusagent.channels.hdfs-channel.type = memory§ cygnusagent.channels.hdfs-channel.capacity = 1000§ cygnusagent.channels.hdfs-channel.§ transactionCapacity = 100
- 23.Configure a basicCygnus agent (Sources)22§ Sources configuration:§ cygnusagent.sources.http-source.channels = hdfs-channel§ cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource§ cygnusagent.sources.http-source.port = 5050§ cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler§ cygnusagent.sources.http-source.handler.notification_target = /notify§ cygnusagent.sources.http-source.handler.default_service = def_serv§ cygnusagent.sources.http-source.handler.default_service_path = def_servpath§ cygnusagent.sources.http-source.handler.events_ttl = 10§ cygnusagent.sources.http-source.interceptors = ts de§ cygnusagent.sources.http-source.interceptors.ts.type = timestamp§ cygnusagent.sources.http-source.interceptors.de.type =es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder§ cygnusagent.sources.http-source.interceptors.de.matching_table = /usr/cygnus/conf/matching_table.conf
- 24.Configure a basicCygnus agent (Sinks)23§ Sinks configuration:§ cygnusagent.sinks.hdfs-sink.channel = hdfs-channel§ cygnusagent.sinks.hdfs-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink§ cygnusagent.sinks.hdfs-sink.cosmos_host = cosmos.lab.fi-ware.org§ cygnusagent.sinks.hdfs-sink.cosmos_port = 14000§ cygnusagent.sinks.hdfs-sink.cosmos_default_username = cosmos_username§ cygnusagent.sinks.hdfs-sink.cosmos_default_password = xxxxxxxxxxxxx§ cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs§ cygnusagent.sinks.hdfs-sink.attr_persistence = column§ cygnusagent.sinks.hdfs-sink.hive_host = cosmos.lab.fi-ware.org§ cygnusagent.sinks.hdfs-sink.hive_port = 10000§ cygnusagent.sinks.hdfs-sink.krb5_auth = false
- 25.References24● FIWARE Catalogueohttps://www.fiware.org/developers/catalogue● FIWARE Cygnuso https://fiware-cygnus.readthedocs.io/en/latest
- 26.
- 27.How? (… isnot always the best)§ The Context Broker only stores the latestattribute values:§ Event-driven action-oriented paradigm§ The Short Time Historic adds memory intothe equation:§ Continuous improvement paradigm26
- 28.STH-COMET: What?§ Timeseries database:§ Optimized to deal with values indexed in time§ Raw data vs. Aggregated data§ Basic aggregation concepts:o Rangeo Resolutiono Origino Offset27
- 29.How? (… isnot always the best)§ Collateral aspects to take into consideration:§ Risk:§ Maturity§ Performance§ Expertise§ Flexibility (future evolution)§ Current inversions28
- 30.STH – Comet: How(birds-eye functioning: minimal)29subscribeContextupdatenotificationquery (raw & aggregated)ClientOrionContextBroker
- 31.STH – Comet: How(birds-eye functioning: formal)30subscribeContextupdatenotificationquery (raw & aggregated)ClientOrionContextBrokerCygnusUpdate(raw & aggregated)
- 32.
- 33.Data schemas andpre-pagination▪ Although the STH stores the evolution of (raw) data (i.e., attributes values) in time,its real power comes from the storage of aggregated data▪ The STH should be able to respond to queries such as:• Give me the maximum temperature of this room during the last month (range)aggregated by day (resolution)• Give me the mean temperature of this room today (range) aggregated by hour or evenminute (resolution)• Give me the standard deviation of the temperature of this room this last year (range)aggregated by day (resolution)• Give me the number of times the air conditioner of this room was switched on or off lastMonday (range) aggregated by hour32
- 34.Data schemas andpre-aggregation33
- 35.API: get rawdata34
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.Log level retrievaland update40
- 42.References41● FIWARE Catalogueohttps://www.fiware.org/developers/catalogue● FIWARE STH-Cometo https://fiware-sth-comet.readthedocs.io/en/latest
- 43.42QuantumLeap: Managing theStream of ContextInformation History using Time Series DBs
- 44.Why QuantumLeap§ Providehistorical data support for NGSIv2 data§ Leverage on an efficient time-series database§ Deploy and scale it easily on containerized environments43
- 45.Data is no-moremono dimensional§ The attribute§ Temperature is 20§ The history of the attribute§ Temperature is 20 on 19th May 8:59:01.020 AM CEST§ Temperature is 21 on 19th May 9:12:03.045 AM CEST§ The geo-localisation of the attribute§ Temperature is 20 in 41°51'16.8"N 12°28'15.0"E44 ValueValue in timeValueinspace
- 46.The core ofthe solution45QuantumLeap
- 47.
- 48.Step 1: Extract§QuantumLeap:§ Needs to know where CrateDB is§ Crate:§ Needs a volume to persist data§ Grafana:§ Needs `crate-datasource`§ (or `PostgreSQL`) plugin47QuantumLeap in 4 steps
- 49.Step 2: “Connect”to Orion Context Broker§ Create a Subscription in Orion CB for the entities you are interested in!§ A) Do it directly talking to Orion§ https://fiware-orion.readthedocs.io/en/master/user/walkthrough_apiv2/index.html#subscriptions§ B) Or ask QuantumLeap to do it for you§ https://app.swaggerhub.com/apis/smartsdk/ngsi-tsdb/0.1.1#/input/reporter.reporter.subscribe§ More info:§ https://quantumleap.readthedocs.io/en/latest/user/#orion-subscription48
- 50.Step 2: “Connect”to Orion Context Broker§ POST orion/v2/subscriptions/§ Note notification url must be avalid url for Orion container.§ Note the inclusion ofdateModified§ Use the same insertion headers§ FIWARE-service§ FIWARE-servicepath49{"description": "Notify QuantumLeap on luminosity changes on any Lamp","subject": {"entities": [{"idPattern": "Lamp.*"}],"condition": {"attrs": ["luminosity"]}},"notification": {"http": {"url": "http://quantumleap:8668/v2/notify"},"attrs": ["luminosity"],"metadata": ["dateCreated", "dateModified"]},"throttling": 1}
- 51.Step 3: Getyour data§ Check you are sending data toOrion§ Check Orion notifications did notfail§ GET orion/v2/subscriptions§ Get your data§ Nothing?§ Checkout QuantumLeap logs!50# REQUESTcurl -X GET 'http://localhost:8668/v2/entities/Lamp:001/attrs/luminosity?=3&limit=3' -H 'Accept: application/json' -H 'Fiware-Service: openiot' -H 'Fiware-ServicePath: /’# RESPONSE{"data": {"attrName": "luminosity","entityId": "Lamp:001","index": ["2018-10-29T14:27:26","2018-10-29T14:27:28","2018-10-29T14:27:29"],"values": [2000,1991,1998]}}
- 52.Step 3: Getyour data51
- 53.1. Create aGrafana data-source for each entity type2. Create Grafana dashboards using your datasourceStep 4: Prepare your Dashboards52
- 54.References53● FIWARE Catalogueohttps://www.fiware.org/developers/catalogue● Tutorialo https://fiware-tutorials.readthedocs.io/en/latest/time-series-data/index.html● Codeo https://github.com/smartsdk/ngsi-timeseries-api● Docso https://quantumleap.readthedocs.io/en/latest● APIo https://app.swaggerhub.com/apis/smartsdk/ngsi-tsdb
- 55.
- 56.
- 57.