Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings
This repository was archived by the owner on Dec 15, 2022. It is now read-only.

Stream Information & Example Applications for Processing JHU and CovidTracking.com COVID-19 data available as streams over Solace

License

NotificationsYou must be signed in to change notification settings

SolaceLabs/covid19-stream-processors

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

54 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

NOTICEThe streams below are no longer continuously running on a publicly available broker. If you would like to use them you can spin up a Solace Broker for free inSolace Cloud and run thePublisher apps in theStream Processors section to publish the event streams.

Overview & Data Sources

This repo provides streams information and example applications on how to consume COVID19 related topics aggregated from different data sources. Below is a list of the different data sources used to feed the published streams.

The Covid Tracking Project

A team from theThe Atlantic launchedThe Covid Tracking Project to collect and publish testing data available for US states and territories.

John Hopkins University

A team atJohns Hopkins University Center for Systems Science and Engineering (JHU CSSE) has developed thisinteractive web-based dashboard to provide researchers, public health authorities, and the general public with a user-friendly tool to track the COVID-19 pandemic as it unfolds. Their team has also been nice enough to periodically (usually once or twice a day) upload the data to thisGithub Repo which has been very popular in the developer community; but has led to developers wanting to receive updates in a more efficient manner. To further their teams’ efforts, help the community, and to “do our part”, we at Solace have created an application that polls the feature service, looks for differences in the data, and publishes updates into an Event Broker we are making available for public use. This allows anyone to consume data updates in near real-time. The data is being published into the event brokers using dynamic topics which allows subscribers to pick, choose and filter on the specific data that they want to consume; e.g., a developer could choose to only get updated when an update it available for a specific Country & Province/State that they are interested in.

How to consume

Any client application could consume those streams by subscribing to the topic of choice as per the diagram below. Note that the streams are provided over Solace PubSub+ Event Broker that has built-in support for a variety of open standard protocols and APIs.

PubSub+ E2E

1. Connection information

Connect to the PubSub+ broker using one of the connection end points below. For more details, check out theSolace Developer Portal and theAPI docs samples for the different messaging APIs and protocols
Supported APIs and Protocols

Connections credentials

Credentials

MessageVPN :covid-19-streams
Client Username :covid-public-client
Client Password:covid19

Connection Points

API/ProtocolHost
SMF Hosttcp://covid.messaging.solace.cloud:55555
Web Messaging Hostws://covid.messaging.solace.cloud:80
Secured Web Messaging Hostwss://covid.messaging.solace.cloud:443
MQTT Hosttcp://covid.messaging.solace.cloud:1883
Secured MQTT Hostssl://covid.messaging.solace.cloud:8883
WebSocket MQTT Hostws://covid.messaging.solace.cloud:8000
AMQP 1.0amqp://covid.messaging.solace.cloud:5672

Note: if you want to receive notifications via a REST end point that supportsPOST, contact us atcovid19-project@solace.com

2. Choose your Topics

Streams are available on these topics

As of right now, the following is the list of streams available for consumption

John Hopkins University

DescriptionSchemaTopicNotes
Raw data for all countries published every ~45 secondsRaw Data Schemajhu/csse/covid19/raw
Updated Active/Deaths/Confirmed/Recovered for a region/state in one eventRegional Update Schemajhu/csse/covid19/cases/region/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Deaths for a region/stateUpdate Type Schemajhu/csse/covid19/cases/deaths/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Active cases for a region/stateUpdate Type Schemajhu/csse/covid19/cases/active/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Confirmed cases for a region/stateUpdate Type Schemajhu/csse/covid19/cases/confirmed/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Recovered cases for a region/stateUpdate Type Schemajhu/csse/covid19/cases/recovered/update/{attributes.countryRegion}/{attributes.provinceState}
Shows the updated percentage of the population affectedUpdate Population Schemajhu/csse/covid19/cases/active/population/update/US/{attributes.provinceState}Only US for now

The Covid Tracking Project

DescriptionSchemaTopicNotes
Raw data for all the states in the US published every ~45 secondsRaw Data Schemacom/covidtracking/states/current/get/raw
Updated US state informationRaw Data Schemacom/covidtracking/states/current/update/{state}

Subscribe to one or more of the available topics above to receive the required data.Note that the streams defined asupdate above only send events when updates actually occur which can be infrequently. For development purposes we are providingtest topics that get published every few minutes

For more more information on using topic wildcards check outSMF Topic Subscriptions andMQTT Topic Subscriptions

Note: If you have your own COVID-19 event stream that you'd like to share please contact us atcovid19-project@solace.com

Example topic subscriptions

Subscribing to the following streams will give you the corresponding results:

SubscriptionResult
jhu/csse/covid19/cases/region/update/>All region updates
jhu/csse/covid19/cases/region/update/US/>All updates for country US
jhu/csse/covid19/cases/region/update/US/FloridaAll updates where state is Florida
jhu/csse/covid19/cases/recovered/update/Canada/OntarioAll recovered in Ontario, Canada
jhu/csse/covid19/cases/active/update/Canada/*All active in Canada
jhu/csse/covid19/cases/deaths/update/Brazil/*All deaths in Brazil (Note: country with no province/state)
jhu/csse/covid19/cases/*/update/Diamond Princess/*All case types from the Diamond Princess cruise ship

3. Create your application

After connecting and choosing the topics to listen on, it's time to build your application. Check out the examples below for applications that consume the streams.

Sample Applications to Get Started Quickly

ApplicationLanguageDescription
Spring Cloud Stream SampleJava + SpringThis microservice shows how to consume events using the Spring Cloud Stream project, which abstracts the messaging APIs from the developer and allows them to focus on implementing their business logic.
JavaScript SampleJavaScriptThis html page allows you to test out multiple topics and view the stream body response. You can check it out on theCovidStreamViewer
MQTT JS SampleJavaScriptThis html page demonstrates connecting to the Solace PubSub+ COVID19 Broker using MQTT JS Library. You can check it out on theMQTT CovidStreamViewer
NodeJS SampleNodeJSThis NodeJs application runs in the terminal using mqtt.js npm library
Python SamplePythonThis NodeJs executable runs in the terminal using the paho-mqtt python library

COVID19 Stream Processors

Below are Spring Boot microservices used to create the event current streams available for consumption. They are also good examples if you would like to consume the streams and republish them into an event broker. Note that they are not intended to be run with the provided credentials as we are running them as a services.

ApplicationVersionIntegrationDescription
COVID19RawPublisher0.1SpringThis application publihses RAW JHU data
COVID19CaseSplitter0.1SpringThis application consumes RAW JHU COVID19 data, and publishes smaller updates
COVID19RegionalSplitter0.1SpringThis application consumes RAW JHU COVID19 data, splits it into regional updates and publishes smaller regional updates
COVID19PopulationProvider0.1SpringThis application publishes the percentage of population affected in the areas
CovidTrackingStatesPublisher0.1SpringThis application consumes RAW CovidTracking.com data, and publishes smaller updates
CovidTrackingStatesSplitterPublisher0.1SpringThis application consumes RAW CovidTracking.com data, splits it into states and publishes smaller updates

EventPortal

Note: Contact us atcovid19-project@solace.com to add more event stream into the current broker.

Test Topics

This section includes information about test streams that are available. Since actual updates can occur infrequently, we are providing these test streams strictly for development purposes. Instead of only receiving events when updates occur, these topics, which includetest as a level, will regularly receive events whether or not an update has actually occurred. Once you have completed development you should remove thetest level from your topic and you’ll only receive events when actual updates occur.

DescriptionTopicNotes
Updated Active/Deaths/Confirmed/Recovered for a region/state in one eventjhu/csse/covid19/test/cases/region/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Deaths for a region/statejhu/csse/covid19/test/cases/deaths/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Active cases for a region/statejhu/csse/covid19/test/cases/active/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Confirmed cases for a region/statejhu/csse/covid19/test/cases/confirmed/update/{attributes.countryRegion}/{attributes.provinceState}
Updated Recovered cases for a region/statejhu/csse/covid19/test/cases/recovered/update/{attributes.countryRegion}/{attributes.provinceState}
Shows the updated percentage of the population affectedjhu/csse/covid19/test/cases/active/population/update/US/{attributes.provinceState}Only US for now
Updated US state informationcom/covidtracking/test/states/current/update/{state}

Note that theraw streams is regularly updated every ~45 seconds

Contribution

SeeContribution guidelines form more details

Credit

Disclaimer

  1. As per Johns Hopkins UniversityTerms of Use this data is public strictly for educational and academic research purposes.
  2. Data streams are provided and maintained bySolace under the same terms of use.

About

Stream Information & Example Applications for Processing JHU and CovidTracking.com COVID-19 data available as streams over Solace

Topics

Resources

License

Contributing

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors5

Languages


[8]ページ先頭

©2009-2025 Movatter.jp