Available data sources and tools

Anumber of organizations use PXWEB to distribute hierarchical data.You can browse the available data sets at:

• StatisticsSweden withAPIDescription
• StatisticsFinlandStatFi APIDescription
• Otherorganizations using PX-WEB

About PXWEB APIs

The data in PXWEB APIs consists of metadata and data parts. Metadatais structured in a hierarchical node tree, where each node containsinformation about subnodes. The leaf nodes have information on which thedimensions are available for the data at that leaf node.

install.packages("pxweb")

library("remotes")remotes::install_github("ropengov/pxweb")

library(pxweb)

vignette(topic="pxweb")

Installation issues

We also recommend setting the UTF-8 encoding since each API may havelocal specific letters:

Sys.setlocale(locale ="UTF-8")

# Navigate through all pxweb api:s in the R package API catalogued<-pxweb_interactive()# Get data from SCB (Statistics Sweden)d<-pxweb_interactive("api.scb.se")# Fetching data from statfi (Statistics Finland)d<-pxweb_interactive("pxnet2.stat.fi")# Fetching data from StatBank (Statistics Norway)d<-pxweb_interactive("data.ssb.no")# To see all available PXWEB APIs usepxweb_apis<-pxweb_api_catalogue()

# Start with a specific path.d<-pxweb_interactive("https://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A")

file.edit(pxweb_api_catalogue_path())

# Get PXWEB levelspx_levels<-pxweb_get("https://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/")px_levels

# Get PXWEB metadata about a tablepx_meta<-pxweb_get("https://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/BefolkningNy")px_meta

Creating data queries

To download data, we need both the URL to the table and a queryspecifying what parts of the table are of interest. An URL to a table isan URL that will return a metadata object if not a query is supplied.Creating a query can be done in three main ways. The first and moststraightforward approach is to usepxweb_interactive() toexplore the table URL and create a query interactively.

d<-pxweb_interactive("https://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/BefolkningNy")

The interactive function will return the query and the URL, even ifthe data is not downloaded.

d$url

## [1] "http://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/BefolkningNy"

d$query

## PXWEB QUERY## query:##  [[1]] Region (item):##    00##  [[2]] Civilstand (item):##    OG, G, ÄNKL, SK##  [[3]] Alder (item):##    tot##  [[4]] ContentsCode (item):##    BE0101N1##  [[5]] Tid (item):##    2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017

We can also turn the query into a JSON query that we can use outsideR.

pxweb_query_as_json(d$query,pretty =TRUE)

## {##   "query": [##     {##       "code": "Region",##       "selection": {##         "filter": "item",##         "values": ["00"]##       }##     },##     {##       "code": "Civilstand",##       "selection": {##         "filter": "item",##         "values": ["OG", "G", "ÄNKL", "SK"]##       }##     },##     {##       "code": "Alder",##       "selection": {##         "filter": "item",##         "values": ["tot"]##       }##     },##     {##       "code": "ContentsCode",##       "selection": {##         "filter": "item",##         "values": ["BE0101N1"]##       }##     },##     {##       "code": "Tid",##       "selection": {##         "filter": "item",##         "values": ["2010", "2011", "2012", "2013", "2014", "2015", "2016", "2017"]##       }##     }##   ],##   "response": {##     "format": "json"##   }## }

The second approach is to specify the query either as an R list or aJSON object. Some Statistical Agencies, such as Statistics Sweden,supply queries directly as a JSON object on their web pages. We can usethese queries directly. Below is another example of a JSON query for thetable above. For details on setting up a JSON query, see the PXWEB APIdocumentation.

{  "query": [    {      "code": "Civilstand",      "selection": {        "filter": "item",        "values": ["OG", "G", "ÄNKL", "SK"]      }    },    {      "code": "Kon",      "selection": {        "filter": "item",        "values": ["1", "2"]      }    },    {      "code": "ContentsCode",      "selection": {        "filter": "item",        "values": ["BE0101N1"]      }    },    {      "code": "Tid",      "selection": {        "filter": "item",        "values": ["2015", "2016", "2017"]      }    }  ],  "response": {    "format": "json"  }}

To use this JSON query, we store the JSON query as a file and supplythe path to the file to the “pxweb_query()”function.

pxq<-pxweb_query("path/to/the/json/query.json")

Finally, we can create a PXWEB query from an R list where each listelement is a variable and selected observation.

pxweb_query_list<-list("Civilstand"=c("*"),# Use "*" to select all"Kon"=c("1","2"),"ContentsCode"=c("BE0101N1"),"Tid"=c("2015","2016","2017")  )pxq<-pxweb_query(pxweb_query_list)pxq

## PXWEB QUERY## query:##  [[1]] Civilstand (all):##    *##  [[2]] Kon (item):##    1, 2##  [[3]] ContentsCode (item):##    BE0101N1##  [[4]] Tid (item):##    2015, 2016, 2017

We can validate the query against the metadata object to asses thatwe can use the query. This validation is done automatically when thedata is fetched withpxweb_get() but can also be donemanually.

pxweb_validate_query_with_metadata(pxq, px_meta)

Downloading data

When we have the URL to a data table and a query, we can download thedata with “pxweb_get()”. The function returns apxweb_data object that contains the downloaded data.

pxd<-pxweb_get("https://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/BefolkningNy",  pxq)pxd

## PXWEB DATA## With 4 variables and 24 observations.

If we instead want a JSON-stat object, we change the response formatto JSON-stat, and we will get a JSON-stat object returned.

pxq$response$format<-"json-stat"pxjstat<-pxweb_get("https://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/BefolkningNy",  pxq)pxjstat

## {##   "dataset": {##     "dimension": {##       "Civilstand": {##         "label": ["marital status"],##         "category": {##           "index": {##             "OG": [0],##             "G": [1],##             "ÄNKL": [2],##             "SK": [3]##           },##           "label": {##             "OG": ["single"],##             "G": ["married"],##             "ÄNKL": ["widowers/widows"],##             "SK": ["divorced"]##           }##         },##         "extension": {##           "show": ["value"]##         }##       },##       "Kon": {##         "label": ["sex"],##         "category": {##           "index": {##             "1": [0],##             "2": [1]##           },##           "label": {##             "1": ["men"],##             "2": ["women"]##           }##         },##         "link": {##           "describedby": [##             {##               "extension": {##                 "Kon": ["Kön"]##               }##             }##           ]##         },##         "extension": {##           "show": ["value"]##         }##       },##       "ContentsCode": {##         "label": ["observations"],##         "category": {##           "index": {##             "BE0101N1": [0]##           },##           "label": {##             "BE0101N1": ["Population"]##           },##           "unit": {##             "BE0101N1": {##               "base": ["number"],##               "decimals": [0]##             }##           }##         },##         "extension": {##           "show": ["value"]##         }##       },##       "Tid": {##         "label": ["year"],##         "category": {##           "index": {##             "2015": [0],##             "2016": [1],##             "2017": [2]##           },##           "label": {##             "2015": ["2015"],##             "2016": ["2016"],##             "2017": ["2017"]##           }##         },##         "extension": {##           "show": ["code"]##         }##       },##       "id": [##         ["Civilstand"],##         ["Kon"],##         ["ContentsCode"],##         ["Tid"]##       ],##       "size": [##         [4],##         [2],##         [1],##         [3]##       ],##       "role": {##         "metric": [##           ["ContentsCode"]##         ],##         "time": [##           ["Tid"]##         ]##       }##     },##     "label": ["Population by marital status, sex, observations and year"],##     "source": ["Statistics Sweden"],##     "updated": ["2023-02-09T07:57:00Z"],##     "value": [##       [2762601],##       [2820248],##       [2870477],##       [2394842],##       [2437315],##       [2477012],##       [1651482],##       [1672460],##       [1687016],##       [1639519],##       [1657129],##       [1671381],##       [99751],##       [99654],##       [99682],##       [345008],##       [340709],##       [335961],##       [417132],##       [420985],##       [425487],##       [540682],##       [546653],##       [553226]##     ],##     "extension": {##       "px": {##         "infofile": ["BE0101"],##         "tableid": ["TAB638"],##         "decimals": [0]##       }##     }##   }## }

Some return formats return files. Then, these responses are stored inthe Rtempdir() folded, and the file paths are returned bypxweb_get(). Currently,px andsdmx formats can be downloaded as files, but file an issueif you need other response formats.

pxq$response$format<-"px"pxfp<-pxweb_get("https://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/BefolkningNy",  pxq)pxfp

## [1] "/var/folders/x9/dsgck_4s5mx2nrzzs8zd64rc0000gq/T//RtmpFdmiD7/50026bd2b2d8df2e3f190ca568b3b587d8207465.px"

If the queries are large (contain more values than the PXWEB APImaximum allowed values), the query is chunked into optimal chunks and isthen downloaded sequentially. PXWEB data objects are then combined intoone large PXWEB data object, while JSON-stat objects are returned as alist of JSON-stat objects, and other files are stored intempdir() as separate files.

For more advanced connections to the API, thepxweb_advanced_get() gives the flexibility to access theunderlying HTTP calls usinghttr and log the HTTP calls fordebugging.

We can then convert the downloaded PXWEB data objects to adata. frame or to a character matrix. The character matrixcontains the “raw” data whiledata. frame returns an Rdata.frame in a tidy format. This conversion means missingvalues (such as “..” are converted toNA) in adata. frame. Using the argumentsvariable.value.type andcolumn.name.type, wecan choose if we want the code or the text column names and valuetypes.

pxdf<-as.data.frame(pxd,column.name.type ="text",variable.value.type ="text")head(pxdf)

##   marital status   sex year Population## 1         single   men 2015    2762601## 2         single   men 2016    2820248## 3         single   men 2017    2870477## 4         single women 2015    2394842## 5         single women 2016    2437315## 6         single women 2017    2477012

pxdf<-as.data.frame(pxd,column.name.type ="code",variable.value.type ="code")head(pxdf)

##   Civilstand Kon  Tid BE0101N1## 1         OG   1 2015  2762601## 2         OG   1 2016  2820248## 3         OG   1 2017  2870477## 4         OG   2 2015  2394842## 5         OG   2 2016  2437315## 6         OG   2 2017  2477012

Similarly, we can access the raw data as a character matrix withas.matrix.

pxmat<-as.matrix(pxd,column.name.type ="code",variable.value.type ="code")head(pxmat)

##      Civilstand Kon Tid    BE0101N1 ## [1,] "OG"       "1" "2015" "2762601"## [2,] "OG"       "1" "2016" "2820248"## [3,] "OG"       "1" "2017" "2870477"## [4,] "OG"       "2" "2015" "2394842"## [5,] "OG"       "2" "2016" "2437315"## [6,] "OG"       "2" "2017" "2477012"

Access data footnotes/comments

In addition to the data, the PXWEB DATA object may also containcomments for the data. This can be accessed usingpxweb_data_comments() function.

pxdc<-pxweb_data_comments(pxd)pxdc

## NO PXWEB DATA COMMENTS

In this case, we did not have any comments. If we have comments, wecan turn the comments into adata. frame with one commentper row.

as.data.frame(pxdc)

pxweb_cite(pxd)

sessionInfo()