- Notifications
You must be signed in to change notification settings - Fork73
No-nonsense Elasticsearch library for Elixir
License
danielberkompas/elasticsearch-elixir
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
A simple, no-nonsense Elasticsearch library for Elixir. Highlights include:
- No DSLs. Interact directly with the
ElasticsearchJSON API. - Zero-downtime index (re)building. Via
Mix.Tasks.Elasticsearch.Buildtask. - Dev Tools. Helpers for running Elasticsearch as part of your supervisiontree during development.
Addelasticsearch to your list of dependencies inmix.exs:
defdepsdo[{:elasticsearch,"~> 1.0.0"}]end
Then, create anElasticsearch.Cluster in your application:
defmoduleMyApp.ElasticsearchClusterdouseElasticsearch.Cluster,otp_app::my_append
Once you have created your cluster, add it to your application's supervision tree:
children=[MyApp.ElasticsearchCluster]
Finally, you can issue requests to Elasticsearch using it.
Elasticsearch.get(MyApp.ElasticsearchCluster,"/_cat/health")
See the annotated example configuration below.
config:my_app,MyApp.ElasticsearchCluster,# The URL where Elasticsearch is hosted on your systemurl:"http://localhost:9200",# If your Elasticsearch cluster uses HTTP basic authentication,# specify the username and password here:username:"username",password:"password",# If you want to mock the responses of the Elasticsearch JSON API# for testing or other purposes, you can inject a different module# here. It must implement the Elasticsearch.API behaviour.api:Elasticsearch.API.HTTP,# Customize the library used for JSON encoding/decoding.json_library:Poison,# or Jason# You should configure each index which you maintain in Elasticsearch here.# This configuration will be read by the `mix elasticsearch.build` task,# described below.indexes:%{# This is the base name of the Elasticsearch index. Each index will be# built with a timestamp included in the name, like "posts-5902341238".# It will then be aliased to "posts" for easy querying.posts:%{# This file describes the mappings and settings for your index. It will# be posted as-is to Elasticsearch when you create your index, and# therefore allows all the settings you could post directly.settings:"priv/elasticsearch/posts.json",# This store module must implement a store behaviour. It will be used to# fetch data for each source in each indexes' `sources` list, below:store:MyApp.ElasticsearchStore,# This is the list of data sources that should be used to populate this# index. The `:store` module above will be passed each one of these# sources for fetching.## Each piece of data that is returned by the store must implement the# Elasticsearch.Document protocol.sources:[MyApp.Post],# When indexing data using the `mix elasticsearch.build` task,# control the data ingestion rate by raising or lowering the number# of items to send in each bulk request.bulk_page_size:5000,# Likewise, wait a given period between posting pages to give# Elasticsearch time to catch up.bulk_wait_interval:15_000,# 15 seconds# By default bulk indexing uses the "create" action. To allow existing# documents to be replaced, use the "index" action instead.bulk_action:"create"}}
config:my_app,MyApp.ElasticsearchCluster,default_options:[timeout:5_000,recv_timeout:5_000,hackney:[pool::pool_name]]
Your app must provide aStore module, which will fetch data to upload toElasticsearch. This module must implement theElasticsearch.Storebehaviour.
The example below usesEcto, but you can implement the behaviour on topof any persistence layer.
defmoduleMyApp.ElasticsearchStoredo@behaviourElasticsearch.StoreimportEcto.QueryaliasMyApp.Repo@impltruedefstream(schema)doRepo.stream(schema)end@impltruedeftransaction(fun)do{:ok,result}=Repo.transaction(fun,timeout::infinity)resultendend
Each result returned by your store must implement theElasticsearch.Documentprotocol.
defimplElasticsearch.Document,for:MyApp.Postdodefid(post),do:post.iddefrouting(_),do:falsedefencode(post)do%{title:post.title,author:post.author}endend
You can plug in a different module to make API requests, as long as itimplements theElasticsearch.API behaviour.
This can be used in test mode, for example:
# config/test.exsconfig:my_app,MyApp.ElasticsearchCluster,api:MyApp.ElasticsearchMock
Your mock can then stub requests and responses from Elasticsearch.
defmoduleMyApp.ElasticsearchMockdo@behaviourElasticsearch.API@impltruedefrequest(_config,:get,"/posts/1",_data,_opts)do{:ok,%HTTPoison.Response{status_code:404,body:%{"status"=>"not_found"}}}endend
As AWS does not provide credentials' based http authentication, you can use theElasticsearch.API.AWS module if you want to use AWS Elasticsearch Service with AWS Signature V4 signed HTTP connections.
To use this, just addsigaws to your dependencies and add this to your configuration:
# Add to depsdefdepsdo[# ...{:sigaws,">= 0.0.0"}]end# config/prod.exsconfig:my_app,MyApp.ElasticsearchCluster,api:Elasticsearch.API.AWS,default_options:[aws:[region:"us-east-1",service:"es",access_key:"aws_access_key_id",secret:"aws_secret_access_key"]]
Use themix elasticsearch.build task to build indexes using a zero-downtime,hot-swap technique with Elasticsearch aliases.
# This will read the `indexes[posts]` configuration seen above, to build# an index, `posts-123123123`, which will then be aliased to `posts`.$ mix elasticsearch.build posts --cluster MyApp.ElasticsearchCluster
See the docs onMix.Tasks.Elasticsearch.Build andElasticsearch.Indexfor more details.
UseElasticsearch.put_document/3 to upload a document to a particular index.
# MyApp.Post must implement Elasticsearch.DocumentElasticsearch.put_document(MyApp.ElasticsearchCluster,%MyApp.Post{},"index-name")
To remove documents, useElasticsearch.delete_document/3:
Elasticsearch.delete_document(MyApp.ElasticsearchCluster,%MyApp.Post{},"index-name")
You can query Elasticsearch thepost/3 function:
# Raw queryElasticsearch.post(MyApp.ElasticsearchCluster,"/posts/_doc/_search",'{"query": {"match_all": {}}}')# Using a mapElasticsearch.post(MyApp.ElasticsearchCluster,"/posts/_doc/_search",%{"query"=>%{"match_all"=>%{}}})
See the official Elasticsearchdocumentationfor how to write queries.
This package provides two utilities for developing with Elasticsearch:
mix elasticsearch.install: A mix task to install Elasticsearch and Kibanato a folder of your choosing.Elasticsearch.Executable. Use this to start and stop Elasticsearch as partof your supervision tree.children=[worker(Elasticsearch.Executable,["Elasticsearch","./vendor/elasticsearch/bin/elasticsearch",# assuming elasticsearch is in your vendor/ dir9200],id::elasticsearch),worker(Elasticsearch.Executable,["Kibana","./vendor/kibana/bin/kibana",# assuming kibana is in your vendor/ dir5601],id::kibana)]
As of version0.3.0 of this client library, multiple document types are notsupported, because support for these was removed in Elasticsearch 6.x. Youcan still use this library with Elasticsearch 5.x, but you must design yourindexes in the Elasticsearch 6.x style.
Read more about this in Elasticsearch's guide,"Removal of MappingTypes".
Runmix docs to generate local documentation.
To contribute code to this project, you'll need to:
- Fork the repo
- Clone your fork
- Run
bin/setup - Create a branch
- Commit your changes
- Open a PR
About
No-nonsense Elasticsearch library for Elixir
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Uh oh!
There was an error while loading.Please reload this page.