- Notifications
You must be signed in to change notification settings - Fork0
Manipulation of graphs in NebulaGraph using the NetworkX API.
License
wey-gu/NebulaGraph-NX
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Manipulate and analyze NebulaGraph data using the NetworkX API
Documentation:https://github.com/wey-gu/nebulagraph-nx#documentation
Source Code:https://github.com/wey-gu/nebulagraph-nx
NebulaGraph NetworkX (ng_nx) is a powerful tool that bridges NebulaGraph and NetworkX, enabling you to leverage NetworkX's rich set of graph algorithms and analysis tools on data stored in NebulaGraph. This integration combines NebulaGraph's advanced storage capabilities with NetworkX's extensive graph analysis functionality.
- Seamless integration between NebulaGraph and NetworkX
- Multiple reader types for flexible data retrieval
- Easy-to-use writers for storing analysis results back to NebulaGraph
- Support for both vertex and edge data operations
- Compatibility with NetworkX's extensive library of graph algorithms
Ensure you have a NebulaGraph cluster running. For a quick setup, you can useNebulaGraph Lite to set up a cluster in Colab within 5 minutes.
Install ng_nx using pip:
pip install ng_nx
With GraphD Client
fromng_nximportNebulaReaderfromng_nx.utilsimportNebulaGraphConfigconfig=NebulaGraphConfig(space="basketballplayer",graphd_hosts="127.0.0.1:9669",metad_hosts="127.0.0.1:9559")reader=NebulaReader(edges=["follow","serve"],properties=[["degree"], ["start_year","end_year"]],nebula_config=config,limit=10000)g=reader.read()
With Storage Client, this requires ng_nx being run within NebulaGraph Network or expose the metad and storaged to it, with same host and port being registered in NebulaGraph(
SHOW HOSTS META;andSHOW HOSTS;).
fromng_nximportNebulaScanReaderfromng_nx.utilsimportNebulaGraphConfig# Here, we need to be able to resolve the metad and storaged hosts, where they are the same with `SHOW HOSTS META;` and `SHOW HOSTS;`config=NebulaGraphConfig(space="demo_basketballplayer",graphd_hosts="graphd0:9669",metad_hosts="metad0:9559")reader=NebulaScanReader(edges=["follow","serve"],properties=[["degree"], ["start_year","end_year"]],nebula_config=config,limit=10000,with_rank=True)g=reader.read()
We need to install louvain and graspologic first, to run louvain and leiden algorithm that are not included in NetworkX(not like Pagerank etc).
pip3 install python-louvain graspologic
Then we can run the algorithms.
importnetworkxasnximportcommunityascommunity_louvain# Run PageRank algorithmpr=nx.pagerank(g,alpha=0.85,max_iter=100,tol=1e-06,weight='degree')# Run Louvain community detectionug=g.to_undirected()louvain=community_louvain.best_partition(ug)# Run Leiden community detectionfromgraspologic.partitionimporthierarchical_leiden# Cast Multi-Graph to Homogeneous Graphg=nx.Graph(g)community_hierarchical_clusters=hierarchical_leiden(ug,max_cluster_size=10)
Typical use cases are:
- Write the result of graph algorithm to NebulaGraph as vertex data.
- Write the result of graph algorithm to NebulaGraph as edge data.
We could create schema for pagerank and louvain like this:
CREATE TAG IF NOT EXISTS pagerank ( pagerank doubleNOT NULL);CREATE TAG IF NOT EXISTS louvain ( cluster_idintNOT NULL);
Then we can run pagerank and louvain algorithm and write the result to NebulaGraph like this:
fromng_nximportNebulaWriterpr_writer=NebulaWriter(data=pr,nebula_config=config)# properties to writeproperties= ["pagerank"]pr_writer.set_options(label="pagerank",properties=properties,batch_size=256,write_mode="insert",sink="nebulagraph_vertex",)# write back to NebulaGraphpr_writer.write()# write louvain resultlouvain_writer=NebulaWriter(data=louvain,nebula_config=config)# properties to writeproperties= ["cluster_id"]louvain_writer.set_options(label="louvain",properties=properties,batch_size=256,write_mode="insert",sink="nebulagraph_vertex",)louvain_writer.write()# write back to NebulaGraph
Say we have a graph with player and follow edge, we can write the result to NebulaGraph like this:
CREATE TAG IF NOT EXISTS player ( name stringNOT NULL, ageintNOT NULL);CREATE EDGE IF NOT EXISTS follow ( start_yearintNOT NULL, end_yearintNOT NULL);
We can write the result to NebulaGraph like this:
fromng_nximportNebulaWriter# Example edge dataedge_data= [ ("player1","player2",0, [2022,2023]),# src, dst, rank, [start_year, end_year] ("player2","player3",1, [2021,2022]),# ... more edges ...]edge_writer=NebulaWriter(data=edge_data,nebula_config=config)# properties to write, map the properties to the edge dataproperties= ["start_year","end_year"]edge_writer.set_options(label="follow",# Edge type nameproperties=properties,batch_size=256,write_mode="insert",sink="nebulagraph_edge",)# Write edges to NebulaGraphedge_writer.write()
TheNebulaQueryReader allows you to execute any NebulaGraph query and construct a NetworkX graph from the result.
fromng_nximportNebulaQueryReaderfromng_nx.utilsimportNebulaGraphConfigconfig=NebulaGraphConfig(space="demo_basketballplayer",graphd_hosts="127.0.0.1:9669",metad_hosts="127.0.0.1:9559")reader=NebulaQueryReader(nebula_config=config)# Execute a custom queryquery="MATCH p=(v:player{name:'Tim Duncan'})-[e:follow*1..3]->(v2) RETURN p"g=reader.read(query)
This approach allows you to leverage the full power of NebulaGraph's query language while still being able to analyze the results using NetworkX.
TheNebulaQueryReader allows you to execute any NebulaGraph query and construct a NetworkX graph from the result.
fromng_nximportNebulaQueryReaderfromng_nx.utilsimportNebulaGraphConfigconfig=NebulaGraphConfig(space="demo_basketballplayer",graphd_hosts="127.0.0.1:9669",metad_hosts="127.0.0.1:9559")reader=NebulaQueryReader(nebula_config=config)# Execute a custom queryquery="MATCH p=(v:player{name:'Tim Duncan'})-[e:follow*1..3]->(v2) RETURN p"g=reader.read(query)
NG-NX provides three types of readers to fetch data from NebulaGraph:
NebulaReader(Load from Edge and Properties via MATCH): Reads a graph from NebulaGraph based on specified edges and properties, returning a NetworkX graph. It uses the MATCH clause internally to fetch data from NebulaGraph.NebulaQueryReader(Construct from Any Query): Executes a custom NebulaGraph query and constructs a NetworkX graph from the result. This reader is particularly useful when you need to perform complex queries or have specific data retrieval requirements.NebulaScanReader(Better for LARGE datasets): Will read graph data from NebulaGraph using a configuration similar toNebulaReader, but it will bypass the MATCH clause and utilize the SCAN interface with the Storage Client for potentially improved performance on large datasets. Note that this reader requires the NebulaGraph cluster to configure the metad and storaged accessible from the ng_nx.
Each reader is designed to cater to different use cases, providing flexibility in how you interact with and retrieve data from NebulaGraph for analysis with NetworkX.
Contributions are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request on theGitHub repository.
About
Manipulation of graphs in NebulaGraph using the NetworkX API.
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Uh oh!
There was an error while loading.Please reload this page.