- Notifications
You must be signed in to change notification settings - Fork0
Manipulation of graphs in NebulaGraph using the NetworkX API.
License
wey-gu/NebulaGraph-NX
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Manipulate and analyze NebulaGraph data using the NetworkX API
Documentation:https://github.com/wey-gu/nebulagraph-nx#documentation
Source Code:https://github.com/wey-gu/nebulagraph-nx
NebulaGraph NetworkX (ng_nx) is a powerful tool that bridges NebulaGraph and NetworkX, enabling you to leverage NetworkX's rich set of graph algorithms and analysis tools on data stored in NebulaGraph. This integration combines NebulaGraph's advanced storage capabilities with NetworkX's extensive graph analysis functionality.
- Seamless integration between NebulaGraph and NetworkX
- Multiple reader types for flexible data retrieval
- Easy-to-use writers for storing analysis results back to NebulaGraph
- Support for both vertex and edge data operations
- Compatibility with NetworkX's extensive library of graph algorithms
Ensure you have a NebulaGraph cluster running. For a quick setup, you can useNebulaGraph Lite to set up a cluster in Colab within 5 minutes.
Install ng_nx using pip:
pip install ng_nxWith GraphD Client
from ng_nx import NebulaReaderfrom ng_nx.utils import NebulaGraphConfigconfig = NebulaGraphConfig( space="basketballplayer", graphd_hosts="127.0.0.1:9669", metad_hosts="127.0.0.1:9559")reader = NebulaReader( edges=["follow", "serve"], properties=[["degree"], ["start_year", "end_year"]], nebula_config=config, limit=10000)g = reader.read()With Storage Client, this requires ng_nx being run within NebulaGraph Network or expose the metad and storaged to it, with same host and port being registered in NebulaGraph(
SHOW HOSTS META;andSHOW HOSTS;).
from ng_nx import NebulaScanReaderfrom ng_nx.utils import NebulaGraphConfig# Here, we need to be able to resolve the metad and storaged hosts, where they are the same with `SHOW HOSTS META;` and `SHOW HOSTS;`config = NebulaGraphConfig( space="demo_basketballplayer", graphd_hosts="graphd0:9669", metad_hosts="metad0:9559")reader = NebulaScanReader( edges=["follow", "serve"], properties=[["degree"], ["start_year", "end_year"]], nebula_config=config, limit=10000, with_rank=True)g = reader.read()We need to install louvain and graspologic first, to run louvain and leiden algorithm that are not included in NetworkX(not like Pagerank etc).
pip3 install python-louvain graspologicThen we can run the algorithms.
import networkx as nximport community as community_louvain# Run PageRank algorithmpr = nx.pagerank( g, alpha=0.85, max_iter=100, tol=1e-06, weight='degree')# Run Louvain community detectionug = g.to_undirected()louvain = community_louvain.best_partition(ug)# Run Leiden community detectionfrom graspologic.partition import hierarchical_leiden# Cast Multi-Graph to Homogeneous Graphg = nx.Graph(g)community_hierarchical_clusters = hierarchical_leiden(ug, max_cluster_size=10)Typical use cases are:
- Write the result of graph algorithm to NebulaGraph as vertex data.
- Write the result of graph algorithm to NebulaGraph as edge data.
We could create schema for pagerank and louvain like this:
CREATE TAG IF NOT EXISTS pagerank ( pagerank double NOT NULL);CREATE TAG IF NOT EXISTS louvain ( cluster_id int NOT NULL);Then we can run pagerank and louvain algorithm and write the result to NebulaGraph like this:
from ng_nx import NebulaWriterpr_writer = NebulaWriter(data=pr, nebula_config=config)# properties to writeproperties = ["pagerank"]pr_writer.set_options( label="pagerank", properties=properties, batch_size=256, write_mode="insert", sink="nebulagraph_vertex",)# write back to NebulaGraphpr_writer.write()# write louvain resultlouvain_writer = NebulaWriter(data=louvain, nebula_config=config)# properties to writeproperties = ["cluster_id"]louvain_writer.set_options( label="louvain", properties=properties, batch_size=256, write_mode="insert", sink="nebulagraph_vertex",)louvain_writer.write() # write back to NebulaGraphSay we have a graph with player and follow edge, we can write the result to NebulaGraph like this:
CREATE TAG IF NOT EXISTS player ( name string NOT NULL, age int NOT NULL);CREATE EDGE IF NOT EXISTS follow ( start_year int NOT NULL, end_year int NOT NULL);We can write the result to NebulaGraph like this:
from ng_nx import NebulaWriter# Example edge dataedge_data = [ ("player1", "player2", 0, [2022, 2023]), # src, dst, rank, [start_year, end_year] ("player2", "player3", 1, [2021, 2022]), # ... more edges ...]edge_writer = NebulaWriter(data=edge_data, nebula_config=config)# properties to write, map the properties to the edge dataproperties = ["start_year", "end_year"]edge_writer.set_options( label="follow", # Edge type name properties=properties, batch_size=256, write_mode="insert", sink="nebulagraph_edge",)# Write edges to NebulaGraphedge_writer.write()TheNebulaQueryReader allows you to execute any NebulaGraph query and construct a NetworkX graph from the result.
from ng_nx import NebulaQueryReaderfrom ng_nx.utils import NebulaGraphConfigconfig = NebulaGraphConfig( space="demo_basketballplayer", graphd_hosts="127.0.0.1:9669", metad_hosts="127.0.0.1:9559")reader = NebulaQueryReader(nebula_config=config)# Execute a custom queryquery = "MATCH p=(v:player{name:'Tim Duncan'})-[e:follow*1..3]->(v2) RETURN p"g = reader.read(query)This approach allows you to leverage the full power of NebulaGraph's query language while still being able to analyze the results using NetworkX.
TheNebulaQueryReader allows you to execute any NebulaGraph query and construct a NetworkX graph from the result.
from ng_nx import NebulaQueryReaderfrom ng_nx.utils import NebulaGraphConfigconfig = NebulaGraphConfig( space="demo_basketballplayer", graphd_hosts="127.0.0.1:9669", metad_hosts="127.0.0.1:9559")reader = NebulaQueryReader(nebula_config=config)# Execute a custom queryquery = "MATCH p=(v:player{name:'Tim Duncan'})-[e:follow*1..3]->(v2) RETURN p"g = reader.read(query)NG-NX provides three types of readers to fetch data from NebulaGraph:
NebulaReader(Load from Edge and Properties via MATCH): Reads a graph from NebulaGraph based on specified edges and properties, returning a NetworkX graph. It uses the MATCH clause internally to fetch data from NebulaGraph.NebulaQueryReader(Construct from Any Query): Executes a custom NebulaGraph query and constructs a NetworkX graph from the result. This reader is particularly useful when you need to perform complex queries or have specific data retrieval requirements.NebulaScanReader(Better for LARGE datasets): Will read graph data from NebulaGraph using a configuration similar toNebulaReader, but it will bypass the MATCH clause and utilize the SCAN interface with the Storage Client for potentially improved performance on large datasets. Note that this reader requires the NebulaGraph cluster to configure the metad and storaged accessible from the ng_nx.
Each reader is designed to cater to different use cases, providing flexibility in how you interact with and retrieve data from NebulaGraph for analysis with NetworkX.
Contributions are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request on theGitHub repository.
About
Manipulation of graphs in NebulaGraph using the NetworkX API.