- Notifications
You must be signed in to change notification settings - Fork5
dllib is a distributed deep learning library running on Apache Spark
License
NotificationsYou must be signed in to change notification settings
Lewuathe/dllib
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
dllib is a distributed deep learning framework running on Apache Spark. See more detail indocumentation. dllib is designed to be simple and easy to use for Spark users.
Since dllib has completely same interface of MLlib algorithms, libraries in MLlib can be used for feature engineering or transformation.
dllib is uploaded onSpark Packages. You can use fromsperk-shell directly.
$ ./bin/spark-shell --packages Lewuathe:dllib:0.0.9
If you want use jar package to extend, you can write the configuration in your pom.xml.
<dependency> <groupId>com.lewuathe</groupId> <artifactId>dllib_2.11</artifactId> <version>0.0.9</version></dependency>This is an example for classification ofMNIST dataset. Full code can be seenhere.
importcom.lewuathe.dllib.graph.Graphimportcom.lewuathe.dllib.layer.{AffineLayer,ReLULayer,SoftmaxLayer}importcom.lewuathe.dllib.network.Network// Define the network structure as calculation graph.valgraph=newGraph(Array(newAffineLayer(100,784),newReLULayer(100,100),newAffineLayer(10,100),newSoftmaxLayer(10,10)))// Model keeps whole network parameters which should be trained.// Default is in-memory model.valmodel=Model(nn3Graph)valnn3=Network(model, graph)// MultilayerPerceptron defines the optimization algorithms and hyper parameters.valmultilayerPerceptron=newMultiLayerPerceptron("MNIST", nn3)// We can pass Dataset of Spark to the network.valtrainedModel= multilayerPerceptron.fit(df)valresult= trainedModel.transform(df)result.filter("label = prediction").count()
- Kai Sasaki(@Lewuathe)
About
dllib is a distributed deep learning library running on Apache Spark
Topics
Resources
License
Stars
Watchers
Forks
Packages0
No packages published