Repository files navigation

Scalable tabular (SFrame,SArray) and graph (SGraph) data-structures built for out-of-core data analysis.

The SFrame package provides the complete implementation of:

• SFrame
• SArray
• SGraph
• The C++ SDK surface area (gl_sframe, gl_sarray, gl_sgraph)

The SFrame contains the open source componentsGraphLab Create fromTuri.

Some documentation to help get started:

• Getting started with SFrame
• SFrame user guide
• SGraph user guide

For more details on GraphLab Create (including documentation and tutorials) seehttp://turi.com.

Some of the key features of this package are.

• A scalable column compressed disk-backed dataframe optimized for machine learning and data science needs.
• Designed for bothtabular (SFrame, SArray) as well asgraph data (SGraph)
• Support forstrictly typed columns (int, float, str, datetime),weakly typed columns (schema free lists, dictionaries) as well asspecialized types such as Image.
• Uniform support formissing data.
• Query optimization and Lazy evaluation.
• A C++ API (gl_sarray, gl_sframe, gl_sgraph) with direct native access via the C++ SDK.
• A Python API (SArray, SFrame, SGraph) with an indirect access via an interprocess layer.

The SFrame Package is licensed under a BSD license. Seelicense file.

We release an updated version about once every 1.5 months or so. You can downloadthe most recent version directly frompypior using pip:

pip install -U sframe

Requirements

SFrame requires a 64-bit operating system.

Operating Systems

• Mac OS X: 10.8+
• Linux: Any distribution with GLIBC >= 2.11
• Ubuntu >= 11.04
• Debian >= 6
• RHEL >= 6
• SLES >= 11
• Windows (7, 8, 10, Server 2012 R2)

Python

• Python 2.7.x
• Pyhton 3.4.x
• Note: Unfortunately, Python 3.5.x is currently not supported. This is coming soon.

To simplify the development process, we use a pre-built dependency toolkit wecall dato-deps which prepackages all compile-time dependencies like boost, curl,etc, some with patches which we should contribute back. On Linux, we alsoprepackage the entire compiler toolchain. On configuration, dato-deps isdownloaded and unpacked into the deps/ directory

At least OS X 10.9. OS X 10.10 preferred.

• On OS X: Apple XCode 6 Command Line Tools [Required]

  • "clang --version" should report at least"Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn)"

• ccache [Optional]

  • Not required, but highly recommended

• cmake >= 3.2 [Optional]

  • Not required, but highly recommended

• *nix build tools: patch, make [Required]

  • Should come with most Mac/Linux systems by default. Recent Ubuntu versionswill require installing the build-essential package.

• zlib [Required]

  • Comes with most Mac/Linux systems by default. Recent Ubuntu version willrequire the zlib1g-dev package.

• ccache [Optional]

  • Not required, but highly recommended

• cmake >= 3.2 [Optional]

  • Not required, but highly recommended

This is somewhat more involving. See the wiki.

After you have done a git pull, cd into the SFrame directory and run the configure script:

cd SFrame./configure

Running configure will create two sub-directories,release anddebug. Selecteither of these modes/directories and navigate to theoss_src/unity subdirectory:

cd <repo root>/debug/oss_src/unity/python

or

cd <repo root>/release/oss_src/unity/python

Runningmake will build the project, according to the mode selected.

We recommend using make's parallel build feature to accelerate the compilationprocess. For instance:

make -j 4

Will perform up to 4 build tasks in parallel. The number of tasks to run inparallel should be roughly equal to the number of CPUs you have.

To avoid complicated interactions with python installed in your system, weinstall an entire miniconda toolchain into deps/.

In order to use the dev build that you have just compiled, some environmentvariables will need to be set. This can be done by sourcing a script. You'llneed to pass the script either "debug" or "release" depending on the type ofbuild you have compiled:

source <repo root>/oss_local_scripts/python_env.sh [debug | release ]

This will activate the miniconda toolchain, and you can run python directlyand import sframe, etc.

There is a script that makes it easy to run the python unit test. You will just need to call it and pass ityour build type (either "debug" or "release).

<repo root>/oss_local_scripts/run_python_test.sh [debug | release]

Alternatively, you can run nosetests directly. For instance, if I am usingthe debug build directory:

Activate the python environment

source <repo root>/oss_local_scripts/python_env.sh debug

Switch the build output directory and run nosetests

cd <repo root>/debug/src/unity/pythonnosetests -s -v sframe/test

There are also C++ tests. To compile and run them, do the following:

cd <repo root>/[debug | release]/oss_testmakectest

To summarize the above, to just build and run SFrame in the debug build directory:

./configurecd debug/oss_src/unitymake -j4# where root is the checked out SFrame directorysource <root>/oss_local_scripts/python_env.sh debug# import sframe should work herepython

To build an egg for your platform, run

<repo root>/oss_local_scripts/make_egg.sh

(See --help for options)

See:https://github.com/turi-code/GraphLab-Create-SDK