Repository files navigation

DiVE is an interactive 3D web viewer of up to million points on one screen that represent data. It is meant to provide interaction for viewing high-dimensional data that has been previouslyembedded in 3D. For embedding (non-linear dimensionality reduction, or manifold learning) we recommendLargeVis (a new algorithm by Microsoft Research, ) ortSNE .

For an online demo clickhere. You can also use this site to upload your datasets complying to the data format described below.

New: a pipeline for Linux consisting of LargeVis and DiVE has been released here:https://github.com/sonjageorgievska/Embed-Dive .

The simplest way is to download the code and openindex.html with your browser. Try it by uploading datasets from thedata folder. The application can work completely offline.

To use it with a local http server:

1. Open your command line interpreter (CLI)
2. Clone this repository
3. Go to the main folder ofDiVE in your CLI (whereindex.html is)
4. Installnode.js server together with the node package managernpm from (https://www.npmjs.com/get-npm).
5. Typenpm install connect serve-static
6. Typenode server.js
7. Open your browser and typehttp://localhost:8082/index.html

1. Clone this repository
2. Installnode,npm, andgrunt-cli
3. Runnpm install to install all the build requirements
4. Rungrunt to build. The resulting compiled JavaScript will be indist/ and the docs will be indoc/

• Every point has 3 coordinates and a unique ID. (For a best view, the absolute values of the coordinates should be smaller than 1. When using LargeVis with similarities (weights) as input, this can be achieved by re-scaling the similarities to be smaller than 1.)

• A point also hasProperties:

  • Properties is a list of strings which can be empty. Each string which is a number represents the value of a respective numerical property. Each string which is not a number represents the value of a respective categorical property. These values are used in the Coloring section of the UI of the web-page. When the user selects a property, if the property has categorical (non-numerical) values, each point is colored in a color representing the value of the categorical property. If the property is numerical, then after the user has selected a color, every point is colored with a shade of the selected color. The intensity of the color corresponds to the intensity of the selected property for the particular point.

• A node can also have an image associated to it, see the Data format section for more info.

• A user can search for all points that contain a certain substring in their ids, names or properties, by using theSearch section. Then all points that are a match become red, and the rest become grey. One can search also for boolean expressions of regular expressions. An example of a boolean expression isxx AND yy OR NOT zz, where xx, yy, and zz are regular expressions and NOT binds more than AND, which binds more than OR. In this case all points that contain in their metadata the regular espressions xx and yy, or that do not contain zz, will be coloured in red.

• Show only found nodes will show only the nodes that result from the search.

• TheResume colors button at the bottom returns the colors of the points to the previous coloring scheme.

• Centralize : will move data back to center of the screen, zoomed-in
• See all data : will zoom-out such that all data is visible
• Scase point size: very useful when the user has zoomed-in enough. When this option is not selected, the points do not get bigger as the camera moves closer to them, so that they can be separated and inspected individually.
• Show point info in popup : when selected, the information about a point when hovering over it will be displayed in a pop-up message rather than at the top left corner of the screen

As explained in sectionData description and functionality .

• The data is in a JSON (JavaScript Object Notation) format. (See folderdata for examples.)To obtaindata.js, first a data structure

    Dictionary<string, Point>

is created in any programming language, where the keys are the id’s of the points andPoint is an object of the class

public class Point    {        public List<double> Coordinates;        public List<double> Properties;    }

Coordinates andProperties are as discussed in the previous section.

Next, the dictionary is serialized using JavaScriptSerializer and written indata.json (name is flexible).Here is an example of an entry of the serialized dictionary in adata.json file:

"3951": {"Coordinates": [0.99860800383893167,0.61276015046241838,0.450976426942296],"Properties": ["0","1","5","12688892","0.998","5","True","0","False","5","1","True","1","518","0","-1","Rhodotorula","","Sporidiobolales","Microbotryomycetes"]}

Optionally, if data has properties, the dictionary should also contain an entry

"NamesOfProperties":["name1","name2",  ,"name_n"]

Optionally, if images are associated to the nodes, the node image can be displayed in a pop-up when hovering over the node.If the datafile starts withnamedataset_ then the folder with images should beimages_namedataset in folderdata.(see examples in folderdata, sorry for their size). The name of an image should benodeId.jpg.

If your images have a.png extension then thefingerprints_namedataset folder is an option, although it is currently made for the Sherlock purposes.

The output ofLargeVis is a text file - every line has the id of the point, and 3 coordinates (real numbers). Only the first line is an exception: it contains the number of points and the dimension. Here is an example:

4271 30 -33.729916 17.692684 17.4667491 -32.923210 17.249269 18.111458

It can be processed into an input of the viewer by using the python script "MakeVizDataWithProperMetaData.py" in the folder "scripts_prepareData". It is called with

python MakeVizDataWithProperMetaData.py -coord coordinatesFile -metadata metaDataFile -dir baseDir -np -namesOfPropertiesFile

• coordinatesFile: the output file of LargeVis

• metaData: file containing meta information about data. Format:[id] [metadata]. Format of metadata:"first_line" "second_line" "third_line" (number of lines is not limited). Example line ofmetadata:35 "A dog" "Age:2" "Color brown".

• baseDir: base directory to store output file

• namesOfPropertiesFile: A json file containing list of properties names. Ex:["Height", "Weight", "Place of birth"]. If file is omitted, its name should be"No"

The software is released under the GPL2 licence.Contact the author if you would like a version with an Apache licence