Stratio/deep-spark

*Disclaimer: As of 01/06/2015 this project has been deprecated. Thank you for your understanding and continued help throughout the project's life.Deep is a thin integration layer between Apache Spark and several NoSQL datastores.We actually support Apache Cassandra, MongoDB, Elastic Search, Aerospike, HDFS, S3 and any database accessible through JDBC, but in the near future we will add support for sever other datastores.JIRA: https://deep-spark.atlassian.netIn order to compile the deep-jdbc module is necessary to add the Oracle ojdbc driver into your local repository. You can download it from the URL: http://www.oracle.com/technetwork/database/features/jdbc/default-2280470.html. When you are on the web you must click in "Accept License Agreement" and later downlad ojdbc7.jar library. You need a free oracle account to download the official driver.To install the ojdbc driver in your local repository you must execute the command below:mvn install:install-file -Dfile= -DgroupId=com.oracle -DartifactId=ojdbc7 -Dversion=12.1.0.2 -Dpackaging=jarAfter that you can compile Deep executing the following steps:cd deep-parentmvn clean installIf you want to create a Deep distribution you must execute the following steps:cd deep-scriptsmake-distribution-deep.shDuring the creation you'll see the following question:What tag want to use for Aerospike native repository?You must type 0.7.0 and press enter.The integration is not based on the Cassandra's Hadoop interface.Deep comes with an user friendly API that lets developers create Spark RDDs mapped to Cassandra column families.We provide two different interfaces:The first one will let developers map Cassandra tables to plain old java objects (POJOs), just like if you were using any other ORM. We call this API the 'entity objects' API.This abstraction is quite handy, it will let you work on RDD (under the hood Deep will transparently map Cassandra's columns to entity properties).Your domain entities must be correctly annotated using Deep annotations (take a look at deep-core example entities in package com.stratio.deep.core.entity).The second one is a more generic 'cell' API, that will let developerss work on RDD<com.stratio.deep.entity.Cells> where a 'Cells' object is a collection of com.stratio.deep.entity.Cell objects.Column metadata is automatically fetched from the data store. This interface is a little bit more cumbersome to work with (see the example below),but has the advantage that it doesn't require the definition of additional entity classes.Example: you have a table called 'users' and you decide to use the 'Cells' interface. Once you get an instance 'c' of the Cells object,to get the value of column 'address' you can issue a c.getCellByName("address").getCellValue().Please, refer to the Deep API documentation to know more about the Cells and Cell objects.We encourage you to read the more comprehensive documentation hosted on the Openstratio website.Deep comes with an example sub project called 'deep-examples' containing a set of working examples, both in Java and Scala.Please, refer to the deep-example project README for further information on how to setup a working environment.Spark-MongoDB connector is based on Hadoop-mongoDB.Support for MongoDB has been added in version 0.3.0.We provide two different interfaces:ORM API, you just have to annotate your POJOs with Deep annotations and magic will begin, you will be able to connect MongoDB with Spark using your own model entities.Generic cell API, you do not need to specify the collection's schema or add anything to your POJOs, each document will be transform to an object "Cells".We added a few working examples for MongoDB in deep-examples subproject, take a look at:Entities:com.stratio.deep.examples.java.ReadingEntityFromMongoDBcom.stratio.deep.examples.java.WritingEntityToMongoDBcom.stratio.deep.examples.java.GroupingEntityWithMongoDBCells:com.stratio.deep.examples.java.ReadingCellFromMongoDBcom.stratio.deep.examples.java.WritingCellToMongoDBcom.stratio.deep.examples.java.GroupingCellWithMongoDBYou can check out our first steps guide here:First steps with Deep-MongoDBWe are working on further improvements!Support for ElasticSearch has been added in version 0.5.0.Support for Aerospike has been added in version 0.6.0.Examples:Entities:com.stratio.deep.examples.java.ReadingEntityFromAerospikecom.stratio.deep.examples.java.WritingEntityToAerospikecom.stratio.deep.examples.java.GroupingEntityWithAerospikeCells:com.stratio.deep.examples.java.ReadingCellFromAerospikecom.stratio.deep.examples.java.WritingCellToAerospikecom.stratio.deep.examples.java.GroupingCellWithAerospikeSupport for JDBC has been added in version 0.7.0.Examples:Entities:package com.stratio.deep.examples.java.ReadingEntityWithJdbcpackage com.stratio.deep.examples.java.WritingEntityWithJdbcCells:package com.stratio.deep.examples.java.ReadingCellWithJdbcpackage com.stratio.deep.examples.java.WritingCellWithJdbcCassandra, we tested versions from 1.2.8 up to 2.0.11 (for Spark <=> Cassandra integration).MongoDB, we tested the integration with MongoDB versions 2.2, 2.4 y 2.6 using Standalone, Replica Set and Sharded Cluster (for Spark <=> MongoDB integration).ElasticSearch, 1.3.0+Aerospike, 3.3.0+Spark 1.1.1Apache Maven >= 3.0.4Java 1.7Scala 2.10.3Clone the projectTo configure a development environment in Eclipse: import as Maven project. In IntelliJ: open the project by selecting the deep-parent POM fileInstall the project in you local maven repository. Enter deep-parent subproject and perform: mvn clean install (add -DskipTests to skip tests)Put Deep to work on a working cassandra + spark cluster. You have several options:Download a pre-configured Stratio platform VM Stratio's BigData platform (SDS).This VM will work on both Virtualbox and VMWare, and comes with a fully configured distribution that also includes Stratio Deep. We also distribute the VM with several preloaded datasets in Cassandra. This distribution will include Stratio's customized Cassandra distribution containing our powerful open-source lucene-based secondary indexes, see Stratio documentation for further information.Once your VM is up and running you can test Deep using the shell. Enter /opt/sds and run bin/stratio-deep-shell.Install a new cluster using the Stratio installer. Please refer to Stratio's website to download the installer and its documentation.You already have a working Cassandra server on your development machine: you need a spark+deep bundle, we suggest to create one by running:cd deep-scripts./make-distribution-deep.shthis will build a Spark distribution package with StratioDeep and Cassandra's jars included (depending on your machine this script could take a while, since it will compile Spark from sources).The package will be called spark-deep-distribution-X.Y.Z.tgz, untar it to a folder of your choice, enter that folder and issue a ./stratio-deep-shell, this will start an interactive shell where you can test StratioDeep (you may have noticed this is will start a development cluster started with MASTER="local").You already have a working installation os Cassandra and Spark on your development machine: this is the most difficult way to start testing Deep, but you know what you're doing you will have tocopy the Stratio Deep jars to Spark's 'jars' folder ($SPARK_HOME/jars).copy Cassandra's jars to Spark's 'jar' folder.copy Datastax Java Driver jar (v 2.0.x) to Spark's 'jar' folder.start spark shell and import the following:import com.stratio.deep.commons.annotations._import com.stratio.deep.commons.config._import com.stratio.deep.commons.entity._import com.stratio.deep.core.context._import com.stratio.deep.cassandra.config._import com.stratio.deep.cassandra.extractor._import com.stratio.deep.mongodb.config._import com.stratio.deep.mongodb.extractor._import com.stratio.deep.es.config._import com.stratio.deep.es.extractor._import com.stratio.deep.aerospike.config._import com.stratio.deep.aerospike.extractor._import org.apache.spark.rdd._import org.apache.spark.SparkContext._import org.apache.spark.sql.api.java.JavaSQLContextimport org.apache.spark.sql.api.java.JavaSchemaRDDimport org.apache.spark.sql.api.java.Rowimport scala.collection.JavaConversions._Once you have a working development environment you can finally start testing Deep. This are the basic steps you will always have to perform in order to use Deep:Build an instance of a configuration object: this will let you tell Deep the Cassandra endpoint, the keyspace, the table you want to access and much more.It will also let you specify which interface to use (the domain entity or the generic interface).We have a factory that will help you create a configuration object using a fluent API. Creating a configuration object is an expensive operation.Please take the time to read the java and scala examples provided in 'deep-examples' subproject and to read the comprehensive documentation at OpenStratio website.Create an RDD: using the DeepSparkContext helper methods and providing the configuration object you've just instantiated.Perform some computation over this RDD(s): this is up to you, we only help you fetching the data efficiently from Cassandra, you can use the powerful Spark API.(optional) write the computation results out to Cassandra: we provide a way to efficiently save the result of your computation to Cassandra.In order to do that you must have another configuration object where you specify the output keyspace/column family. We can create the output column family for you if needed.Please, refer to the comprehensive Stratio Deep documentation at Stratio website.Build an instance of a configuration object: this will let you tell Stratio Deep the MongoDB endpoint, the MongoDB database and collection you want to access and much more.It will also let you specify which interface to use (the domain entity).We have a factory that will help you create a configuration object using a fluent API. Creating a configuration object is an expensive operation.Please take the time to read the java and scala examples provided in 'deep-examples' subproject and to read the comprehensive Deep documentation at OpenStratio website.Create an RDD: using the DeepSparkContext helper methods and providing the configuration object you've just instantiated.Perform some computation over this RDD(s): this is up to you, we only help you fetching the data efficiently from MongoDB, you can use the powerful Spark API.(optional) write the computation results out to MongoDB: we provide a way to efficiently save the result of your computation to MongoDB.From version 0.4.x, Deep supports multiple datastores, in order to correctly implement this new feature Deep has undergone an huge refactor between versions 0.2.9 and 0.4.x. To port your code to the new version you should take into account a few changes we made.New Project StructureFrom version 0.4.x, Deep supports multiple datastores, in your project you should import only the maven dependency you will use: deep-cassandra, deep-mongodb, deep-elasticsearch or deep-aerospike.Changes to 'com.stratio.deep.entity.Cells'Until version 0.4.x the 'Cells' was implicitly associated to a record coming from a specific table. When performing a join in Spark, 'Cell' objects coming from different tables are mixed into an single 'Cells' object.Deep now keeps track of the original table a Cell object comes from, changing the internal structure of 'Cells', where each 'Cell' is associated to its 'table'.If you are a user of 'Cells' objects returned from Deep, nothing changes for you. The 'Cells' API keeps working as usual.If you manually create 'Cells' objects you can keep using the original API, in this case each Cell you add to your Cells object is automatically associated to a default table name.You can specify the default table name, or let Deep chose an internal default table name for you.We added a new constructor to 'Cells' accepting the default table name. This way the 'old' API will always manipulate 'Cell' objects associated to the specified default table.For each method manipulating the content of a 'Cells' object, we added a new method that also accepts the table name: if you call the method whose signature does not have the table name, the table action is performed over the Cell associated to the default table, otherwise the action is performed over the 'Cell'(s) associated to the specified table.size() y isEmpty() will compute their results taking into account all the 'Cell' objects contained.size(String tableName) and isEmpty(tableName) compute their result taking into account only the 'Cell' objects associated to the specified table.Obviously, when dealing with Cells objects, Deep always associates a Cell to the correct table name.Examples:Cells cells1 = new Cells(); // instantiate a Cells object whose default table name is generated internally.Cells cells2 = new Cells("my_default_table"); // creates a new Cells object whose default table name is specified by the usercells2.add(new Cell(...)); // adds to the 'cells2' object a new Cell object associated to the default tablecells2.add("my_other_table", new Cell(...)); // adds to the 'cells2' object a new Cell associated to "my_other_table" Changes to objects hierarchyIDeepJobConfig interface has been splitted into ICassandraDeepJobConfig and IMongoDeepJobConfig sub-interfaces. Each sub-interface exposes only the configuration properties that make sense for each data base.com.stratio.deep.config.DeepJobConfigFactory's factory methods now return the proper subinterface.DeepSparkContext has been splitted into CassandraDeepSparkContext and MongoDeepSparkContext.DeepJobConfigFactory has been renamed to ConfigFactory (to reduce verbosity).RDD creationMethods used to create Cell and Entity RDD has been merged into one single method:DeepSparkContext: createRDD(...)