Illustration Image

Cassandra.Link

The best knowledge base on Apache Cassandra®

Helping platform leaders, architects, engineers, and operators build scalable real time data platforms.

2/21/2019

Reading time:1 min

scylladb/scylla-migrator

by John Doe

Make sure sbt is installed on your machine, and run build.sh.Create a config.yaml for your migration using the template config.yaml in the repository root. Read the comments throughout carefully.The Scylla Migrator is built against Spark 2.3.1, so you'll need to run that version on your cluster.After running build.sh, copy the jar from ./target/scala-2.11/scylla-migrator-assembly-0.0.1.jar and the config.yaml you've created to the Spark master server.Then, run this command on the Spark master server:spark-submit --class com.scylladb.migrator.Migrator \ --master spark://<spark-master-hostname>:7077 \ --conf spark.scylla.config=<path to config.yaml> <path to scylla-migrator-assembly-0.0.1.jar>To run in the local Docker-based setup:First start the environment:docker-compose up -dLaunch cqlsh in Cassandra's container and create a keyspace and a table with some data:docker-compose exec cassandra cqlsh<create stuff>Launch cqlsh in Scylla's container and create the destination keyspace and table with the same schema as the source table:docker-compose exec scylla cqlsh<create stuff>Edit the config.yaml file; note the comments throughout.Run build.sh.Then, launch spark-submit in the master's container to run the job:docker-compose exec spark-master spark-submit --class com.scylladb.migrator.Migrator \ --master spark://spark-master:7077 \ --conf spark.driver.host=spark-master \ --conf spark.scylla.config=/app/config.yaml \ /jars/scylla-migrator-assembly-0.0.1.jarThe spark-master container mounts the ./target/scala-2.11 dir on /jars and the repository root on /app. To update the jar with new code, just run build.sh and then run spark-submit again.

Illustration Image

Make sure sbt is installed on your machine, and run build.sh.

Create a config.yaml for your migration using the template config.yaml in the repository root. Read the comments throughout carefully.

The Scylla Migrator is built against Spark 2.3.1, so you'll need to run that version on your cluster.

After running build.sh, copy the jar from ./target/scala-2.11/scylla-migrator-assembly-0.0.1.jar and the config.yaml you've created to the Spark master server.

Then, run this command on the Spark master server:

spark-submit --class com.scylladb.migrator.Migrator \
  --master spark://<spark-master-hostname>:7077 \
  --conf spark.scylla.config=<path to config.yaml>
  <path to scylla-migrator-assembly-0.0.1.jar>

To run in the local Docker-based setup:

  1. First start the environment:
docker-compose up -d
  1. Launch cqlsh in Cassandra's container and create a keyspace and a table with some data:
docker-compose exec cassandra cqlsh
<create stuff>
  1. Launch cqlsh in Scylla's container and create the destination keyspace and table with the same schema as the source table:
docker-compose exec scylla cqlsh
<create stuff>
  1. Edit the config.yaml file; note the comments throughout.

  2. Run build.sh.

  3. Then, launch spark-submit in the master's container to run the job:

docker-compose exec spark-master spark-submit --class com.scylladb.migrator.Migrator \
  --master spark://spark-master:7077 \
  --conf spark.driver.host=spark-master \
  --conf spark.scylla.config=/app/config.yaml \
  /jars/scylla-migrator-assembly-0.0.1.jar

The spark-master container mounts the ./target/scala-2.11 dir on /jars and the repository root on /app. To update the jar with new code, just run build.sh and then run spark-submit again.

Related Articles

python
java
cassandra

Vald

John Doe

2/11/2024

Checkout Planet Cassandra

Claim Your Free Planet Cassandra Contributor T-shirt!

Make your contribution and score a FREE Planet Cassandra Contributor T-Shirt! 
We value our incredible Cassandra community, and we want to express our gratitude by sending an exclusive Planet Cassandra Contributor T-Shirt you can wear with pride.

Join Our Newsletter!

Sign up below to receive email updates and see what's going on with our company

Explore Related Topics

AllKafkaSparkScyllaSStableKubernetesApiGithubGraphQl

Explore Further

data.engineering

scylladb

scylladb
cassandra