Illustration Image

Cassandra.Link

The best knowledge base on Apache Cassandra®

Helping platform leaders, architects, engineers, and operators build scalable real time data platforms.

8/31/2020

Reading time:2 min

intuit/dse-pronto

by intuit

An automation suite for deploying and managing DataStax Cassandraclusters in AWS.This repository collects Intuit's DSE automation. We've taken all of our learning for managing Cassandra in AWS andcondensed it into a single package for others to leverage. It uses standard tools(Packer, Terraform, andAnsible) and can be run from a laptop. That said, we have a hardpreference for automated deployments using a CICD orchestrator along the lines of Jenkins,CodeBuild/CodeDeploy,Bamboo, GitLab, or Spinnaker.The tools in this repo can take you from an empty AWS account to a fully-functional DSE cluster, but you should have anunderstanding of AWS resources, Cassandra cluster management, and at least a passing familiarity with Packer, Terraform,and Ansible.This is not a "managed" Cassandra solution. If you need one of those, AWS has you covered.If you need a fully managed DataStax solution including OpsCenter and other DSE features,DataStax Astra is now officially a thing.On the other hand, if what you're looking for is an open source framework to help you manage your own DSE cluster...then welcome to DSE Pronto!Notes and FeaturesSupport for every phase of deployment, from an empty account to production:Baking an AMIDeploying a new VPCCreating account-wide resources (like IAM roles) and VPC-wide resources (like a bastion host for SSH)Launching a clusterRuntime operationsRestacking and resizing a clusterBringing nodes up and downConfiguring OpsCenterIncluding a number of predefined alerts and best practicesTransparent restacking operations, to keep in compliance with latest baseline imagesData stored on persistent EBS volumes, static EIP for predictable address, both located (using EC2 tags) and reattachedduring restackDSE 5 & DSE 6 both supported, along with DSE OpsCenter & DSE StudioLatest Amazon Linux 2.0 & Python 3 in useMore FAQs and details hereTools RequiredOn MacOS: brew install awscli coreutils packer ansible tfenv jq && tfenv install 0.12.24The scripts in this repo require a minimum of aws-cli/1.16.280 and botocore/1.13.16. Type aws --version to verify.Everything has also been tested with aws-cli/2.0.0 and associated prerequisites.Some scripts also require Python 3 (installation).In Docker: the included Dockerfile will produce a suitable Docker image, including all tools needed.Elsewhere:Install Packer: https://www.packer.io/intro/getting-started/install.htmlInstall Terraform (0.12.24): https://www.terraform.io/intro/getting-started/install.htmlinstall Ansible: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.htmlWhy Terraform 0.12.24? Go here to find out!1. Initial SetupThere's a bunch of one-time setup you'll need to do before you start baking AMIs or deploying clusters.Please follow all of the steps here before proceeding.2. Baking AMIsInstructions for baking AWS images with Packer are here.3. DeployingInstructions for deploying AWS resources with Terraform are here.4. Runtime OperationsInstructions for running playbooks with Ansible are here.5. OpsCenterInstructions for deploying and managing an OpsCenter node are here.6. DebuggingIf you're having trouble getting anything to work, go here for tips on debugging!7. Cleaning UpInstructions for deleting everything deployed by this repo are here.LinksContributingLicenseCopyright 2020 Intuit Inc.

Illustration Image

An automation suite for deploying and managing DataStax Cassandra clusters in AWS.

pronto

This repository collects Intuit's DSE automation. We've taken all of our learning for managing Cassandra in AWS and condensed it into a single package for others to leverage. It uses standard tools (Packer, Terraform, and Ansible) and can be run from a laptop. That said, we have a hard preference for automated deployments using a CICD orchestrator along the lines of Jenkins, CodeBuild/CodeDeploy, Bamboo, GitLab, or Spinnaker.

The tools in this repo can take you from an empty AWS account to a fully-functional DSE cluster, but you should have an understanding of AWS resources, Cassandra cluster management, and at least a passing familiarity with Packer, Terraform, and Ansible.

This is not a "managed" Cassandra solution. If you need one of those, AWS has you covered. If you need a fully managed DataStax solution including OpsCenter and other DSE features, DataStax Astra is now officially a thing.

On the other hand, if what you're looking for is an open source framework to help you manage your own DSE cluster... then welcome to DSE Pronto!

Notes and Features

  • Support for every phase of deployment, from an empty account to production:
    • Baking an AMI
    • Deploying a new VPC
    • Creating account-wide resources (like IAM roles) and VPC-wide resources (like a bastion host for SSH)
    • Launching a cluster
    • Runtime operations
      • Restacking and resizing a cluster
      • Bringing nodes up and down
    • Configuring OpsCenter
      • Including a number of predefined alerts and best practices
  • Transparent restacking operations, to keep in compliance with latest baseline images
    • Data stored on persistent EBS volumes, static EIP for predictable address, both located (using EC2 tags) and reattached during restack
  • DSE 5 & DSE 6 both supported, along with DSE OpsCenter & DSE Studio
  • Latest Amazon Linux 2.0 & Python 3 in use
  • More FAQs and details here

Tools Required

1. Initial Setup

There's a bunch of one-time setup you'll need to do before you start baking AMIs or deploying clusters.

Please follow all of the steps here before proceeding.

2. Baking AMIs

Instructions for baking AWS images with Packer are here.

3. Deploying

Instructions for deploying AWS resources with Terraform are here.

4. Runtime Operations

Instructions for running playbooks with Ansible are here.

5. OpsCenter

Instructions for deploying and managing an OpsCenter node are here.

6. Debugging

If you're having trouble getting anything to work, go here for tips on debugging!

7. Cleaning Up

Instructions for deleting everything deployed by this repo are here.

Links


Copyright 2020 Intuit Inc.

Related Articles

cluster
troubleshooting
datastax

GitHub - arodrime/Montecristo: Datastax Cluster Health Check Tooling

arodrime

4/3/2024

Checkout Planet Cassandra

Claim Your Free Planet Cassandra Contributor T-shirt!

Make your contribution and score a FREE Planet Cassandra Contributor T-Shirt! 
We value our incredible Cassandra community, and we want to express our gratitude by sending an exclusive Planet Cassandra Contributor T-Shirt you can wear with pride.

Join Our Newsletter!

Sign up below to receive email updates and see what's going on with our company

Explore Related Topics

AllKafkaSparkScyllaSStableKubernetesApiGithubGraphQl

Explore Further

packer