{{ message }}
/ embulk Public
Embulk: Pluggable Bulk Data Loader.
\n
Apache-2.0
LICENSEApache-2.0
LICENSE-jctoolsUnknown
LICENSE-logbackMIT
LICENSE-slf4jMIT
LICENSE-slf4j-nettyYou can’t perform that action at this time.
Visual Flow is an ETL tool designed for effective data manipulation via convenient and user-friendly interface. The tool has the following capabilities:
Visual Flow application is divided into the following repositories:
Visual flow is an open-source software licensed under the Apache-2.0 license.
","id":"86395185-03f1-53f4-94a4-e3a2d2d45779","title":"GitHub - ibagroup-eu/Visual-Flow: Visual-Flow main repository","origin_url":"https://github.com/ibagroup-eu/Visual-Flow","url":"https://github.com/ibagroup-eu/Visual-Flow","wallabag_created_at":"2024-12-02T13:34:31+00:00","published_at":null,"published_by":"['ibagroup-eu']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/9187fdecad3a37939c1971bcdec19ffed4090307ee508b009f47c7bcd49a7f8d/ibagroup-eu/Visual-Flow","tags":["mongo","nocode","elasticsearch","open.source","cassandra","data.pipeline","elastic","aws.s3","etl","low.code","postgres"],"description":"Visual Flow is an ETL tool designed for effective data manipulation via convenient and user-friendly interface. The tool has the following capabilities:Can integrate data from heterogeneous sources:\nA..."},{"content":"![\"GitHub](\"https://github.com/datastax/cql-proxy/actions/workflows/test.yml/badge.svg\"){kind=icon}
![\"cql-proxy\"](\"https://github.com/datastax/cql-proxy/blob/main/cql-proxy.png\")
cql-proxy is designed to forward your application's CQL traffic to an appropriate database service. It listens on a local address and securely forwards that traffic.
The cql-proxy sidecar enables unsupported CQL drivers to work with DataStax Astra. These drivers include both legacy DataStax drivers and community-maintained CQL drivers, such as the gocql driver and the rust-driver.
cql-proxy also enables applications that are currently using Apache Cassandra or DataStax Enterprise (DSE) to use Astra without requiring any code changes. Your application just needs to be configured to use the proxy.
If you're building a new application using DataStax drivers, cql-proxy is not required, as the drivers can communicate directly with Astra. DataStax drivers have excellent support for Astra out-of-the-box, and are well-documented in the driver-guide guide.
Use the -h or --help flag to display a listing all flags and their corresponding descriptions and environment variables (shown below as items starting with $):
$ ./cql-proxy -h\nUsage: cql-proxy\nFlags:\n -h, --help Show context-sensitive help.\n -b, --astra-bundle=STRING Path to secure connect bundle for an Astra database. Requires '--username' and '--password'. Ignored if using the\n token or contact points option ($ASTRA_BUNDLE).\n -t, --astra-token=STRING Token used to authenticate to an Astra database. Requires '--astra-database-id'. Ignored if using the bundle path\n or contact points option ($ASTRA_TOKEN).\n -i, --astra-database-id=STRING Database ID of the Astra database. Requires '--astra-token' ($ASTRA_DATABASE_ID)\n --astra-api-url=\"https://api.astra.datastax.com\" URL for the Astra API ($ASTRA_API_URL)\n --astra-timeout=10s Timeout for contacting Astra when retrieving the bundle and metadata ($ASTRA_TIMEOUT)\n -c, --contact-points=CONTACT-POINTS,... Contact points for cluster. Ignored if using the bundle path or token option ($CONTACT_POINTS).\n -u, --username=STRING Username to use for authentication ($USERNAME)\n -p, --password=STRING Password to use for authentication ($PASSWORD)\n -r, --port=9042 Default port to use when connecting to cluster ($PORT)\n -n, --protocol-version=\"v4\" Initial protocol version to use when connecting to the backend cluster (default: v4, options: v3, v4, v5, DSEv1,\n DSEv2) ($PROTOCOL_VERSION)\n -m, --max-protocol-version=\"v4\" Max protocol version supported by the backend cluster (default: v4, options: v3, v4, v5, DSEv1, DSEv2)\n ($MAX_PROTOCOL_VERSION)\n -a, --bind=\":9042\" Address to use to bind server ($BIND)\n -f, --config=CONFIG YAML configuration file ($CONFIG_FILE)\n --debug Show debug logging ($DEBUG)\n --health-check Enable liveness and readiness checks ($HEALTH_CHECK)\n --http-bind=\":8000\" Address to use to bind HTTP server used for health checks ($HTTP_BIND)\n --heartbeat-interval=30s Interval between performing heartbeats to the cluster ($HEARTBEAT_INTERVAL)\n --idle-timeout=60s Duration between successful heartbeats before a connection to the cluster is considered unresponsive and closed\n ($IDLE_TIMEOUT)\n --readiness-timeout=30s Duration the proxy is unable to connect to the backend cluster before it is considered not ready\n ($READINESS_TIMEOUT)\n --idempotent-graph If true it will treat all graph queries as idempotent by default and retry them automatically. It may be\n dangerous to retry some graph queries -- use with caution ($IDEMPOTENT_GRAPH).\n --num-conns=1 Number of connection to create to each node of the backend cluster ($NUM_CONNS)\n --proxy-cert-file=STRING Path to a PEM encoded certificate file with its intermediate certificate chain. This is used to encrypt traffic\n for proxy clients ($PROXY_CERT_FILE)\n --proxy-key-file=STRING Path to a PEM encoded private key file. This is used to encrypt traffic for proxy clients ($PROXY_KEY_FILE)\n --rpc-address=STRING Address to advertise in the 'system.local' table for 'rpc_address'. It must be set if configuring peer proxies\n ($RPC_ADDRESS)\n --data-center=STRING Data center to use in system tables ($DATA_CENTER)\n --tokens=TOKENS,... Tokens to use in the system tables. It's not recommended ($TOKENS)To pass configuration to cql-proxy, either command-line flags, environment variables, or a configuration file can be used. Using the docker method as an example, the following samples show how the token and database ID are defined with each method.
docker run -p 9042:9042 \\\n --rm datastax/cql-proxy:v0.1.5 \\\n --astra-token <astra-token> --astra-database-id <astra-datbase-id>docker run -p 9042:9042 \\\n --rm datastax/cql-proxy:v0.1.5 \\\n -e ASTRA_TOKEN=<astra-token> -e ASTRA_DATABASE_ID=<astra-datbase-id>Proxy settings can also be passed using a configuration file with the --config /path/to/proxy.yaml flag. This can be mixed and matched with command-line flags and environment variables. Here are some example configuration files:
contact-points:\n - 127.0.0.1\nusername: cassandra\npassword: cassandra\nport: 9042\nbind: 127.0.0.1:9042\n# ...or with a Astra token:
astra-token: <astra-token>\nastra-database-id: <astra-database-id>\nbind: 127.0.0.1:9042\n# ...All configuration keys match their command-line flag counterpart, e.g. --astra-bundle is astra-bundle:, --contact-points is contact-points: etc.
Multi-region failover with DC-aware load balancing policy is the most useful case for a multiple proxy setup.
When configuring peers: it is required to set --rpc-address (or rpc-address: in the yaml) for each proxy and it must match is corresponding peers: entry. Also, peers: is only available in the configuration file and cannot be set using a command-line flag.
Here's an example of configuring multi-region failover with two proxies. A proxy is started for each region of the cluster connecting to it using that region's bundle. They all share a common configuration file that contains the full list of proxies.
Note: Only bundles are supported for multi-region setups.
cql-proxy --astra-bundle astra-region1-bundle.zip --username token --password <astra-token> \\\n --bind 127.0.0.1:9042 --rpc-address 127.0.0.1 --data-center dc-1 --config proxy.yamlcql-proxy ---astra-bundle astra-region2-bundle.zip --username token --password <astra-token> \\\n --bind 127.0.0.2:9042 --rpc-address 127.0.0.2 --data-center dc-2 --config proxy.yamlThe peers settings are configured using a yaml file. It's a good idea to explicitly provide the --data-center flag, otherwise; these values are pulled from the backend cluster and would need to be pulled from the system.local and system.peers table to properly setup the peers data-center: values. Here's an example proxy.yaml:
peers:\n - rpc-address: 127.0.0.1\n data-center: dc-1\n - rpc-address: 127.0.0.2\n data-center: dc-2Note: It's okay for the peers: to contain entries for the current proxy itself because they'll just be omitted.
There are three methods for using cql-proxy:
cql-proxycql-proxy installedcql-proxyBuild cql-proxy.
go buildRun with your desired database.
\nDataStax Astra cluster:
\n./cql-proxy --astra-token <astra-token> --astra-database-id <astra-database-id>The <astra-token> can be generated using these instructions. The proxy also supports using the Astra Secure Connect Bundle along with a client ID and secret generated using these instructions:
./cql-proxy --astra-bundle <your-secure-connect-zip> \\\n--username <astra-client-id> --password <astra-client-secret>\n
Apache Cassandra cluster:
\n./cql-proxy --contact-points <cluster node IPs or DNS names> [--username <username>] [--password <password>]Run with your desired database.
\nDataStax Astra cluster:
\ndocker run -p 9042:9042 \\\n datastax/cql-proxy:v0.1.5 \\\n --astra-token <astra-token> --astra-database-id <astra-database-id>The <astra-token> can be generated using these instructions. The proxy also supports using the Astra Secure Connect Bundle, but it requires mounting the bundle to a volume in the container:
docker run -v <your-secure-connect-bundle.zip>:/tmp/scb.zip -p 9042:9042 \\\n--rm datastax/cql-proxy:v0.1.5 \\\n--astra-bundle /tmp/scb.zip --username <astra-client-id> --password <astra-client-secret>Apache Cassandra cluster:
\ndocker run -p 9042:9042 \\\n datastax/cql-proxy:v0.1.5 \\\n --contact-points <cluster node IPs or DNS names> [--username <username>] [--password <password>]If you wish to have the docker image removed after you are done with it, add --rm before the image name datastax/cql-proxy:v0.1.5.
Using Kubernetes with cql-proxy requires a number of steps:
Generate a token following the Astra instructions. This step will display your Client ID, Client Secret, and Token; make sure you download the information for the next steps. Store the secure bundle in /tmp/scb.zip to match the example below.
Create cql-proxy.yaml. You'll need to add three sets of information: arguments, volume mounts, and volumes. A full example can be found here.
Argument: Modify the local bundle location, username and password, using the client ID and client secret obtained in the last step to the container argument.
\ncommand: [\"./cql-proxy\"]\nargs: [\"--astra-bundle=/tmp/scb.zip\",\"--username=Client ID\",\"--password=Client Secret\"]\n
Volume mounts: Modify /tmp/ as a volume mount as required.
volumeMounts:\n - name: my-cm-vol\n mountPath: /tmp/\n
Volume: Modify the configMap filename as required. In this example, it is named cql-proxy-configmap. Use the same name for the volumes that you used for the volumeMounts.
volumes:\n - name: my-cm-vol\n configMap:\n name: cql-proxy-configmap \n
Create a configmap. Use the same secure bundle that was specified in the cql-proxy.yaml.
kubectl create configmap cql-proxy-configmap --from-file /tmp/scb.zip Check the configmap that was created.
\nkubectl describe configmap cql-proxy-configmap\n Name: cql-proxy-configmap\n Namespace: default\n Labels: <none>\n Annotations: <none>\n Data\n ====\n BinaryData\n ====\n scb.zip: 12311 bytesCreate a Kubernetes deployment with the YAML file you created:
\nkubectl create -f cql-proxy.yamlCheck the logs:
\nkubectl logs <deployment-name>Drivers that use token-aware load balancing may print a warning or may not work when using cql-proxy. Because cql-proxy abstracts the backend cluster as a single endpoint this doesn't always work well with token-aware drivers that expect there to be at least \"replication factor\" number of nodes in the cluster. Many drivers print a warning (which can be ignored) and fallback to something like round-robin, but other drivers might fail with an error. For the drivers that fail with an error it is required that they disable token-aware or configure the round-robin load balancing policy.
","id":"17fac8a9-8b96-51ec-a7dd-dd3809bba528","title":"GitHub - datastax/cql-proxy: A client-side CQL proxy/sidecar.","origin_url":"https://github.com/datastax/cql-proxy","url":"https://github.com/datastax/cql-proxy","wallabag_created_at":"2024-11-01T17:26:01+00:00","published_at":null,"published_by":"['datastax']","reading_time":8,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/c2528e3426d98910ed27819e048b4c1081fab2ed2c7adbea6e6a3b1872deb30a/datastax/cql-proxy","tags":["migration","proxy","cassandra","cql"],"description":" cql-proxy is designed to forward your application's CQL traffic to an appropriate database service. It listens on a local address and securely forwards that traffic.The cql-proxy sidecar enables unsu..."},{"content":"Zero Downtime Migration (ZDM) Proxy is an open-source component developed in Go and based on client-server architecture. It enables you to migrate from one Apache Cassandra® cluster to another without downtime or code changes in the application client.
For details on ZDM Proxy, see zdm-proxy GitHub.
When using ZDM Proxy, the client connects to the proxy rather than to the source cluster. The proxy connects both to the source cluster and the target cluster. It sends read requests to the source cluster only, while write requests are forwarded to both clusters.
For details on how ZDM Proxy works, see Introduction to Zero Downtime Migration.
cqlsh installedConnect to your Aiven for Apache Cassandra service using cqlsh, for example.
cqlsh --ssl-u avnadmin -p YOUR_SECRET_PASSWORD cassandra-target-cluster-name.a.avns.net 12345
You can expect to receive output similar to the following:
Connected to a1b2c3d4-1a2b-3c4d-5e6f-a1b2c3d4e5f6 at cassandra-target-cluster-name.a.avns.net:12345
[cqlsh 6.1.0 | Cassandra 4.0.11 | CQL spec 3.4.5 | Native protocol v5]
In your target service, create the same keyspaces and tables you have in your source Apache Cassandra cluster.
create keyspace KEYSPACE_NAME with replication ={'class':'SimpleStrategy', 'replication_factor':3};
create table KEYSPACE_NAME.TABLE_NAME (n_id int, value int, primary key (n_id));
Download the ZDM Proxy's binary from ZDM Proxy releases.
wget https://github.com/datastax/zdm-proxy/releases/download/v2.1.0/zdm-proxy-linux-amd64-v2.1.0.tgz
tar xf zdm-proxy-linux-amd64-v2.1.0.tgz
Check if the binary has been downloaded successfully using ls in the relevant directory. You can expect to receive output similar to the following:
LICENSE zdm-proxy-linux-amd64-v2.1.0.tgz zdm-proxy-v2.1.0
Specify connection information by setting ZDM_TARGET_* and ZDM_ORIGIN_* environment variables using the export command.
note
ORIGIN refers to the source service.
Run the binary.
\nexportZDM_ORIGIN_CONTACT_POINTS=localhost
exportZDM_ORIGIN_USERNAME=cassandra
exportZDM_ORIGIN_PASSWORD=cassandra
exportZDM_ORIGIN_PORT=1234
exportZDM_TARGET_CONTACT_POINTS=cassandra-target-cluster-name.a.avns.net
exportZDM_TARGET_USERNAME=avnadmin
exportZDM_TARGET_PASSWORD=YOUR_SECRET_PASSWORD
exportZDM_TARGET_PORT=12345
exportZDM_TARGET_TLS_SERVER_CA_PATH=\"/tmp/ca.pem\"
exportZDM_TARGET_ENABLE_HOST_ASSIGNMENT=false
# ZDM_ORIGIN_ENABLE_HOST_ASSIGNMENT=false # (may be needed, see note)
./zdm-proxy-v2.1.0
ENABLE_HOST_ASSIGNMENT
Make sure you set the ZDM_TARGET_ENABLE_HOST_ASSIGNMENT variable. Otherwise, ZDM Proxy tries to connect to one of internal addresses of the cluster nodes, which are unavailable from outside. If this occurs to your source cluster, set ZDM_ORIGIN_ENABLE_HOST_ASSIGNMENT=false.
To connect to ZDM Proxy, use, for example, cqlsh. Provide connection details and, if your source or target require authentication, specify target username and password.
Check more details on using the credentials in Client application credentials.
The port that ZDM Proxy uses is 14002, which can be overridden.
Connect using ZDM Proxy.
\ncqlsh -u avnadmin -p YOUR_SECRET_PASSWORD localhost 14002
You can expect to receive output similar to the following:
\nConnected to CLUSTER_NAME at localhost:14002
[cqlsh 6.1.0 | Cassandra 4.1.3 | CQL spec 3.4.6 | Native protocol v4]
Check data in the table.
\nselect * from KEYSPACE_NAME.TABLE_NAME;
You can expect to receive output similar to the following:
\nn_id | value
------+-------
1|42
2|44
3|46
(3 rows)
Insert more data into the table to test how ZDM Proxy handles write request.
\ninsert into KEYSPACE_NAME.TABLE_NAME (n_id, value) values (4, 48);
insert into KEYSPACE_NAME.TABLE_NAME (n_id, value) values (5, 50);
Check again data inside the table.
\nselect * from KEYSPACE_NAME.TABLE_NAME;
You can expect to receive output similar to the following:
\nn_id | value
------+-------
5|50
1|42
2|44
4|48
3|46
(5 rows)
Connect to the source:
\ncqlsh localhost 1234
You can expect to receive output similar to the following:
\nConnected to SOURCE_CLUSTER_NAME at localhost:1234
[cqlsh 6.1.0 | Cassandra 4.1.3 | CQL spec 3.4.6 | Native protocol v5]
Check data in the table:
\nselect * from KEYSPACE_NAME.TABLE_NAME;
You can expect to receive output similar to the following:
\nn_id | value
------+-------
5|50
1|42
2|44
4|48
3|46
(5 rows)
ZDM Proxy has forwarded both the write request and the read request to the source cluster. As a result, all the values are there: both newly-added ones (50 and 48) and previously added ones (42, 44, and 46).
Connect to the target service.
\ncqlsh --ssl-u avnadmin -p YOUR_SECRET_PASSWORD cassandra-target-cluster-name.a.avns.net 12345
You can expect to receive output similar to the following:
\nConnected to a1b2c3d4-1a2b-3c4d-5e6f-a1b2c3d4e5f6 at cassandra-target-cluster-name.a.avns.net:12345
[cqlsh 6.1.0 | Cassandra 4.0.11 | CQL spec 3.4.5 | Native protocol v5]
Check data in the table.
\nselect * from KEYSPACE_NAME.TABLE_NAME;
You can expect to receive output similar to the following:
\nn_id | value
------+-------
5|50
4|48
(2 rows)
50 and 48 are there in the target table since ZDM Proxy has forwarded the write request to the target service. 42, 44, and 46 are not there since ZDM Proxy has not sent the read request to the target service.
The ZDM Proxy is client-server component written in Go that enables users to migrate with zero downtime from an Apache Cassandra® cluster to another (which may be an Astra cluster) and not requiring code changes in the application client.
The only change to the client is pointing it to the proxy rather than directly to the original cluster (Origin). In turn, the proxy connects to both Origin and Target clusters.
By default, the proxy will forward read requests only to the Origin cluster, though you can optionally configure it to forward reads to both clusters asynchronously, while writes will always be sent to both clusters concurrently.
An overview of the proxy architecture and logical flow can be viewed here.
In order to run the proxy, you'll need to set some environment variables or pass reference to YAML configuration file. Below you'll find a list with the most important variables along with their default values. The required ones are marked with a comment. Variable names for YAML configuration file do not have ZDM_ prefix and are lower-cased.
ZDM_ORIGIN_CONTACT_POINTS=10.0.0.1 #required\nZDM_ORIGIN_USERNAME=cassandra #required\nZDM_ORIGIN_PASSWORD=cassandra #required\nZDM_ORIGIN_PORT=9042\nZDM_TARGET_CONTACT_POINTS=10.0.0.2 #required\nZDM_TARGET_USERNAME=cassandra #required\nZDM_TARGET_PASSWORD=cassandra #required\nZDM_TARGET_PORT=9042\nZDM_PROXY_LISTEN_PORT=14002\nZDM_PROXY_LISTEN_ADDRESS=127.0.0.1\nZDM_PRIMARY_CLUSTER=ORIGIN\nZDM_READ_MODE=PRIMARY_ONLY\nZDM_LOG_LEVEL=INFOThe environment variables (or YAM configuration file) must be set for the proxy to work.
In order to get started quickly, in your local environment, grab a copy of the binary distribution in the Releases page. For the recommended installation in a production environment, check the Production Setup section below.
Now, suppose you have two clusters running at 10.0.0.1 and 10.0.0.2 with cassandra/cassandra credentials and the same key-value schema. You can start the proxy and connect it to these clusters like this:
$ export ZDM_ORIGIN_CONTACT_POINTS=10.0.0.1 \\ \nexport ZDM_TARGET_CONTACT_POINTS=10.0.0.2 \\\nexport ZDM_ORIGIN_USERNAME=cassandra \\\nexport ZDM_ORIGIN_PASSWORD=cassandra \\\nexport ZDM_TARGET_USERNAME=cassandra \\\nexport ZDM_TARGET_PASSWORD=cassandra \\\n./zdm-proxy-v2.0.0 # run the ZDM proxy executableIf you prefer to use YAML configuration file, an equivalent setup would look like:
$ cat zdm-config.yml\norigin_contact_points: 10.0.0.1\ntarget_contact_points: 10.0.0.2\norigin_username: cassandra\norigin_password: cassandra\ntarget_username: cassandra\ntarget_password: cassandra\n$ ./zdm-proxy-v2.0.0 --config=./zdm-config.yml # run the ZDM proxy executableAt this point, you should be able to connect some client such as CQLSH to the proxy and write data to it and the proxy will take care of forwarding the requests to both clusters concurrently.
$ cqlsh <proxy-ip-address> 14002 # this is the proxy's default listen portFrom the CQLSH prompt:
cqlsh> INSERT INTO test.keyvalue (key, value) VALUES (1, 'ABC');\ncqlsh> INSERT INTO test.keyvalue (key, value) VALUES (2, 'DEF');\ncqlsh> SELECT * FROM test.keyvalue;\ncqlsh> UPDATE test.keyvalue SET value='GYEKJF' WHERE key = 1;\ncqlsh> DELETE FROM test.keyvalue WHERE key = 2;You can confirm that the data is stored in both clusters by querying them directly in other cqlsh sessions.
Note: For the moment, the keyspace must be specified when accessing a table, even after using USE <keyspace>.
If you don't have test clusters readily available to try with, check the alternative method with docker-compose in the Contributor's guide, which will set up all the dependencies, including two test clusters and a proxy instance, in a containerized sandbox environment.
ZDM Proxy supports protocol versions v2, v3, v4, DSE_V1 and DSE_V2.
It technically doesn't support v5, but handles protocol negotiation so that the client application properly downgrades the protocol version to v4 if v5 is requested. This means that any client application using a recent driver that supports protocol version v5 can be migrated using the ZDM Proxy (as long as it does not use v5-specific functionality).
ZDM Proxy requires origin and target clusters to have at least one protocol version in common. It is therefore not feasible to configure Apache Cassandra 2.0 as origin and 3.x / 4.x as target. Below table displays protocol versions supported by various C* versions:
| Apache Cassandra | \nProtocol Version | \n
|---|---|
| 2.0 | \nV2 | \n
| 2.1 | \nV2, V3 | \n
| 2.2 | \nV2, V3, V4 | \n
| 3.x | \nV3, V4 | \n
| 4.x | \nV3, V4, V5 | \n
⚠️ Thrift is not supported by ZDM Proxy. If you are using a very old driver or cluster version that only supports Thrift then you need to change your client application to use CQL and potentially upgrade your cluster before starting the migration process.
In practice this means that ZDM Proxy supports the following cluster versions (as Origin and / or Target):
The setup we described above is only for testing in a local environment. It is NOT recommended for a production installation where the minimum number of proxy instances is 3.
For a comprehensive guide with the recommended production setup check the documentation available at Datastax Migration.
There you'll find information about an Ansible-based tool that automates most of the process.
For information on the packaged dependencies of the Zero Downtime Migration (ZDM) Proxy and their licenses, check out our open source report.
For frequently asked questions, please refer to our separate FAQ page.
","id":"4dcc572f-754e-539f-b5e7-e290530b2e94","title":"GitHub - datastax/zdm-proxy: An open-source component designed to seamlessly handle the real-time client application activity while a migration is in progress.","origin_url":"https://github.com/datastax/zdm-proxy","url":"https://github.com/datastax/zdm-proxy","wallabag_created_at":"2024-11-01T17:23:20+00:00","published_at":null,"published_by":"['datastax']","reading_time":4,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/2d9ec70e3a53cb00af12aa2c8f6c9f9568324cfe693483d667268c95dc17d22b/datastax/zdm-proxy","tags":["migration","proxy","datastax","cassandra"],"description":"The ZDM Proxy is client-server component written in Go that enables users to migrate with zero downtime from an Apache Cassandra® cluster to another (which may be an Astra cluster) and not requiring c..."},{"content":"{{ message }}
/ dreamfactory Public
DreamFactory API Management Platform
You can’t perform that action at this time.
docker-compose up -d(Note: The cassandra-env.sh included with this test environment limits the memory used by the setup via MAX_HEAP_SIZE and HEAP_NEWSIZE, allowing it to be run on a laptop with small memory. This would not be the case in production.)Set up the test Cassandra cluster:As the Docker containers are starting up, it can be convenient to see resource utilization via ctop:
![\"Example](\"https://www.pythian.com/hs-fs/hubfs/Imported_Blog_Media/image3-4.png?width=826&height=181&name=image3-4.png\")
Set up the filebeat softwareDo the following on each Cassandra node.1. Download the softwareYou would likely not need to install curl in your environment, but the Docker images used here are bare-bones by design. Theapt updatestatement is also necessary since typically repos are cleared of files after the requested packages are installed via the Dockerfile.apt update\n apt install curl -y\n curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.3-amd64.deb\n dpkg -i filebeat-6.2.3-amd64.debFor other operating systems, see:https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html. 2. Configure filebeatThe beats software allows for basic filtering and transformation via this configuration file. Put the below in /etc/filebeat/filebeat.yml. (This is edited from an example at:https://github.com/thelastpickle/docker-cassandra-bootstrap/blob/master/cassandra/config/filebeat.yml.) The values in the output.elasticsearch and setup.kibana are their respective IP addresses and port numbers. For filebeat.prospectors -- aprospectormanages all the log inputs -- two types of logs are used here, the system log and the garbage collection log. For each, we will exclude any compressed (.zip) files. The multiline* settings define how multiple lines in the log files are handled. Here, the log manager will find files that start with any of the patterns shown and append the following lines not matching the pattern until it reaches a new match. More options available at:https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html.
output.elasticsearch:\n enabled: true\n hosts: [\"172.16.238.31:9200\"]\n setup.kibana:\n host: \"172.16.238.33:5601\"\n filebeat.prospectors:\n - input_type: log\n paths:\n - \"/var/log/cassandra/system.log*\"\n document_type: cassandra_system_logs\n exclude_files: ['\\.zip$']\n multiline.pattern: '^TRACE|DEBUG|WARN|INFO|ERROR'\n multiline.negate: true\n multiline.match: after\n - input_type: log\n paths:\n - \"/var/log/cassandra/debug.log*\"\n document_type: cassandra_debug_logs\n exclude_files: ['\\.zip$']\n multiline.pattern: '^TRACE|DEBUG|WARN|INFO|ERROR'\n multiline.negate: true\n multiline.match: after3. Set up Kibana dashboards
filebeat setup --dashboardsExample output:
Loaded dashboards4. Start the beat
service filebeat startExample output:
2018-04-12T20:43:03.798Z INFO instance/beat.go:468 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]\n 2018-04-12T20:43:03.799Z INFO instance/beat.go:475 Beat UUID: 2f43562f-985b-49fc-b229-83535149c52b\n 2018-04-12T20:43:03.800Z INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.3\n 2018-04-12T20:43:03.801Z INFO elasticsearch/client.go:145 Elasticsearch url: https://172.16.238.31:9200\n 2018-04-12T20:43:03.802Z INFO pipeline/module.go:76 Beat name: C1\n Config OKView the graphs:Then view the Kibana graphs in a local browser at:https://localhost:5601. Run some sample load against one of the nodes to get more logs to experiment with:
cassandra-stress write n=20000 -pop seq=1..20000 -rate threads=4
![\"Example](\"https://www.pythian.com/hs-fs/hubfs/Imported_Blog_Media/image1-4.png?width=720&height=171&name=image1-4.png\")
Here are some sample queries to run in Kibana:![\"Kibana](\"https://www.pythian.com/hs-fs/hubfs/Imported_Blog_Media/image2-5.png?width=1020&height=550&name=image2-5.png\")
If you would like to see what the logs look at each step of the workflow, view logs within the Cassandra container in /var/log/cassandra like this:tail /var/log/cassandra/debug.logExample output:
WARN [PERIODIC-COMMIT-LOG-SYNCER] 2018-05-07 14:01:09,216 NoSpamLogger.java:94 - Out of 0 commit log syncs over the past 0.00s with average duration of Infinityms, 1 have exceeded the configured commit interval by an average of 80.52msView this data stored in Elasticsearch (in JSON format) in a browser like this:https://localhost:9200/_search?q=(message:(ERROR*%20OR%20WARN*)%20AND%20beat.hostname:DC1C2)Example output:
![\"\"](\"https://www.pythian.com/hs-fs/hubfs/Imported_Blog_Media/image4-3.png?width=836&height=570&name=image4-3.png\")
","id":"294431cd-3823-54d9-8293-635cc0794b4a","title":"Cassandra open-source log analysis in Kibana, using filebeat, modeled in Docker","origin_url":"https://www.pythian.com/blog/cassandra-open-source-log-analysis-kibana-using-filebeat-modeled-docker","url":"https://www.pythian.com/blog/cassandra-open-source-log-analysis-kibana-using-filebeat-modeled-docker","wallabag_created_at":"2024-02-16T18:58:29+00:00","published_at":null,"published_by":null,"reading_time":4,"domain_name":"www.pythian.com","preview_picture":"https://www.pythian.com/hubfs/Pythian-Featured-Meta-OG-Image.jpg","tags":["elastic","logging","kibana","cassandra"],"description":"I was recently asked to set up a solution for Cassandra open-source log analysis to include in an existing Elasticsearch-Logstash-Kibana (ELK) stack. After some research on more of the newer capabilit..."},{"content":"Vald\n![\"\"](\"https://vald.vdaas.org/images/vald_color_1.svg\")
Vald is designed and implemented based on the Cloud-Native architecture. It uses the fastest ANN Algorithm NGT to search neighbors. Vald has automatic vector indexing and index backup, and horizontal scaling which made for searching from billions of feature vector data. Vald is easy to use, feature-rich and highly customizable as you needed.
![\"\"](\"https://vald.vdaas.org/images/features_icon_asynchronize.png\"){kind=icon}
Usually the graph requires locking during indexing, which cause stop-the-world. But Vald uses distributed index graph so it continues to work during indexing.
\n![\"\"](\"https://vald.vdaas.org/images/features_icon_filtering.png\"){kind=icon}
Vald implements it's own highly customizable Ingress/Egress filter. Which can be configured to fit the gRPC interface.
\n![\"\"](\"https://vald.vdaas.org/images/features_icon_cloud-native.png\"){kind=icon}
Horizontal scalable on memory and cpu for your demand.
\n![\"\"](\"https://vald.vdaas.org/images/features_icon_auto-indexing.png\"){kind=icon}
Vald supports to auto backup feature using Object Storage or Persistent Volume which enables disaster recovery.
\n![\"\"](\"https://vald.vdaas.org/images/features_icon_distributed.png\"){kind=icon}
Vald distribute vector index to multiple agent, each agent stores different index.
\n![\"\"](\"https://vald.vdaas.org/images/features_icon_replication.png\"){kind=icon}
Vald stores each index in multiple agents which enables index replicas. Automatically rebalance the replica when some Vald agent goes down.
\n![\"\"](\"https://vald.vdaas.org/images/features_icon_easy.png\"){kind=icon}
Vald can be easily installed in a few steps.
\n![\"\"](\"https://vald.vdaas.org/images/features_icon_customizable.png\"){kind=icon}
You can configure the number of vector dimension, the number of replica and etc.
\n![\"\"](\"https://vald.vdaas.org/images/features_icon_multi-language.png\"){kind=icon}
Golang, Java, Nodejs and python is supported.
\nA managed service based on the open source library Para. Adding your own features is a pull request away on GitHub.
We are hosted on Amazon Web Services, inside a private cloud, which allows us to scale up and down quickly.
You can always switch to another hosting service or run Para on your infrastructure. We're cool with that.
Our simple JSON API and client SDKs allow you to easily integrate Para into your code.
Your users can be authenticated with their social accounts and Para will issue a JSON Web Token (JWT). LDAP and Active Directory are also supported. This combined with powerful resource permissions for each user, gives you great flexibility and security.
All your objects are indexed automatically by Elasticsearch - a robust and scalable search engine. Para supports many types of queries like basic field queries, wildcard, geo point, and \"more-like-this\" queries.
The API is simple and secure. Using basic HTTP methods you can create, read, update and delete objects. It also supports pagination and field limiting. All API calls are signed using the latest AWS signature v4 algorithm, used by Amazon.
Your objects are cached when they are created or updated. Caching allows for faster read times and decreases the load on the database. This works great for \"hot\" objects that don't change very often.
Para stores data in several regions of the AWS cloud and is powered by DynamoDB - a scalable, fault tolerant database with very low latency. Your data is stored in globally replicated tables, encrypted, on Solid State Drives (SSDs).
Premium plans feature an easy-to-use backup and restore control panel which enables you to import and export data from Para. Simply download the JSON data archive and restore it when needed.
per app, monthly
\nThe prices don't include European Union VAT (not applicable to customers outside the EU).
Billing is on a monthly basis. Annual billing is also available - pay 10 months and get 2 months free. Additional storage and app credits can be purchased at any time.
Frequently Asked QuestionsNo registration required, just sign in with a social account.
Multitenant backend server for building web and mobile apps rapidly. The backend for busy developers. (self-hosted or hosted)
507 stars 141 forks Branches Tags Activity
","id":"0533fa1a-4095-538d-8426-7b0ed0c2b5c1","title":"GitHub - Erudika/para: Multitenant backend server for building web and mobile apps rapidly. The backend for busy developers. (self-hosted or hosted)","origin_url":"https://github.com/Erudika/para","url":"https://github.com/Erudika/para","wallabag_created_at":"2024-01-26T14:53:22+00:00","published_at":null,"published_by":"['']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/2204f6ef5bbccb71656d9d43ff9116ca157ebf50d578547bbd1d8678bf3a3232/Erudika/para","tags":["mongo","rest","elasticsearch","cassandra","elastic","lucene","api","dynamo","baas"],"description":"Multitenant backend server for building web and mobile apps rapidly. The backend for busy developers. (self-hosted or hosted)paraio.orgLicenseApache-2.0 license507 stars 141 forks Branches Tags Activi..."},{"content":"{{ message }}
/ loopback-next Public
LoopBack makes it easy to build modern API applications that require complex integrations.
You can’t perform that action at this time.
Free multi-platform database tool for developers, database administrators, analysts and all people who need to work with databases. Supports all popular databases: MySQL, PostgreSQL, SQLite, Oracle, DB2, SQL Server, Sybase, MS Access, Teradata, Firebird, Apache Hive, Phoenix, Presto, etc.
","id":"62edea6c-0d3e-5635-a5ab-ead1f7c1785f","title":"DBeaver Community | Free Universal Database Tool","origin_url":"https://dbeaver.io/","url":"https://dbeaver.io/","wallabag_created_at":"2023-05-18T00:13:27+00:00","published_at":null,"published_by":null,"reading_time":null,"domain_name":"dbeaver.io","preview_picture":null,"tags":["hive","firebird","oracle","sybase","manager","sqlite","sql","database","datastax","cassandra","db2","mysql","postgres","presto"],"description":"Free multi-platform database tool for developers, database administrators, analysts and all people who need to work with databases. Supports all popular databases: MySQL, PostgreSQL, SQLite, Oracle, D..."},{"content":"![\"Sandeep](\"https://miro.medium.com/fit/c/96/96/0*dfwt6aN713ngmttu.jpg\")
Big data world is messy! The choice of frameworks, platforms, and data sources is simply mind-boggling. Several teams at Walmart use a mix of various stacks and tools to make sense of the data. While having choices is great, it does become a bit overwhelming when someone wants to access data in a coherent and uniform fashion. Each system exposes a different set of APIs, a different language to write queries in, and potentially a different way of analyzing the data at large scale.
It became clear to us early on that we need to invest on a tech stack, which can provide access to TBs of structured and semi-structured data for quick analysis and still be manageable with minimal overhead. It was also desirable that the query language be as close to SQL as possible so that it could be used with relative ease.
Some of the systems we looked at were Apache Impala, Apache Spark, and Presto. Each of these systems are battle tested and support a large number of production systems today. We started gravitating towards Presto for the following reasons:
In this article, we will explore our Presto set up and examine its performance for some common use cases. We will also focus on Apache Zeppelin, which is an excellent open source data visualization tool for interactive queries and helps team collaborate in real time.
Presto is an excellent distributed SQL engine that is optimized for running queries over huge datasets across multiple data sources. It supports distributed joins across multiple data sources like Hive, Kafka, Elastic Search, Cassandra, etc, thereby allowing a uniform access model for analytics.
Setting up the Presto cluster is easy. We set up a 8 node (each having 8 cores and 16G RAM) presto cluster in literally under 20 mins. Presto provides live query plans, which is a big plus. It helps us understand how to tune the queries for more performance. Presto also provides a JDBC driver, so accessing it from Java applications is very convenient as well.
Presto requires minimal configuration. We need to provide node.properties file under etc/, which requires bare minimum properties, e.g.
node.environment=prod
node.id=node1
node.data-dir=/var/presto/data
Each node also requires a config.properties.
coordinator=false
http-server.http.port=8080
query.max-memory=50GB
query.max-memory-per-node=8GB
discovery-server.enabled=true
discovery.uri=http://${coordinator_ip}:8080
One of the nodes has to be made coordinator and needs some more properties
coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=8080
discovery-server.enabled=true
discovery.uri=http://localhost:8080
One issue we ran into while configuring Presto was regarding the discovery.url. Presto uses its own discovery mechanism to identify the cluster workers. We observed that in the coordinator’s config.properties the discovery.url should be pointing to localhost while in all other nodes it should be point to coordinator’s IP
Finally, we need to add catalogs (connectors for different data sources) under etc/catalog directory. Example below shows how to add a Cassandra connector with minimal configuration
connector.name=cassandra
cassandra.contact-points=<comma separated IPs>
cassandra.consistency-level=LOCAL_ONE
cassandra.username=readonly_u
cassandra.password=readonly_p
cassandra.consistency-level=LOCAL_ONE
cassandra.load-policy.use-dc-aware=true
cassandra.load-policy.dc-aware.local-dc=DC1
cassandra.load-policy.use-token-aware=true
As mentioned before, Presto can connect to a lot of data sources including Cassandra, Hive, and Kafka among many others. A full list of connectors can be found here
Once the cluster is set up, the console can be accessed at the discovery.url
The first task that we set out to do was to measure query performance. We had a table handy with close to 260M rows. Each row had roughly 1 KB data. The schema of the table is as follows:
CREATE TABLE event_lookup (
id text,
bucket_id timestamp,
payload text,
PRIMARY KEY (( id ))
)
A simple query was fired on Cassandra which returned the count of total partitions in Cassandra.
SELECT COUNT(*) FROM event_lookup;
This resulted in a full table scan, by Presto with an impressive rate of ~ 418K rows / second! The inflow rate was ~ 400KB/sec. The screen shot below shows the rates along with other metrics.
During the query execution, there were occasional blips where read rate reached almost 1 million rows / second
Worker nodes also had almost equal distribution of load. From the screen shot below, each worker was processing on an average 51K rows / second.
We wanted to execute some thing more complicated than a count(*) query to examine performance better. We used a group by query as follows:
SELECT bucket_id, count(*) FROM event_lookup GROUP BY bucket_id ORDER BY count(*) LIMIT 10;
This time around, the results were even more impressive. The average query rate reached an impressive of 560K rows / sec!
The worker node distribution was a little skewed, possibly because of the way “bucket_id” field was partitioned among the Cassandra token ranges. What was stunning was that one of worker nodes reached a maximum of ~ 100K rows/sec TPS.
Having obtained satisfactory results, we moved on the other end of spectrum, the visualization!
Data analytics is not appealing if not complemented by a strong visualization. Visualization goes a long way in quickly deriving data patterns and correlations. While there are tools and libraries that provide excellent visualization support, they require a fairly good understanding of JavaScript and CSS, which is not something that data analysts are expected to know. Apache Zeppelin aims to fill that gap precisely. It provides analysts an extremely convenient way to create interactive web notebooks to write and execute the queries and visualize the results immediately. The web notebooks can be shared and collaborated in real time for valuable feedback. The queries can be scheduled so that they can be executed periodically. Like Presto, Zeppelin can connect to many data sources as well. In this article we will focus on JDBC connector for accessing Presto.
Adding a JDBC connector in Zeppelin is very easy. We just need to provide url, user, password, and driver properties. Below screen shot shows how to configure a Presto connection
Once the connector is added, we can create a notebook and add a new note. Inside the new note, the query can be executed as follows and the results can be visualized immediately in a variety of charts.
%jdbc SELECT bucket_id, COUNT(*) FROM event_buckets GROUP BY bucket_id
Note: if Presto is not configured as default interpreter, then you need to provide the name of the interpreter in the query
%jdbc(presto) SELECT bucket_id, COUNT(*) FROM event_buckets GROUP BY bucket_id
Zeppelin makes it very easy to build interactive form based UIs. Let’s try that by building a SQL input form. A general purpose SQL query executor requires following inputs:
We wrote a simple template zeppelin query
%jdbc
SELECT ${checkbox:Select Fields=Field1, Field1 | Field2 | Field3 | Field4} ${Free Form Fields = }
FROM
${Select Table=keyspace1.table1, keyspace1.table1 | keyspace2.table2 | keyspace3.table3}
${PREDICATE CLAUSE e.g. WHERE, GROUP BY = }
${ORDER BY CLAUSE = }
LIMIT ${limit = 10}
which resulted in the following UI
If we hide the query part, the UI becomes extremely simple
Now that’s the power of Zeppelin! In literally under 10 minutes, we are able to create a UI, where a user can
Using Zeppelin’s powerful display system (AngularJS) and variable binding mechanism, it’s very easy to create a chain of paragraphs which can execute in succession or in any arbitrary custom defined manner.
In this blog we explored how we can leverage Presto to run SQL queries over data sources. We also explored how we can wire Presto with Zeppelin to create compelling visualizations and analyze patterns quickly. This has resulted in quick analysis and collaboration between different teams. In the next article, we will be exploring how the Presto cluster is allowing us to join data across Cassandra, Kafka, Hive, etc for instant analysis on fast moving data. Stay tuned for more updates.
","id":"ea514087-85d6-5908-8463-0c121d6f6b68","title":"Exploring Presto and Zeppelin for fast data analytics and visualization","origin_url":"https://medium.com/walmartlabs/exploring-presto-and-zeppelin-for-fast-data-analytics-and-visualization-9cb4dca91c3d","url":"https://medium.com/walmartlabs/exploring-presto-and-zeppelin-for-fast-data-analytics-and-visualization-9cb4dca91c3d","wallabag_created_at":"2020-07-15T17:01:25+00:00","published_at":"2019-02-01T16:47:01+00:00","published_by":"['']","reading_time":7,"domain_name":"medium.com","preview_picture":"https://miro.medium.com/v2/resize:fit:640/1*BIAJryOgIbbUmYix6VOSGQ.jpeg","tags":["elastic","hive","zeppelin","cassandra","kafka","presto"],"description":"Photo credit MeditationsIntroductionBig data world is messy! The choice of frameworks, platforms, and data sources is simply mind-boggling. Several teams at Walmart use a mix of various stacks and too..."},{"content":"![\"image\"](\"https://zeppelin.apache.org/docs/0.7.0/assets/themes/zeppelin/img/notebook.png\")
Apache Zeppelin interpreter concept allows any language/data-processing-backend to be plugged into Zeppelin.\nCurrently Apache Zeppelin supports many interpreters such as Apache Spark, Python, JDBC, Markdown and Shell.
![\"image\"](\"https://zeppelin.apache.org/docs/0.7.0/assets/themes/zeppelin/img/available_interpreters.png\")
Adding new language-backend is really simple. Learn how to create your own interpreter.
Especially, Apache Zeppelin provides built-in Apache Spark integration. You don't need to build a separate module, plugin or library for it.
![\"image\"](\"https://zeppelin.apache.org/docs/0.7.0/assets/themes/zeppelin/img/spark_logo.png\"){kind=logo}
Apache Zeppelin with Spark integration provides
For the further information about Apache Spark in Apache Zeppelin, please see Spark interpreter for Apache Zeppelin.
Some basic charts are already included in Apache Zeppelin. Visualizations are not limited to Spark SQL query, any output from any language backend can be recognized and visualized.
![\"image\"](\"https://zeppelin.apache.org/docs/0.7.0/assets/themes/zeppelin/img/graph1.png\")
![\"image\"](\"https://zeppelin.apache.org/docs/0.7.0/assets/themes/zeppelin/img/graph2.png\")
Apache Zeppelin aggregates values and displays them in pivot chart with simple drag and drop. You can easily create chart with multiple aggregated values including sum, count, average, min, max.
![\"image\"](\"https://zeppelin.apache.org/docs/0.7.0/assets/themes/zeppelin/img/screenshots/pivot.png\")
Learn more about display systems in Apache Zeppelin.
Apache Zeppelin can dynamically create some input forms in your notebook.\n
![\"image\"](\"https://zeppelin.apache.org/docs/0.7.0/assets/themes/zeppelin/img/screenshots/dynamicform.png\")
Your notebook URL can be shared among collaborators. Then Apache Zeppelin will broadcast any changes in realtime, just like the collaboration in Google docs.
![\"image\"](\"https://zeppelin.apache.org/docs/0.7.0/assets/themes/zeppelin/img/screenshots/publish.png\")
Apache Zeppelin provides an URL to display the result only, that page does not include any menus and buttons inside of notebooks.\nYou can easily embed it as an iframe inside of your website in this way.\nIf you want to learn more about this feature, please visit this page.
![\"image\"](\"https://zeppelin.apache.org/docs/0.7.0/assets/themes/zeppelin/img/asf_logo.png\"){kind=logo}
Apache Zeppelin is Apache2 Licensed software. Please check out the source repository and how to contribute.\nApache Zeppelin has a very active development community.\nJoin to our Mailing list and report issues on Jira Issue tracker.
--
Why: We had a lot of very useful data in our Warehouse and wanted to take advantage of those data in some of our production service to enhance the user’s experience. So we choose to server them from Cassandra for all it’s pros which I’m am not going to get into in this blog.
First stage we went about writing a spark-cassandra exporter. It’s pretty simple and only a couple of line,
This works and took around ~ 30 mins to write ~150 Million rows. But once our services went live we saw the read latencies going a bit high during the bulk insertion time.
The spark-cassandra-connector that we are using here had a few configs that can be used to tune the writes here. Tried a bunch of tuning along the line of reducing concurrent and reducing throughput_mb_per_sec. They helped a bit but still there’s a clear increase in read latency.
Cassandra has sstableloader and we thought of testing it for this case. And so changed the code to use and saw that there’s barely any notable read latency during this task (only a slight increase in the 99 percentile, caused by the IO waits).
Also if you see the networks graph, the traffic is only on “network in” as now we are generating SSTables in spark and then pushing those tables directly to cassandra. The last spike in below network graph is from SSTable method and the rest are from batched writes.
Now let’s get into how to do that in code,
Fyi,
We’ve been using this method in production for over 6 months now, writing around ~ 300 million rows in < 30 mins without any issue to the read latencies.
Full example code can be found here, https://github.com/therako/sparkles/blob/master/src/main/scala/util/cassandra/SSTableExporter.scala
Visual Flow is an ETL tool designed for effective data manipulation via convenient and user-friendly interface. The tool has the following capabilities:
Visual Flow application is divided into the following repositories:
Visual flow is an open-source software licensed under the Apache-2.0 license.
","id":"86395185-03f1-53f4-94a4-e3a2d2d45779","title":"GitHub - ibagroup-eu/Visual-Flow: Visual-Flow main repository","origin_url":"https://github.com/ibagroup-eu/Visual-Flow","url":"https://github.com/ibagroup-eu/Visual-Flow","wallabag_created_at":"2024-12-02T13:34:31+00:00","published_at":null,"published_by":"['ibagroup-eu']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/9187fdecad3a37939c1971bcdec19ffed4090307ee508b009f47c7bcd49a7f8d/ibagroup-eu/Visual-Flow","tags":["mongo","nocode","elasticsearch","open.source","cassandra","data.pipeline","elastic","aws.s3","etl","low.code","postgres"],"description":"Visual Flow is an ETL tool designed for effective data manipulation via convenient and user-friendly interface. The tool has the following capabilities:Can integrate data from heterogeneous sources:\nA..."},{"content":"Multitenant backend server for building web and mobile apps rapidly. The backend for busy developers. (self-hosted or hosted)
507 stars 141 forks Branches Tags Activity
","id":"0533fa1a-4095-538d-8426-7b0ed0c2b5c1","title":"GitHub - Erudika/para: Multitenant backend server for building web and mobile apps rapidly. The backend for busy developers. (self-hosted or hosted)","origin_url":"https://github.com/Erudika/para","url":"https://github.com/Erudika/para","wallabag_created_at":"2024-01-26T14:53:22+00:00","published_at":null,"published_by":"['']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/2204f6ef5bbccb71656d9d43ff9116ca157ebf50d578547bbd1d8678bf3a3232/Erudika/para","tags":["mongo","rest","elasticsearch","cassandra","elastic","lucene","api","dynamo","baas"],"description":"Multitenant backend server for building web and mobile apps rapidly. The backend for busy developers. (self-hosted or hosted)paraio.orgLicenseApache-2.0 license507 stars 141 forks Branches Tags Activi..."},{"content":"As our business has been growing rapidly over the years, showcasing relevant content in the form of banners and carousels on high-traffic surfaces like the home page has become harder to support reliably. There has been an exponential increase in load on multiple systems such as application pods, databases, and caches, which is expensive to support and maintain. Before diving deeper into the details, let’s define some of the content such as banners and carousels.
Banners - These are discovery units represented by a creative with some content that could appear on any page within the app. Examples of banners in the app are shown in Figure 1 and Figure 2. They are usually used to merchandise stores/businesses/deals or to inform consumers about an event. We typically show multiple of them as a horizontally scrollable unit. Each of them could be clickable and lead to a carousel, specific store, webpage, etc.
![\"Figure](\"https://doordash.engineering/wp-content/uploads/2022/06/image5-473x1024.jpg\")
![\"Figure](\"https://doordash.engineering/wp-content/uploads/2022/06/image3-1-473x1024.jpg\")
Carousels - These are discovery units that could appear on any page within the app. They are usually used to group stores into a common theme/category so that consumers are able to discover content in a more organized way. The stores inside these units are horizontally scrollable. On clicking the gray arrow, a broader selection of the stores belonging to this theme is shown. Examples of carousels in the app are shown in Figure 3 and Figure 4.
![\"Figure](\"https://doordash.engineering/wp-content/uploads/2022/06/image2-473x1024.png\")
![\"Figure](\"https://doordash.engineering/wp-content/uploads/2022/06/image1-473x1024.png\")
The challenge we faced was that too many discovery units had to be fetched in real-time which could be relevant for a consumer address’ deliverable radius. This scaling challenge was causing a huge toll on the availability and reliability of carousels.
![\"Figure](\"https://doordash.engineering/wp-content/uploads/2022/06/search-service-14-1024x321.jpg\")
When using DoorDash the user experience starts the second you open the consumer app. On our backend systems, a lot starts happening immediately. One of the first things that happens is the set of stores (includes restaurants, grocery stores, pet stores, and so on) that are in the consumer address’ deliverable radius are fetched from search service which has business logic to determine what stores are relevant for customers given the logistical and geographical constraints. The number of stores available in a dense location like LA or NYC could easily reach thousands compared to hundreds in suburban areas.
Once relevant context like store data, consumer data, geographical information (like lat/long, city, district), etc. is calculated, a call is made from the Discovery system to the Campaign system to get a list of carousels and banners eligible, available, and relevant for the context that was passed along.
The Discovery system is responsible for content gathering, grouping and ranking of different entities for a given surface such as the home page.
The Campaign system internally tries to fetch campaigns for each store in the context to maximize recall.
Our banner and carousel system relies on campaign objects, which are containers that hold configuration rules such as:
These objects are configured at the store/business or a higher-order geographical level such as city, district, country, etc. Here, an example of a store could be the Safeway at 303 2nd St in San Francisco. A business is a bigger entity than a store that could have a list of stores belonging to it; for example, McDonalds could have 10,000+ stores.
The campaign system gives DoorDash strategy operators a very powerful way to be able to control the discovery surface content. Today we have banners and carousels that are manually curated, machine learning curated, and rule-based curated. All of them can be highly targeted to a set of users, shown during certain times of the day, have discounts associated with them, capped on how often they could show during a given time period, displayed at different start and end dates, and so on.
A single campaign could be targeting thousands of stores and each store in turn could have its own specific targeting, for example, a consumer needs to be new to the store to be eligible for the campaign.
Below is a demonstration of a simple campaign configuration that targets a store with store id = 999 to show a banner on the store page, and has specific start dates and end dates that it should show, and is only visible on the DoorDash app.
![\"Arunkumar\"](\"https://miro.medium.com/v2/resize:fill:88:88/1*Pgfv2m0dFhDO2zF2IUAGMQ.jpeg\")
{\n \"campaign\": {\n \"limitations\": [\n {\n \"type\": \"LIMITATION_TYPE_IS_ACTIVE\",\n \"is_active\": {\n \"value\": true\n },\n \"value\": \"is_active\"\n },\n {\n \"type\": \"LIMITATION_TYPE_EXPERIENCE\",\n \"experiences\": {\n \"experience\": [\n \"DOORDASH\"\n ]\n },\n \"value\": \"experiences\"\n },\n {\n \"type\": \"LIMITATION_TYPE_ACTIVE_DATES\",\n \"active_dates\": {\n \"start_time\": {\n \"seconds\": \"1613635200\",\n \"nanos\": 0\n },\n \"end_time\": {\n \"seconds\": \"1672559940\",\n \"nanos\": 0\n }\n },\n \"value\": \"active_dates\"\n }\n ],\n \"placements\": [\n {\n \"limitations\": [\n {\n \"type\": \"LIMITATION_TYPE_IS_ACTIVE\",\n \"is_active\": {\n \"value\": true\n },\n \"value\": \"is_active\"\n }\n ],\n \"type\": \"PLACEMENT_TYPE_STORE_PAGE_BANNER\",\n \"content_id\": {\n \"value\": \"most-loved-2022-store\"\n },\n \"sort_order\": {\n \"value\": 5\n },\n \"experiment_name\": {\n \"value\": \"testMostLoved2022\"\n }\n }\n ],\n \"memberships\": [\n {\n \"ids\": [\n \"9999999\"\n ],\n \"limitations\": [],\n \"user_criteria\": [],\n \"type\": \"MEMBERSHIP_ENTITY_TYPE_STORE\"\n }\n ],\n \"user_criteria\": [],\n \"id\": {\n \"value\": \"35145320-69bc-45cd-bb89-fc721b94a21d\"\n },\n \"name\": {\n \"value\": \"Campaign - BNY - Most Loved (Feb 2021)\"\n },\n \"description\": {\n \"value\": \"Most Loved tile - February refresh\"\n },\n \"created_by\": \"ujjwal.gulecha@doordash.com\",\n \"created_at\": {\n \"seconds\": \"1613690199\",\n \"nanos\": 0\n }\n }\n}For dense locations like Los Angeles, a single request would fan out to thousands of calls to our internal systems. During peak traffic, we would easily reach millions of queries per second to our database systems. This volume is particularly bad because it puts a lot of load on all our microservice systems involved such as BFFs, service apps, and database systems. We had to massively horizontally scale all of our systems to meet this demand. As the number of stores and campaigns are increasing at a rapid pace to highlight content, it becomes harder to support everything at such a scale.
So to summarize, there was a massive fan-out problem that kept growing and we were not sure how to proceed with it. We came up with a few solutions that we attempted to try to tame this problem.
The most obvious attempt to reduce the load on the application server sides was to batch the calls. We started experimenting with batching the calls to send X stores simultaneously, instead of all at once
After doing some performance testing, we empirically derived the optimal
batch size that worked for us. However, we soon started seeing that even this approach was ultimately not able to support our ever-growing expansion, selection, and discovery content. We could theoretically horizontally scale all our systems to support this, however that had its own challenges and we felt that was not the best use of our resources, nor was it sustainable in the longer term.
The four factors that did not allow us to support this in the long run can be summarized by this fan-out formula:
T * V * S * C (Traffic * Verticals * Stores * Campaigns)
Going back to the original problem, we were able to alleviate the load on application pods, but still had a load on our database systems. We had to research how to alleviate the load on our database systems.
As we began thinking more about this problem, one thing became clear to us: we need to reduce the cardinality of this fan-out. We needed a way to not request so many stores at a time but also not reduce the selection of stores; a way to group these stores which reduced this fan-out while fetching. Grouping stores by their geographical location made the most sense specifically in dense areas where you have lots of stores packed in a small area and then choose the best campaigns in those areas.
We looked into multiple existing solutions that would help us achieve this in a consistent, reliant, and scalable way. We looked at systems such as S2, Geohash, and H3
We did some testing, and based on empirical evidence, we chose H3 over other libraries. Here we outline some of the reasons that we thought H3 was a better fit.
H3 is Open source
H3 is an open-source project and is maintained by an active community with a wide list of high traffic production use cases. It is used by other technology companies, libraries like geojson2H3, and applications like kepler.gl.
High Availability and reliability
The API is simple, fast, and available in the languages DoorDash uses most frequently.
Relevance to DoorDash use case
H3 uses a hexagonal system which makes it easier to roughly approximate it to a circle which is closer to what DoorDash uses for calculating delivery radii. We compared the APIs and tested circle filling between S2 and H3 in our use cases. We found that H3 fits our use cases better and both S2 and H3 performed similarly in computational complexities. We would need to make geometric approximation work on top of geohash while H3 and S2 are both mature out of the box full solutions with good performance.
We could use the H3 library to visualize the world into different hexagons. There are different resolutions 1-15 that allow us to geographically condense stores into a large entity.
This solution allowed us to organize geo’s by hex’s instead of stores or what we were using before. We could now call hexes instead of individual stores and fetch the best campaigns for each hex thereby reducing cardinality.
Then the question arose: what size hexagon should we use? We wanted to run some benchmark tests to see what the best fit was for our situation. We did real-time analysis for proof of concept and were able to reduce the fan-out by a factor of 500x for non-dense areas and roughly 200x for dense areas.
We found that we reached the empirical optimal balance between computational complexity and approximation effectiveness at H3 resolution level of 9.
Once we finalized on using geo-hashes as our geographical filter for campaigns, we started looking at other ways of optimizing our fetching. Formerly we were fetching all campaigns and doing in-memory eligibility/filtering. This meant that the amount of data we fetched online was large.
We saw room for optimization if we could reduce the amount of data fetched by filtering closer to the storage layer. Essentially we wanted to move from “fetch all and filter in-memory” to “fetch filtered data”. This optimization was challenging to do with our existing non-relational database Cassandra which is great for fast lookups but not filtering on multiple keys.
Based on existing technologies at DoorDash, to optimize for filtering at data retrieval layer, we chose to go with Elasticsearch as this seemed a good fit for filtering at a data retrieval layer at high scale. This index contained campaign data which was denormalized in a way for efficient filtering and retrieval based on request context such as the geohash, start/end date, time of day and so on.
Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable efficient data retrieval system. We selected it for the following reasons:
Needle in a haystack
Elasticsearch was great for needle-in-a-haystack queries where we would want to filter out and retrieve a smaller amount of campaigns compared to the total data-set. We calculated that we could reduce fetching for ~50% of campaigns if we could filter them at the data-retrieval layer.
Boosting/Ranking
Elasticsearch has in-built support for boosting search results in case we want to prefer some campaigns over others while fetching. There could be cases where we manually would want to fetch certain campaigns over others due to any business logic reasons, elasticsearch provided an easy way to achieve this
Scalability
We knew with our growth, we would need a system that could easily scale by simply adding more servers. Elasticsearch is highly horizontally scalable
Multi-tenancy
We wanted to ensure we can use a system that can be extended for other use cases if needed. Elasticsearch can support our needs by allowing multiple indexes to be created, each having its own configurations
Support
It was widely being used already at DoorDash. This meant we would have expert support in case we ran into issues
We were able to massively reduce our operational costs while still maintaining high reliability and quality. In particular we were able to reduce ~50% costs for our Cassandra and Redis clusters and around 75% costs on our Kubernetes application hosting costs.
DoorDash is constantly evolving and expanding every single day. We believe this system has helped us serve our needs at this rapid growth pace, however we believe this is not the final solution. With DoorDash going into more countries internationally, expanding into other verticals, acquiring more consumers, and adding more stores to its platform, we will continue investing and iteratively improving our platform. Some ideas we are considering include, but are not limited to:
Thank you to Grace Chin, Chao Li, Fahd Ahmed, Jacob Greenleaf, Jennifer Be, Yichen Qiu, Shaohua Zhou, Shahrooz Ansari and Sonic Wang for their involvement and contribution to this project
","id":"7fe5400a-6b47-5795-b6aa-b84017025d43","title":"Taming Content Discovery Scaling Challenges with Hexagons and Elasticsearch","origin_url":"https://doordash.engineering/2022/06/28/taming-content-discovery-scaling-challenges-with-hexagons-and-elasticsearch/","url":"https://doordash.engineering/2022/06/28/taming-content-discovery-scaling-challenges-with-hexagons-and-elasticsearch/","wallabag_created_at":"2022-09-08T20:49:07+00:00","published_at":"2022-06-28T14:34:00+00:00","published_by":"['Ujjwal Gulecha']","reading_time":12,"domain_name":"doordash.engineering","preview_picture":"https://careersatdoordash.com/wp-content/uploads/2024/03/antoine-merour-QJDj2VWuOQg-unsplash.jpg","tags":["elasticsearch","cassandra"],"description":"As our business has been growing rapidly over the years, showcasing relevant content in the form of banners and carousels on high-traffic surfaces like the home page has become harder to support relia..."},{"content":"In Apache Cassandra Lunch #61, we will discuss different ways of indexing and working with Elassandra as well as showcasing a project I built utilizing Kafka with Elassandra. The live recording of Cassandra Lunch, which includes a more in-depth discussion and a demo, is embedded below in case you were not able to attend live. If you would like to attend Apache Cassandra Lunch live, it is hosted every Wednesday at 12 PM EST. Register here now!
![\"Elassandra](\"https://www.elassandra.io/elassandra.png\")
Apache Cassandra supports asynchronous multi-datacenters replication and various mechanisms to repair lost data. By closely integrating Elasticsearch with Cassandra, Elassandra provides search features on many datacenters.
When you need to increase read/write throughput, Elassandra automatically re-shards your Elasticsearch indices as new machines are added, allowing you to smoothly scale out to fit your business needs without downtime or heavy maintenance operations requirements.
By indexing Cassandra’s data into Elasticsearch, Kibana will allow you to get continuous and real-time data visualization of your applications.
By using a distributed transaction, Elassandra removes the single point of failure of Elasticsearch to manage its configuration.
Cassandra is designed for write-intensive workloads, hence, making Elassandra suitable for applications where a large amount of data is to be inserted (such as infrastructure logging, IoT, or events). So, Elasticsearch indices can be rebuilt whenever needed using the Cassandra tables without the creation of data duplication.
Failover-based approaches do not truly achieve high availability as far as write operations are concerned. Thanks to its multi-master design, Elassandra is always available either when a server/container fails or restarts because of some maintenance operations.
Elassandra closely integrates Elasticsearch within Apache Cassandra as a secondary index, allowing near-realtime search with all existing Elasticsearch APIs, plugins, and tools like Kibana. When you index a document, the JSON document is stored as a row in a Cassandra table and synchronously indexed in Elasticsearch.
![\"Diagram](\"http://doc.elassandra.io/en/latest/_images/elassandra1.jpg\")
Unlike Elasticsearch, sharding depends on the number of nodes in the datacenter, and the number of replicas is defined by your keyspace Replication Factor. Elasticsearch number of shards is just information about the number of nodes.
Write operations (Elasticsearch index, update, delete and bulk operations) are converted into CQL write requests managed by the coordinator node. The Elasticsearch document _id is converted into an underlying primary key, and the corresponding row is stored on many nodes according to the Cassandra replication factor. Then, on each node hosting this row, an Elasticsearch document is indexed through a Cassandra custom secondary index. Every document includes a _token fields used when searching.
![\"Diagram](\"http://doc.elassandra.io/en/latest/_images/write-path.png\")
The search request is done in two phases. First, in the query phase, the coordinator node adds a token_ranges filter to the query and broadcasts a search request to all nodes. This token_ranges filter covers the entire Cassandra ring and avoids duplicating results. Secondly, in the fetch phases, the coordinator fetches the required fields by issuing a CQL request in the underlying Cassandra table and builds the final JSON response.
![\"Diagram](\"http://doc.elassandra.io/en/latest/_images/search-path.png\")
Cassandra.Link is a knowledge base that we created for all things Apache Cassandra. Our goal with Cassandra.Link was to not only fill the gap of Planet Cassandra but to bring the Cassandra community together. Feel free to reach out if you wish to collaborate with us on this project in any capacity.
We are a technology company that specializes in building business platforms. If you have any questions about the tools discussed in this post or about any of our services, feel free to send us an email!
This is the second post in my series on improving node density and lowering costs with Apache Cassandra. In the previous post, I examined how streaming performance impacts node density and operational costs. In this post, I’ll focus on compaction throughput, and a recent optimization in Cassandra 5.0.4 that significantly improves it, CASSANDRA-15452.
This post assumes some familiarity with Apache Cassandra storage engine fundamentals. The documentation has a nice section covering the storage engine if you’d like to brush up before reading this post.
Compaction in Cassandra is the process of merging multiple SSTables and writing out new ones, discarding tombstones, resolving overwrites, and generally organizing data for efficient reads. It’s an I/O intensive background operation that directly competes with foreground operations for system resources. In a later post I’ll look at how compaction strategies impact node density, but for now, I’ll just focus on throughput.
As we continue to increase the amount of data we store per node, compaction performance becomes increasingly important. It affects:
Simply put: as your data volume and write throughput increase, compaction throughput must as well. If it doesn’t, you’ll hit a performance wall that effectively caps your maximum practical node density.
Despite the significant improvements to compaction throughput over the years, there are some circumstances where compaction performance is inadequate. Let’s take a look at the reason why, then dive into what can be done about it.
When doing any performance evaluation, it’s important to understand how to measure where your time is spent. A lot of folks make incorrect assumptions, and then waste a lot of time trying to optimize something that doesn’t matter. I’ve written several posts about how useful profiling with the async-profiler can be from an application perspective. For looking at the OS and hardware, the eBPF based toolkit bcc-tools can help you identify process bottlenecks. I’ve used these tools extensively over the years, and in this post I’ll show how they’ve helped identify two major performance bottlenecks in compaction. My easy-cass-lab software includes all these tools, as well as integration with AxonOps for Cassandra dashboards and operational tooling.
When investigating compaction behavior, I discovered an major inefficiency in how Cassandra was accessing disk. The problem was especially severe in cloud environments with disaggregated storage like AWS EBS, where IOPS (Input/Output Operations Per Second) are both limited and expensive when used improperly.
When Cassandra would read in data during compaction, it would read individual compressed chunks off disk, one small read at a time. Using bcc-tools, we can monitor every filesystem operation. Here I’m using xfsslower to record every read operation on the filesystem (original headers back in for clarity):
$ sudo /usr/share/bcc/tools/xfsslower 0 -p 26988 | awk '$4 == \"R\" { print $0 }'\nTracing XFS operations\nTIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME\n22:27:38 CompactionExec 26988 R 4096 0 0.01 nb-7-big-Statistics.db\n22:27:38 CompactionExec 26988 R 4096 4 0.00 nb-7-big-Statistics.db\n22:27:38 CompactionExec 26988 R 2062 8 0.00 nb-7-big-Statistics.db\n22:27:38 CompactionExec 26988 R 14907 0 0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988 R 14924 14 0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988 R 14896 29 0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988 R 14844 43 0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988 R 14923 58 0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988 R 14931 72 0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988 R 14905 87 0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988 R 14891 101 0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988 R 14919 116 0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988 R 14965 130 0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988 R 14918 145 0.01 nb-7-big-Data.db\n22:27:38 CompactionExec 26988 R 14930 160 0.01 nb-7-big-Data.db\nThe above is showing we’re reading about 14KB at a time. That’s the size of the compressed page. This pattern is terrible for performance on cloud storage systems like EBS, where:
Looking at a wall clock performance profile, we can see compaction is spending a LOT of time waiting on disk, in the really wide column with the pread call at the top:
![\"wall-clock-profile-compaction.png\"](\"https://rustyrazorblade.com/images/2025/wall-clock-profile-compaction.png\")
Readahead is a disk optimization strategy where the operating system reads a larger block of data than was requested into memory. The objective is reduce latency and improve performance for sequential read operations. Unfortunately, when you don’t need the data it’s reading, it can be the source of major performance problems. In my experience, read ahead is one of the worst culprits in the world of Cassandra performance. It’s especially terrible for lightweight transactions and counters, where we perform a read before write.
My advice to Cassandra operators is to reduce readahead to 4KB to avoid unnecessary read amplification on the read path.
Readahead does have one place, however, where it can benefit performance. You may have already guessed that it’s compaction. Let’s take a step back and look at how the size of our reads impacts our throughput in a simple benchmark. Larger reads, initiated either from read ahead or the user, should deliver improved throughput, especially when we’re dealing with a quota on our IOPS (EBS), our drives have higher latency (SAN), or both.
I ran benchmark tests with sequential fio workloads using different request sizes on a 3K IOPS GP3 EBS volume. Here’s the configuration used:
[global]\nrw=read\ndirectory=data\ndirect=1\ntime_based=1\nfile_service_type=normal\nstonewall\nsize=100M\nnumjobs=12\ngroup_reporting\n[bs4]\nstonewall\nruntime=60s\nblocksize=4k\n[bs8]\nstonewall\nruntime=60s\nblocksize=8k\n[bs16]\nstonewall\nruntime=60s\nblocksize=16k\n[bs32]\nstonewall\nruntime=60s\nblocksize=32k\n[bs64]\nstonewall\nruntime=60s\nblocksize=64k\n[bs128]\nstonewall\nblocksize=128k\nruntime=60s\n[bs256]\nstonewall\nruntime=60s\nblocksize=256k\nWhen reviewing the results, the benefits of using larger request sizes were evident:
| Request Size | \nIOPS | \nThroughput | \n
|---|---|---|
| 4K | \n3049 | \n11.9 MB/s | \n
| 8K | \n3012 | \n23 MB/s | \n
| 16K | \n3013 | \n47 MB/s | \n
| 32K | \n3013 | \n94 MB/s | \n
| 64K | \n1938 | \n121 MB/s | \n
| 128K | \n957 | \n120 MB/s | \n
| 256K | \n478 | \n120 MB/s | \n
The data shows that using 256KB reads instead of 16KB reads would deliver almost 3x the throughput while using only 1/6th of the provisioned IOPS. That’s a massive efficiency improvement. Rather than chewing through all our IOPS to deliver a paltry 47MB/s of throughput, we’re only using about 500 for 120MB/s. That means if we can see these gains in the database, we’ll be able to compact faster, put more data on each node, and lower our total cost.
In CASSANDRA-15452, I worked with my fellow Cassandra committer Jordan West to implement a solution: an efficient, internal read-ahead buffer for bulk reading operations. Here’s how it works:
This approach maximizes IOPS efficiency by using larger reads during compaction (as well as repair and range reads) that deliver more data per operation. For cloud environments, it’s a game-changer that directly aligns with storage provider recommendations. AWS EBS, for instance, considers any I/O operation up to 256KB as a single operation, so by using the largest possible size we should get optimal performance.
When Jordan and I tested the implementation using easy-cass-lab on EBS, the results were nothing short of spectacular. The 10.0.2.171 node is running our patched version, the other two nodes are running an unpatched release. The graphs clearly show a 2-3x improvement to throughput and a 3x reduction in IOPS.
![\"15452-bytes-read.png\"](\"https://rustyrazorblade.com/images/2025/15452-bytes-read.png\")
![\"Compaction](\"https://rustyrazorblade.com/images/2025/15452-compaction.png\")
You can see the results in the flamegraph as well. The calls to pread take up significantly less time.
![\"wall-clock-profile-compaction-after-15452.png\"](\"https://rustyrazorblade.com/images/2025/wall-clock-profile-compaction-after-15452.png\")
We can use xfsslower from bcc-tools again to watch the filesystem access:
$ sudo /usr/share/bcc/tools/xfsslower 0 -p $(cassandra-pid) | awk '$4 == \"R\" { print $0 }'\nTIME COMM PID T BYTES OFF_KB LAT(ms) FILENAME\n14:40:29 CompactionExec 1782 R 262144 256 0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782 R 262144 512 0.06 nb-4-big-Data.db\n14:40:29 CompactionExec 1782 R 262144 768 0.06 nb-4-big-Data.db\n14:40:29 CompactionExec 1782 R 262144 1024 0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782 R 262144 1280 0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782 R 262144 1536 0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782 R 241123 1792 0.07 nb-4-big-Data.db\nThis is a lot better, now we’re fetching 256KB at a time using way fewer requests.
The EBS test configuration used a GP3 volume with 3K IOPS and 256MB throughput. With the existing code, compaction was bottlenecked by IOPS, peaking at exactly 3K IOPS but achieving only about 51MB/s throughput. With our optimization, the same operation used only ~500 IOPS to achieve around 106MB/s—a more than 2x improvement in throughput with 1/3IOPS.
In our most aggressive testing, we actually hit the EBS throughput limit rather than the IOPS limit. That’s a significant transformation in Cassandra’s resource utilization profile.
The patch also has the benefit of applying to anti-compaction, repair, and range reads. We can see a significant reduction in range reads, aka table scans:
![\"15452-range-reads.png\"](\"https://rustyrazorblade.com/images/2025/15452-range-reads.png\")
If you’re running Spark jobs using the Cassandra connector, you should see an improvement in performance, and your repair times should decrease.
Yes, absolutely! There’s several more improvements to IO that will help improve things. I’ll cover them here very quickly, and if there’s interest I’ll write about them in detail in a future post.
When compacting, we read data out of the Statistics.db file before reading the data itself. This is completely unnecessary, as it’s stats about the data we’re about to read. Skipping this can reduce IO even further. Looking at a compaction’s IO activity, I see about 30% of the filesystem access is reading from Statistics.db:
14:40:29 CompactionExec 1782 R 4096 0 0.00 nb-3-big-Statistics.db\n14:40:29 CompactionExec 1782 R 701 4 0.00 nb-3-big-Statistics.db\n14:40:29 CompactionExec 1782 R 4096 0 0.00 nb-4-big-Statistics.db\n14:40:29 CompactionExec 1782 R 4096 4 0.00 nb-4-big-Statistics.db\n14:40:29 CompactionExec 1782 R 1962 8 0.00 nb-4-big-Statistics.db\n14:40:29 CompactionExec 1782 R 262144 0 0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782 R 2115 0 0.01 nb-3-big-Data.db\n14:40:29 CompactionExec 1782 R 262144 256 0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782 R 262144 512 0.06 nb-4-big-Data.db\n14:40:29 CompactionExec 1782 R 262144 768 0.06 nb-4-big-Data.db\n14:40:29 CompactionExec 1782 R 262144 1024 0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782 R 262144 1280 0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782 R 262144 1536 0.07 nb-4-big-Data.db\n14:40:29 CompactionExec 1782 R 241123 1792 0.07 nb-4-big-Data.db\nThis has already been fixed in trunk by Branimir Lambov in CASSANDRA-20092 and is being backported to 5.0 by Jordan.
Let’s talk more about page cache. Since we go through the Linux page cache when doing reads, we want to make sure it’s working optimally. Page cache lets us avoid going to disk! Unfortunately we also use it when reading for compaction. This is a problem because we’re pulling data into the page cache that we plan on deleting. To make room for the new data, other data will be evicted. If we compact 10GB of data, we’re pushing out a lot of valuable data from the page cache, meaning it needs to be fetched back into memory later on. Using Direct I/O we can bypass the page cache entirely, which will prevent data from being evicted. This can be a huge help in latency sensitive systems or systems where IOPS are limited like EBS.
I’ve filed CASSANDRA-19987 to look at this.
Next, compression. When we’re writing to disk, we fill a buffer, sized by the chunk_length_in_kb table setting, compress, and write to disk. The compression here is a blocking call, which means we can spend a lot of time waiting on compression to finish, when we could be reading and merging the next chunk in parallel This can show up as a performance bottleneck, so I’ve filed CASSANDRA-20085 to look into it.
When a system is not bottlenecked on disk I/O, such as when using NVMe, the main issue we’ll run into is our heap allocation rate. I’ll go into details in a future post, but for now, it’s enough to know that the more memory we allocate, the worse our performance. Being smart about memory allocations can make a big difference in overall time spent, as allocations aren’t free. It also reduces both the frequency and duration of Garbage Collection. Big wins all around.
I recently profiled an instance where the row size was about 2KB (not out of the ordinary) and found that a single call was accounting for roughly 50% of memory allocated. Fixing this one thing has the potential to deliver a massive performance improvement, especially in workloads where we have either lots of fields, or large fields like serialized blobs.
Reaching again for async-profiler, this time we run it with -e alloc to track allocations and --reverse to reverse the stacks. I do this because the same underlying call comes from the read path and compaction, and I want to see the time in aggregate.
![\"allocation-profile-compaction.png\"](\"https://rustyrazorblade.com/images/2025/allocation-profile-compaction.png\")
Addressing this single allocation won’t just deliver faster compaction, but will reduce pressure on the heap, which in turn reduces GC overhead. As part of this series I’ll also be covering GC, as a lot’s changed since I wrote about it last.
I’ve filed CASSANDRA-20428 and there’s already a fair bit of discussion about different approaches to solving the problem.
Maximizing compaction throughput is critical for achieving higher node density with Apache Cassandra. The improvements in CASSANDRA-15452 have removed one of the primary bottlenecks that previously limited practical node size in a lot of clusters.
By upgrading to Cassandra 5.0.4 (or later) you can:
This improvement, combined with the streaming optimizations discussed in the previous post, creates a multiplier effect on your ability to increase node density. Each optimization removes a bottleneck, allowing you to push your hardware further and achieve more with less.
In my next post, I’ll be discussing how and why compaction strategies affect node density. Picking the right strategy can have a significant impact on your cluster’s performance and cost efficiency. Make sure you sign up for my mailing list if you’re interested in getting notified when it’s released!
If you found this post helpful, please consider sharing to your network. I'm also available to help you be successful with your distributed systems! Pleasereach outif you're interested in working with me, and I'll be happy to schedule a free one-hour consultation.","id":"50f3d448-afe6-5e2f-a1f9-87786beecc12","title":"Cassandra Compaction Throughput Performance Explained","origin_url":"https://rustyrazorblade.com/post/2025/04-compaction-throughput/","url":"https://rustyrazorblade.com/post/2025/04-compaction-throughput/","wallabag_created_at":"2025-04-24T12:03:02+00:00","published_at":"2025-04-16T00:00:00+00:00","published_by":"['']","reading_time":14,"domain_name":"rustyrazorblade.com","preview_picture":"https://rustyrazorblade.com/images/2025/wall-clock-profile-compaction.png","tags":["cassandra","performance"],"description":"This is the second post in my series on improving node density and lowering costs with Apache Cassandra. In the previous post, I examined how streaming performance impacts node density and operational..."},{"content":"Welcome to the Awesome Accord repository! This guide provides resources and examples for implementing ACID transactions in Apache Cassandra. Learn how to leverage distributed transactions for building reliable applications.
Accord is in active development and still a feature branch in the Apasche Cassandra® Repo. You will find bug. What we ask is that you help with a contribution of a bug report.
You can use the Github discussions bug report forum for this or use the Planet Cassandra Discord channel for accord listed below. A bug report should have the folowing:
If you have suggestions about syntax or improving the overall developer expirience, we want to hear about that to! Add it as a suggestion or feature request using Github discussions or let us know in the Planet Cassandra Discord.
Now, on to the fun!
docker pull pmcfadin/cassandra-accord\ndocker run -d --name cassandra-accord -p 9042:9042 pmcfadin/cassandra-accordbrew tap rustyrazorblade/rustyrazorblade\nbrew install easy-cass-lab/\n├── docker/ # Docker configuration and setup\n├── easy-cass-lab/ # Multi-node testing environment\n├── examples/ # Implementation examples\n│ ├── banking/ # Financial transaction examples\n│ ├── inventory/ # Stock management examples\n│ └── user-mgmt/ # User operations examples\n└── docs/ # Guides and documentation\n
Our documentation includes:
BEGIN TRANSACTION\n LET fromBalance = (SELECT account_balance \n FROM ks.accounts \n WHERE account_holder='alice');\n IF fromBalance.account_balance >= 20 THEN\n UPDATE ks.accounts \n SET account_balance -= 20 \n WHERE account_holder='alice';\n UPDATE ks.accounts \n SET account_balance += 20 \n WHERE account_holder='bob';\n END IF\nCOMMIT TRANSACTION;Apache License 2.0
","id":"f304c7a0-b03c-5bc4-aa5d-df399a9124f4","title":"GitHub - pmcfadin/awesome-accord: Repository of all kinds of things to help you get up and running with ACID transactions on Apache Cassandra®","origin_url":"https://github.com/pmcfadin/awesome-accord","url":"https://github.com/pmcfadin/awesome-accord","wallabag_created_at":"2025-01-16T16:28:31+00:00","published_at":null,"published_by":"['pmcfadin']","reading_time":1,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/3e477fb2dd2b1ded1c5b53477f4848297badc75ece00c5b49bad1476fdb76167/pmcfadin/awesome-accord","tags":["acid","open.source","cassandra","accord"],"description":"Welcome to the Awesome Accord repository! This guide provides resources and examples for implementing ACID transactions in Apache Cassandra. Learn how to leverage distributed transactions for building..."},{"content":"Visual Flow is an ETL tool designed for effective data manipulation via convenient and user-friendly interface. The tool has the following capabilities:
Visual Flow application is divided into the following repositories:
Visual flow is an open-source software licensed under the Apache-2.0 license.
","id":"86395185-03f1-53f4-94a4-e3a2d2d45779","title":"GitHub - ibagroup-eu/Visual-Flow: Visual-Flow main repository","origin_url":"https://github.com/ibagroup-eu/Visual-Flow","url":"https://github.com/ibagroup-eu/Visual-Flow","wallabag_created_at":"2024-12-02T13:34:31+00:00","published_at":null,"published_by":"['ibagroup-eu']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/9187fdecad3a37939c1971bcdec19ffed4090307ee508b009f47c7bcd49a7f8d/ibagroup-eu/Visual-Flow","tags":["mongo","nocode","elasticsearch","open.source","cassandra","data.pipeline","elastic","aws.s3","etl","low.code","postgres"],"description":"Visual Flow is an ETL tool designed for effective data manipulation via convenient and user-friendly interface. The tool has the following capabilities:Can integrate data from heterogeneous sources:\nA..."},{"content":"
cql-proxy is designed to forward your application's CQL traffic to an appropriate database service. It listens on a local address and securely forwards that traffic.
The cql-proxy sidecar enables unsupported CQL drivers to work with DataStax Astra. These drivers include both legacy DataStax drivers and community-maintained CQL drivers, such as the gocql driver and the rust-driver.
cql-proxy also enables applications that are currently using Apache Cassandra or DataStax Enterprise (DSE) to use Astra without requiring any code changes. Your application just needs to be configured to use the proxy.
If you're building a new application using DataStax drivers, cql-proxy is not required, as the drivers can communicate directly with Astra. DataStax drivers have excellent support for Astra out-of-the-box, and are well-documented in the driver-guide guide.
Use the -h or --help flag to display a listing all flags and their corresponding descriptions and environment variables (shown below as items starting with $):
$ ./cql-proxy -h\nUsage: cql-proxy\nFlags:\n -h, --help Show context-sensitive help.\n -b, --astra-bundle=STRING Path to secure connect bundle for an Astra database. Requires '--username' and '--password'. Ignored if using the\n token or contact points option ($ASTRA_BUNDLE).\n -t, --astra-token=STRING Token used to authenticate to an Astra database. Requires '--astra-database-id'. Ignored if using the bundle path\n or contact points option ($ASTRA_TOKEN).\n -i, --astra-database-id=STRING Database ID of the Astra database. Requires '--astra-token' ($ASTRA_DATABASE_ID)\n --astra-api-url=\"https://api.astra.datastax.com\" URL for the Astra API ($ASTRA_API_URL)\n --astra-timeout=10s Timeout for contacting Astra when retrieving the bundle and metadata ($ASTRA_TIMEOUT)\n -c, --contact-points=CONTACT-POINTS,... Contact points for cluster. Ignored if using the bundle path or token option ($CONTACT_POINTS).\n -u, --username=STRING Username to use for authentication ($USERNAME)\n -p, --password=STRING Password to use for authentication ($PASSWORD)\n -r, --port=9042 Default port to use when connecting to cluster ($PORT)\n -n, --protocol-version=\"v4\" Initial protocol version to use when connecting to the backend cluster (default: v4, options: v3, v4, v5, DSEv1,\n DSEv2) ($PROTOCOL_VERSION)\n -m, --max-protocol-version=\"v4\" Max protocol version supported by the backend cluster (default: v4, options: v3, v4, v5, DSEv1, DSEv2)\n ($MAX_PROTOCOL_VERSION)\n -a, --bind=\":9042\" Address to use to bind server ($BIND)\n -f, --config=CONFIG YAML configuration file ($CONFIG_FILE)\n --debug Show debug logging ($DEBUG)\n --health-check Enable liveness and readiness checks ($HEALTH_CHECK)\n --http-bind=\":8000\" Address to use to bind HTTP server used for health checks ($HTTP_BIND)\n --heartbeat-interval=30s Interval between performing heartbeats to the cluster ($HEARTBEAT_INTERVAL)\n --idle-timeout=60s Duration between successful heartbeats before a connection to the cluster is considered unresponsive and closed\n ($IDLE_TIMEOUT)\n --readiness-timeout=30s Duration the proxy is unable to connect to the backend cluster before it is considered not ready\n ($READINESS_TIMEOUT)\n --idempotent-graph If true it will treat all graph queries as idempotent by default and retry them automatically. It may be\n dangerous to retry some graph queries -- use with caution ($IDEMPOTENT_GRAPH).\n --num-conns=1 Number of connection to create to each node of the backend cluster ($NUM_CONNS)\n --proxy-cert-file=STRING Path to a PEM encoded certificate file with its intermediate certificate chain. This is used to encrypt traffic\n for proxy clients ($PROXY_CERT_FILE)\n --proxy-key-file=STRING Path to a PEM encoded private key file. This is used to encrypt traffic for proxy clients ($PROXY_KEY_FILE)\n --rpc-address=STRING Address to advertise in the 'system.local' table for 'rpc_address'. It must be set if configuring peer proxies\n ($RPC_ADDRESS)\n --data-center=STRING Data center to use in system tables ($DATA_CENTER)\n --tokens=TOKENS,... Tokens to use in the system tables. It's not recommended ($TOKENS)To pass configuration to cql-proxy, either command-line flags, environment variables, or a configuration file can be used. Using the docker method as an example, the following samples show how the token and database ID are defined with each method.
docker run -p 9042:9042 \\\n --rm datastax/cql-proxy:v0.1.5 \\\n --astra-token <astra-token> --astra-database-id <astra-datbase-id>docker run -p 9042:9042 \\\n --rm datastax/cql-proxy:v0.1.5 \\\n -e ASTRA_TOKEN=<astra-token> -e ASTRA_DATABASE_ID=<astra-datbase-id>Proxy settings can also be passed using a configuration file with the --config /path/to/proxy.yaml flag. This can be mixed and matched with command-line flags and environment variables. Here are some example configuration files:
contact-points:\n - 127.0.0.1\nusername: cassandra\npassword: cassandra\nport: 9042\nbind: 127.0.0.1:9042\n# ...or with a Astra token:
astra-token: <astra-token>\nastra-database-id: <astra-database-id>\nbind: 127.0.0.1:9042\n# ...All configuration keys match their command-line flag counterpart, e.g. --astra-bundle is astra-bundle:, --contact-points is contact-points: etc.
Multi-region failover with DC-aware load balancing policy is the most useful case for a multiple proxy setup.
When configuring peers: it is required to set --rpc-address (or rpc-address: in the yaml) for each proxy and it must match is corresponding peers: entry. Also, peers: is only available in the configuration file and cannot be set using a command-line flag.
Here's an example of configuring multi-region failover with two proxies. A proxy is started for each region of the cluster connecting to it using that region's bundle. They all share a common configuration file that contains the full list of proxies.
Note: Only bundles are supported for multi-region setups.
cql-proxy --astra-bundle astra-region1-bundle.zip --username token --password <astra-token> \\\n --bind 127.0.0.1:9042 --rpc-address 127.0.0.1 --data-center dc-1 --config proxy.yamlcql-proxy ---astra-bundle astra-region2-bundle.zip --username token --password <astra-token> \\\n --bind 127.0.0.2:9042 --rpc-address 127.0.0.2 --data-center dc-2 --config proxy.yamlThe peers settings are configured using a yaml file. It's a good idea to explicitly provide the --data-center flag, otherwise; these values are pulled from the backend cluster and would need to be pulled from the system.local and system.peers table to properly setup the peers data-center: values. Here's an example proxy.yaml:
peers:\n - rpc-address: 127.0.0.1\n data-center: dc-1\n - rpc-address: 127.0.0.2\n data-center: dc-2Note: It's okay for the peers: to contain entries for the current proxy itself because they'll just be omitted.
There are three methods for using cql-proxy:
cql-proxycql-proxy installedcql-proxyBuild cql-proxy.
go buildRun with your desired database.
\nDataStax Astra cluster:
\n./cql-proxy --astra-token <astra-token> --astra-database-id <astra-database-id>The <astra-token> can be generated using these instructions. The proxy also supports using the Astra Secure Connect Bundle along with a client ID and secret generated using these instructions:
./cql-proxy --astra-bundle <your-secure-connect-zip> \\\n--username <astra-client-id> --password <astra-client-secret>\n
Apache Cassandra cluster:
\n./cql-proxy --contact-points <cluster node IPs or DNS names> [--username <username>] [--password <password>]Run with your desired database.
\nDataStax Astra cluster:
\ndocker run -p 9042:9042 \\\n datastax/cql-proxy:v0.1.5 \\\n --astra-token <astra-token> --astra-database-id <astra-database-id>The <astra-token> can be generated using these instructions. The proxy also supports using the Astra Secure Connect Bundle, but it requires mounting the bundle to a volume in the container:
docker run -v <your-secure-connect-bundle.zip>:/tmp/scb.zip -p 9042:9042 \\\n--rm datastax/cql-proxy:v0.1.5 \\\n--astra-bundle /tmp/scb.zip --username <astra-client-id> --password <astra-client-secret>Apache Cassandra cluster:
\ndocker run -p 9042:9042 \\\n datastax/cql-proxy:v0.1.5 \\\n --contact-points <cluster node IPs or DNS names> [--username <username>] [--password <password>]If you wish to have the docker image removed after you are done with it, add --rm before the image name datastax/cql-proxy:v0.1.5.
Using Kubernetes with cql-proxy requires a number of steps:
Generate a token following the Astra instructions. This step will display your Client ID, Client Secret, and Token; make sure you download the information for the next steps. Store the secure bundle in /tmp/scb.zip to match the example below.
Create cql-proxy.yaml. You'll need to add three sets of information: arguments, volume mounts, and volumes. A full example can be found here.
Argument: Modify the local bundle location, username and password, using the client ID and client secret obtained in the last step to the container argument.
\ncommand: [\"./cql-proxy\"]\nargs: [\"--astra-bundle=/tmp/scb.zip\",\"--username=Client ID\",\"--password=Client Secret\"]\n
Volume mounts: Modify /tmp/ as a volume mount as required.
volumeMounts:\n - name: my-cm-vol\n mountPath: /tmp/\n
Volume: Modify the configMap filename as required. In this example, it is named cql-proxy-configmap. Use the same name for the volumes that you used for the volumeMounts.
volumes:\n - name: my-cm-vol\n configMap:\n name: cql-proxy-configmap \n
Create a configmap. Use the same secure bundle that was specified in the cql-proxy.yaml.
kubectl create configmap cql-proxy-configmap --from-file /tmp/scb.zip Check the configmap that was created.
\nkubectl describe configmap cql-proxy-configmap\n Name: cql-proxy-configmap\n Namespace: default\n Labels: <none>\n Annotations: <none>\n Data\n ====\n BinaryData\n ====\n scb.zip: 12311 bytesCreate a Kubernetes deployment with the YAML file you created:
\nkubectl create -f cql-proxy.yamlCheck the logs:
\nkubectl logs <deployment-name>Drivers that use token-aware load balancing may print a warning or may not work when using cql-proxy. Because cql-proxy abstracts the backend cluster as a single endpoint this doesn't always work well with token-aware drivers that expect there to be at least \"replication factor\" number of nodes in the cluster. Many drivers print a warning (which can be ignored) and fallback to something like round-robin, but other drivers might fail with an error. For the drivers that fail with an error it is required that they disable token-aware or configure the round-robin load balancing policy.
","id":"17fac8a9-8b96-51ec-a7dd-dd3809bba528","title":"GitHub - datastax/cql-proxy: A client-side CQL proxy/sidecar.","origin_url":"https://github.com/datastax/cql-proxy","url":"https://github.com/datastax/cql-proxy","wallabag_created_at":"2024-11-01T17:26:01+00:00","published_at":null,"published_by":"['datastax']","reading_time":8,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/c2528e3426d98910ed27819e048b4c1081fab2ed2c7adbea6e6a3b1872deb30a/datastax/cql-proxy","tags":["migration","proxy","cassandra","cql"],"description":" cql-proxy is designed to forward your application's CQL traffic to an appropriate database service. It listens on a local address and securely forwards that traffic.The cql-proxy sidecar enables unsu..."}]}]}},"staticQueryHashes":[]}