![\"Firebase](\"https://cdn.analyticsvidhya.com/wp-content/uploads/2023/12/image-56.png\")
Multitenant backend server for building web and mobile apps rapidly. The backend for busy developers. (self-hosted or hosted)
507 stars 141 forks Branches Tags Activity
","id":"f4a9e01b-f3e1-5bea-8d32-5dd26fc9bcf3","title":"GitHub - Erudika/para: Multitenant backend server for building web and mobile apps rapidly. The backend for busy developers. (self-hosted or hosted)","origin_url":"https://github.com/Erudika/para","url":"https://github.com/Erudika/para","wallabag_created_at":"2024-01-26T14:53:22+00:00","published_at":null,"published_by":"['']","reading_time":null,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/2204f6ef5bbccb71656d9d43ff9116ca157ebf50d578547bbd1d8678bf3a3232/Erudika/para","tags":["mongo","rest","elasticsearch","cassandra","elastic","lucene","api","dynamo","baas"],"description":"Multitenant backend server for building web and mobile apps rapidly. The backend for busy developers. (self-hosted or hosted)paraio.orgLicenseApache-2.0 license507 stars 141 forks Branches Tags Activi..."},{"content":"In the dynamic realm of contemporary applications, real-time databases are pivotal for maintaining smooth data management and immediate updates. Engineered to handle substantial data volumes, these databases offer instantaneous access to information. This article delves into the top 10 real-time databases set to make an impact in 2024.
Real-time databases are crafted to manage data needing immediate updates and access. Unlike conventional databases that may encounter synchronization delays, real-time databases guarantee swift reflection of data changes across all connected devices or applications. This makes them well-suited for applications with real-time collaboration, messaging, or monitoring needs.
The significance of real-time databases has grown in contemporary applications, driven by the demand for instant data updates and synchronization. From messaging apps to collaborative document editors and real-time analytics dashboards, these databases form the foundation for smooth data management and instantaneous communication. By removing delays in data synchronization, real-time databases not only improve user experience but also empower efficient, data-driven decision-making.
Here’s our list of the top 10 real-time databases to use in 2024.
Firebase Realtime Database stands as a cloud-hosted NoSQL database, empowering developers to store and synchronize data in real time. Its use of a JSON data model adds flexibility and ease to the development process. As an integral component of the Firebase platform, it contributes to a robust toolkit for crafting both web and mobile applications.
A standout feature of Firebase Realtime Database is its real-time synchronization, ensuring instantaneous updates across all connected devices whenever data changes. This guarantees users constantly have the most up-to-date information. Moreover, the database provides offline support, enabling users to access and modify data even without an internet connection. To enhance security, the Firebase Realtime Database incorporates robust security rules, safeguarding sensitive data from unauthorized access.
Firebase Realtime Database finds widespread use in applications that demand real-time updates, including chat apps, collaborative document editors, and real-time dashboards. For instance, a messaging app leveraging the Firebase Realtime Database can promptly deliver messages to all participants, creating a seamless and real-time communication experience.
MongoDB stands out as a favored document-oriented NoSQL database, renowned for its high performance, scalability, and flexibility. Its storage of data in flexible, JSON-like documents simplifies working with and adjusting to evolving data needs. Embraced in modern applications, MongoDB excels in managing large data volumes and supports real-time updates, making it a preferred choice for developers.
![\"MongoDB](\"https://cdn.analyticsvidhya.com/wp-content/uploads/2023/12/image-57.png\")
MongoDB provides an array of features making it well-suited for real-time applications. Its flexible data model allows developers to effortlessly store and retrieve complex data structures. Notably, MongoDB supports horizontal scalability, enabling applications to manage growing data loads without compromising performance. The added support for automatic sharding further enhances scalability and fault tolerance by distributing data across multiple servers.
MongoDB sees diverse applications in real-time scenarios, such as content management systems, e-commerce platforms, and Internet of Things (IoT) applications. An illustrative example is an e-commerce platform leveraging MongoDB to manage real-time inventory updates, guaranteeing customers consistently accurate information on product availability.
Apache Cassandra stands out as a highly scalable and distributed NoSQL database crafted to manage extensive data across multiple commodity servers. It excels in providing high availability and fault tolerance, making it apt for mission-critical applications. Cassandra’s data model is rooted in a distributed hash table, enabling efficient data distribution and replication for optimal performance.
![\"Apache](\"https://cdn.analyticsvidhya.com/wp-content/uploads/2023/12/apache.png\")
Cassandra offers several features that make it a top choice for real-time applications. Its decentralized architecture ensures high availability and fault tolerance, as data is replicated across multiple nodes. It also provides tunable consistency, allowing developers to balance data consistency and performance according to their application’s requirements. Additionally, Cassandra supports linear scalability, enabling applications to handle massive data loads.
Cassandra finds broad usage in applications demanding high scalability and fault tolerance, including real-time analytics, time series data management, and messaging platforms. Consider a real-time analytics platform leveraging Cassandra; it adeptly processes and analyzes substantial data volumes in real-time, delivering valuable insights to users.
Redis stands as an open-source, in-memory data structure store versatile enough to serve as a database, cache, or message broker. Renowned for high performance and low latency, Redis is particularly well-suited for real-time applications. Its support for diverse data structures such as strings, lists, sets, and sorted sets enables flexible data storage and manipulation.
![\"Redis](\"https://cdn.analyticsvidhya.com/wp-content/uploads/2023/12/image-58.png\")
Redis boasts numerous features that contribute to its popularity in real-time applications. With in-memory storage facilitating rapid data access, it proves ideal for low-latency requirements. The inclusion of built-in replication and high availability ensures data durability and fault tolerance. Moreover, Redis supports pub/sub messaging, fostering real-time communication among diverse components within an application.
Redis finds extensive application in a variety of real-time scenarios, encompassing caching, session management, and real-time leaderboards. In the realm of gaming, for example, Redis can be employed to uphold real-time leaderboards, ensuring timely updates to scores and rankings as players advance through the game.
Apache Kafka stands as a distributed streaming platform, adept at processing and storing real-time data streams. Its strengths lie in high throughput, fault tolerance, and scalability, making it well-suited for managing substantial data volumes. At its core, Kafka operates on a publish-subscribe model, with producers disseminating data to topics and consumers subscribing to these topics to access the data.
![\"Apache](\"https://cdn.analyticsvidhya.com/wp-content/uploads/2023/12/image-59.png\")
A standout feature of Apache Kafka is its prowess in managing high-throughput, low-latency data streams. With the capability to process millions of messages per second, it excels in real-time data processing and analytics. Kafka ensures fault tolerance and high availability by replicating data across multiple brokers. Furthermore, it supports stream processing, enabling real-time data transformations and aggregations.
Applications demanding real-time data streaming and processing, like log aggregation, event sourcing, and real-time analytics, find Apache Kafka indispensable. For instance, a Kafka-powered log aggregation system excels at collecting and processing logs from various sources in real time, delivering valuable insights and robust monitoring capabilities.
Amazon DynamoDB, an AWS fully managed NoSQL database service, stands out with high performance, scalability, and availability, catering to real-time application needs. Built on key-value pairs, DynamoDB ensures rapid data access and retrieval.
![\"Amazon](\"https://cdn.analyticsvidhya.com/wp-content/uploads/2023/12/image-60.png\")
DynamoDB offers several features that make it a top choice for real-time applications. Its fully managed nature eliminates the need for database administration, allowing developers to focus on building their applications. DynamoDB also provides automatic scaling, ensuring that applications can handle increasing data loads without manual intervention. Additionally, it supports global replication, enabling data to be replicated across multiple regions for improved availability and disaster recovery.
Amazon DynamoDB is used in a wide range of real-time applications, including user authentication, real-time bidding, and IoT data management. For instance, a real-time bidding platform built on DynamoDB can handle high volumes of bid requests and deliver real-time ad placements to users.
Microsoft Azure Cosmos DB is a globally distributed, multi-model database service provided by Microsoft Azure. It offers high scalability, low latency, and global availability, making it suitable for real-time applications. Cosmos DB supports multiple data models, including key-value, document, graph, and column family, allowing developers to choose the most appropriate model for their application.
![\"Microsoft](\"https://cdn.analyticsvidhya.com/wp-content/uploads/2023/12/image-61.png\")
Cosmos DB offers several features that make it a powerful choice for real-time applications. Its global distribution allows for low-latency access to data from anywhere in the world, ensuring a seamless user experience. Cosmos DB also provides automatic indexing and query optimization, enabling fast and efficient data retrieval. Additionally, it offers comprehensive SLAs (Service Level Agreements) for throughput, latency, and availability.
In diverse real-time applications like content management systems, social media analytics, and personalized recommendations, Microsoft Azure Cosmos DB finds utility. For instance, a Cosmos DB-based social media analytics platform can analyze real-time social media data, uncovering valuable insights into user behavior and trends.
FaunaDB, a globally distributed, serverless, and transactional NoSQL database, caters to modern applications with features like strong consistency, low latency, and automatic scaling. Its data model, rooted in documents and collections, facilitates flexible data storage and retrieval, making it apt for real-time applications.
![\"FaunaDB](\"https://cdn.analyticsvidhya.com/wp-content/uploads/2023/12/image-62.png\")
FaunaDB offers several features that make it a compelling choice for real-time applications. Its strong consistency ensures that data is always up-to-date and reflects the latest changes. FaunaDB also provides automatic scaling, allowing applications to handle increasing data loads without manual intervention. Additionally, it supports ACID (Atomicity, Consistency, Isolation, Durability) transactions, ensuring data integrity and reliability.
FaunaDB finds applications in diverse real-time scenarios, from e-commerce platforms to financial systems and collaborative applications. For example, a collaborative document editor powered by FaunaDB enables multiple users to edit the same document simultaneously, ensuring real-time updates for all participants.
InfluxDB is an open-source, time series database designed for handling high volumes of time-stamped data. It provides high write and query performance, making it suitable for real-time data analytics and monitoring. InfluxDB’s data model is based on measurements, tags, and fields, allowing for efficient storage and retrieval of time series data.
![\"InfluxDB](\"https://cdn.analyticsvidhya.com/wp-content/uploads/2023/12/image-63.png\")
InfluxDB offers several features that make it a top choice for real-time data analytics. Its high write and query performance enables fast ingestion and retrieval of time series data. It also provides downsampling and data retention policies, allowing for efficient storage and management of historical data. Additionally, InfluxDB supports continuous queries and real-time alerting, enabling proactive monitoring and anomaly detection.
InfluxDB is extensively utilized in applications demanding real-time monitoring and analytics, including IoT data management, system monitoring, and sensor data analysis. For instance, an IoT platform leveraging InfluxDB can gather and analyze real-time sensor data, offering valuable insights into device performance and environmental conditions.
TimescaleDB is an open-source, time-series database built on top of PostgreSQL. It combines the scalability and performance of PostgreSQL with the flexibility and ease of use of a time-series database. TimescaleDB’s data model is based on hypertables, which allow for efficient storage and retrieval of time-series data.
![\"TimescaleDB](\"https://cdn.analyticsvidhya.com/wp-content/uploads/2023/12/image-64.png\")
TimescaleDB offers several features that make it a powerful choice for real-time data management. Its hyper table-based data model enables efficient partitioning and compression of time-series data, resulting in improved query performance and storage efficiency. It also provides automatic data retention and continuous aggregates, allowing for efficient management and analysis of historical data. Additionally, TimescaleDB supports distributed queries, enabling horizontal scalability and fault tolerance.
TimescaleDB finds application in diverse real-time scenarios, such as financial analytics, industrial monitoring, and energy management. For example, a financial analytics platform utilizing TimescaleDB can efficiently store and analyze real-time market data, offering valuable insights into market trends and investment opportunities.
In today’s world, we rely on real-time databases to keep our apps updated and in sync instantly. There are great options out there, like Firebase Realtime Database and TimescaleDB, each with its own special features and uses. Whether you’re aiming for real-time collaboration with Firebase or keeping an eye on things with InfluxDB for monitoring, developers have plenty of choices to match their needs. Looking ahead to 2024, these real-time databases will stay vital in fueling the new wave of applications.
","id":"aa480b56-1238-52d4-a8d4-0fe3f03ae8cb","title":"Top 10 Real-Time Databases to Use in 2024","origin_url":"https://www.analyticsvidhya.com/blog/2023/12/top-real-time-databases-to-use/","url":"https://www.analyticsvidhya.com/blog/2023/12/top-real-time-databases-to-use/","wallabag_created_at":"2024-01-05T20:38:59+00:00","published_at":"2023-12-16T02:30:00+00:00","published_by":"['K.sabreena']","reading_time":9,"domain_name":"www.analyticsvidhya.com","preview_picture":"https://cdn.analyticsvidhya.com/wp-content/uploads/2023/12/ector-database.jpg","tags":["mongo","cassandra","kafka","cosmos","dynamo","firebase","influxdb","microsoft.cosmos.db","redis"],"description":"In the dynamic realm of contemporary applications, real-time databases are pivotal for maintaining smooth data management and immediate updates. Engineered to handle substantial data volumes, these da..."},{"content":"\nPrefer video? View this post on YouTube!
\n
As a programmer, it is important to know the limits of any service that you’re using. In some cases, the limits of a particular service may make it unsuitable for the task at hand, such as using Route53 as a database. In other cases, the limit may alter how you structure your solution, such as how the 15 minute limit on Lambda execution time requires you to break down large work into smaller chunks.
In this post, we’ll talk about limits in DynamoDB. While the DynamoDB documentation has a long list of limits for the service, most of these are not going to drastically change how you use DynamoDB. For example, nested attributes can only go to 32 levels of depth, I’ve never found this to be a factor in building my applications.
However, there are a few limits you must understand to model properly in DynamoDB. If you’re not aware of them, you can run into a brick wall. But if you understand them and account for them, you remove the element of surprise once your app hits production.
Those limits are:
Notice how these limits build on each other. The first is about an individual item, whereas the second is about a collection of items that are read together in a single request. Finally, the partition throughput limit is about the number and size of concurrent requests in a single DynamoDB partition.
In the sections below, we’ll walk through each limit, discuss why it exists, and undestand how it affects how you should model in DynamoDB. Finally, we will conclude with some thoughts on DynamoDB’s philosophy of limits and why they’re helping you.
Let’s get started.
The first important limit to know is the item size limit. An individual record in DynamoDB is called an item, and a single DynamoDB item cannot exceed 400KB.
While 400KB is large enough for most normal database operations, it is significantly lower than the other options. MongoDB allows for documents to be 16MB, while Cassandra allows blobs of up to 2GB. And if you really want to get beefy, Postgres allows rows of up to 1.6TB (1600 columns X 1GB max per field)!
So what accounts for this limitation? DynamoDB is pointing you toward how you should model your data in an OLTP database.
Online transaction processing (or OLTP) systems are characterized by large amounts of small operations against a database. They describe most of how you interact with various services — fetch a LinkedIn profile, show my Twitter timeline, or view my Gmail inbox. For these operations, you want to quickly and efficiently filter on specific fields to find the information you want, such as a username or a Tweet ID. OLTP databases often make use of indexes on certain fields to make lookups faster as well as holding recently-accessed data in RAM.
You should have the relevant properties in your database to find the particular record you want. And if the rest of the information for that record is small, you can include that data there as well.
But if you have a large piece of data associated with your record, such as an image file, some user-submitted prose, or just a giant blob of JSON, it might not be best to store that directly in your database. You’ll clog up the RAM and churn your disk I/O as you read and write that blob.
Put the blob in an object store instead. Amazon S3 is a cheap, reliable way to store blobs of data. Your database record can include a pointer to the object in S3, and you can load it out when it’s needed. S3 has a better pricing model for reading and writing large blobs of data, and it won’t put extra strain on your database.
In addition to making you consider the price & performance implications of large blobs, the 400KB item limit also prevents you from making a data modeling mistake.
When modeling data in DynamoDB, you often denormalize your data. For example, in modeling one-to-many relationships, you may denormalize a collection of child records onto its parent item rather than splitting each child record out into a separate item. This can work well when the number of related items is bounded.
However, if the number of related items is unbounded (think of a customer’s orders in an e-commerce store, or a user’s tweets in Twitter), then you would want to consider other approaches to model this relationship. Without the item size limit, the access patterns around fetching the parent item would get slower and slower as the size of the denormalized relation grew. By imposing this limit, it forces you to consider the nature of the relationship upfront and use a different method if the relationship is unbounded.
In both of these situations, DynamoDB is saving you from reduced performance as your item size grows.
While the first limit we discussed involved an individual item, the second limit involves a grouping of items.
DynamoDB has two APIs for fetching a range of items in a single request. The Query operation will fetch a range of items that have the same partition key, whereas the Scan operation will fetch a range of items from your entire table.
For both of these operations, there is a 1MB limit on the size of an individual request. If your Query parameters match more than 1MB of data or if you issue a Scan operation on a table that’s larger than 1MB, your request will return the initial matching items plus a LastEvaluatedKey property that can be used in the next request to read the next page.
Similar to the second reason for the item size limit, this page size limit forces you to reckon with the performance implications of your data model upfront. Paginated queries are one of the two areas you might see performance problems as your application grows. If you have an access pattern that uses Query and could result in more than 1MB of data being looked at, you need to account for pagination in your application code.
Accounting for pagination (by handling the LastEvaluatedKey and making follow-up requests) has a great property — it’s very easy to see understand which areas might slow down as your application grows.
If I’m doing a PR review and I see LastEvaluatedKey being used, my antenna goes up and I know I need to think about implications:
The page size also fits in well with DynamoDB Filter Expressions (which you probably don’t want to use). The 1MB page limit applies before the Filter Expression is applied. DynamoDB is teaching us what the expensive parts of an operation are — finding and reading lots of data from disk.
Contrast this with other database engines. If you’re using any aggregate functions in Postgres or MongoDB’s aggregation pipeline, the code to execute the query can look deceptively simple. But that code is hiding some unknown performance implications under the hood. Aggregations require scanning all of the relevant data. This may be performant when we’re talking about hundreds of rows in your test environment, but it will crawl when you hit millions of rows at scale.
We now know how big a single item can be. We know how much data a single request can read. Let’s close our discussion of limits with the partition throughput limits, which address how much data concurrent requests can handle.
To begin, you need to know some basics around how DynamoDB models its data.
A DynamoDB table isn’t running on some giant supercomputer in the cloud. Rather, your data will be split across multiple partitions. Each partition contains roughly 10GB of data.
![\"DynamoDB](\"https://user-images.githubusercontent.com/6509926/113156226-690bc880-91ff-11eb-89d3-54810d6808d2.png\")
Each item in your DynamoDB table will contain a primary key that includes a partition key. This partition key determines the partition on which that item will live. This allows for DynamoDB to provide fast, consistent performance as your application scales.
![\"DynamoDB](\"https://user-images.githubusercontent.com/6509926/113156304-7923a800-91ff-11eb-9418-d030679f264d.png\")
For additional background on partitions and how they help DynamoDB scale, check out Why NoSQL databases can scale horizontally.
For an individual partition, there are limits on the maximum throughput you can consume on a per-second basis. You can use up to 3,000 Read Capacity Units (RCUs) and up to 1,000 Write Capacity Units (WCUs) on a single partition per second.
Note — this is a lot of capacity! This would allow you to read 12MB of strongly-consistent data or 24MB of eventually-consistent data per second, as well as to write 1MB of data per second. And remember — this is per-partition! You can have significantly higher traffic across your table as a whole, and the maximum amount of capacity is effectively unbounded, assuming your credit card is good for it. For many DynamoDB users, they’ll never come close to these partitions limits on a full-table basis.
And yet, some users will blow past 3000 RCUs and 1000 WCUs on a table. And this limit helps to guide their data modeling. As you model a high-traffic table, you consider whether you will have partitions that exceed this limit. If so, you can model for it explicitly, whether using DAX to cache hot items or by sharding your partition key to spread the load across your table.
One final note in this area: the DynamoDB team has done a lot of work in the past few years to ensure that the partition throughput limits are the only thing you need to consider when thinking about traffic across your table.
In the old days of DynamoDB (pre-2018), DynamoDB would spread your provisioned throughput equally across all partitions in your table. This could lead to issues if your workload was unbalanced. Imagine you had certain keys that took more traffic than others — think of the distribution of popular tweets or Reddit threads. Because the throughput is spread equally, you could be getting throttled in one partition while you had capacity to burn in others.
This led to DynamoDB users needing to overprovision their tables to account for their hottest partitions rather than for overall traffic. By extension, you needed to care about two things: the partition throughout limit as well as your hottest partitions.
DynamoDB changed that with adaptive capacity. Adaptive capacity works by spreading your provisioned throughput across your table according to demand rather than spreading it evenly. If you have one partition taking more traffic, it will take capacity from one of your other partitions.
Initially, adaptive capacity would take 5 - 30 minutes to spread capacity across your table. In May 2019, adaptive capacity became instant, meaning you could have highly variable workloads from one second to another, and DynamoDB would spread your capacity accordingly.
There are two major implications of adaptive capacity. First, for all users, this meant less that you needed to think about. You don’t need to do the math to calculate the number of partitions in your table as well as the required throughput to give enough capacity to your hottest partition. You only need to consider overall throughput when doing capacity planning.
Second, this opened up some data modeling patterns for tables with lower traffic. Generally, you want to spread your data evenly across your partitions. However, this can make it harder to do queries that look at your entire table, such as finding all entities of a particular type or finding the top N items in the dataset.
If you won’t go anywhere near the partition throughput limits, you can create large partitions that handle these access patterns. You need to be absolutely sure you won’t hit them — otherwise you’re in for a nasty surprise at some point — but it can make it easier to handle these patterns.
In the sections above, we’ve seen how these key limits are designed to point you in the proper direction for modeling your data. But they’re doing more than that — they’re reducing the complexity of understanding your database performance.
With traditional databases, your performance is on a spectrum. The response time for a query can vary significantly based on a number of factors — size of the dataset, the hardware running your database (including CPU, RAM, disk, and network bandwidth), and the other queries running at the same time.
This variability makes it very difficult to plan how your application will perform as it scales. Load tests that simulate actual traffic are difficult, particularly for a new service, so you may just roll something out there and pray for the best.
With DynamoDB, performance isn’t a spectrum. It’s binary — yes or no. You know the exact limits where your access patterns won’t work anymore, whether it’s due to pagination on your Query or throttling on your hot keys.
These limits surely aren’t the absolute maximum that DynamoDB can handle. We know that DynamoDB could handle items over 400KB — after all, they’re handling 1MB of data in Query & Scan operations!
And we know that DynamoDB could handle more than 1MB on a Query or Scan operation, as the BatchGetItem operation can return up to 16MB of data.
Finally, I’m sure DynamoDB could go beyond 3000 RCUs or 1000 WCUs on a single partition if it really wanted to push it.
But the key is that DynamoDB is drawing a line and guaranteeing performance everywhere before that line.
Your database has physical limits. If you don’t know what they are, you’re flirting with danger. I’ve personally experienced issues with Elasticsearch under high load where everything is going swimingly, then the entire database falls over suddenly and without warning. In Postgres, you might lock up your database with multi-transactions. And don’t get me started with MongoDB’s aggregation framework.
The more I work with other databases, the more I prefer the explicit limits of DynamoDB as compared to the unstated limits of other tools.
In this post, we covered the three limits that matter in DynamoDB. First, we saw the limit of an individual item. Next, we looked at the page size limit for Query and Scan operations. Then, we saw how the partition throughput limit regulates concurrent access to certain items.
Finally, we closed with some thoughts on how the DynamoDB limits are helping you. With most databases, performance is on a spectrum with a number of often-unknowable factors combining to determine your response time. With DynamoDB, performance is binary — as long as you fit within the limits, your performance is knowable.
If you have questions or comments on this piece, feel free to leave a note below or email me directly.
","id":"6da8a02c-bee9-5e44-b79a-7b24d537e387","title":"The Three DynamoDB Limits You Need to Know","origin_url":"https://www.alexdebrie.com/posts/dynamodb-limits/","url":"https://www.alexdebrie.com/posts/dynamodb-limits/","wallabag_created_at":"2023-02-15T20:26:32+00:00","published_at":null,"published_by":null,"reading_time":12,"domain_name":"www.alexdebrie.com","preview_picture":"https://user-images.githubusercontent.com/6509926/53833786-0166ee80-3f4f-11e9-8edf-0fd168b04c82.png","tags":["scylladb","cassandra","dynamo"],"description":"\nPrefer video? View this post on YouTube!\nAs a programmer, it is important to know the limits of any service that you’re using. In some cases, the limits of a particular service may make it unsuitable..."},{"content":"To fulfill my capstone project requirement for the Carnegie Mellon University (CMU) Master of Computational Data Science (MCDS) program, I and two of my classmates, Ziyan Zhang and Xiang Yue, collaborated with DataStax to develop a new module in the Stargate system to bring Amazon DynamoDB compatibility to Apache Cassandra.Ⓡ I will introduce our journey in two blog posts. Here in Part 1, I will provide some background on the project and describe the overall design of our system. In Part 2, I will discuss some interesting challenges we encountered and how we solved them.
![\"DynamoDB](\"https://stargate.io/assets/images/towards-dynamodb-compatibility-for-cassandra/stargate-dynamodb-compatibility.png\")
Cassandra and DynamoDB are two popular NoSQL databases inspired by Google’s BigTable and Amazon’s Dynamo papers. They have many similarities but I think it is more useful to look at some of their biggest differences:
There are many more differences, but the first two illustrate why it might be a good idea to use Cassandra instead of DynamoDB, and the last two point to the potential difficulties in switching to Cassandra if you decide to use DynamoDB and later regret that choice. For users that are already using DynamoDB or already have expertise in using DynamoDB, switching from DynamoDB to Cassandra might be too costly.
It is difficult for users that are already using and/or are familiar with DynamoDB to switch to Cassandra, so why don’t we make Cassandra compatible with DynamoDB? Wouldn’t it be nice for users to be able to switch from DynamoDB to Cassandra without having to change a single line of their existing codebase if they want to? Bingo! That’s what our project is for. Basically, we leverage a third framework called “Stargate” to build a middleware for Cassandra that is compatible with DynamoDB.
Stargate is an open source data gateway that sits between your app and your databases.
Stargate is an open-source middleware that sits on top of a database, e.g. Apache Cassandra. It abstracts Cassandra-specific concepts entirely from app developers and supports different API options, removing barriers of entry for new software developers. Right now, Stargate supports REST API, Document API, gRPC API, and GraphQL API. These different API options are pluggable and can be installed when needed.
The figure below shows the Stargate (v2) architecture. As described in this blog post, Stargate (v2) is highly modular. There are already many services that provide different kinds of APIs. Our goal was to create a new service that provides a DynamoDB API. We wanted this API to be able to understand DynamoDB queries and transform them into Cassandra queries and for users to be able to continue using their existing DynamoDB client code to interact seamlessly with Cassandra.
![\"DynamoDB](\"https://stargate.io/assets/images/towards-dynamodb-compatibility-for-cassandra/stargate-v2-dynamodb.png\")
We didn’t want users to have to change a single line of code when switching to Cassandra. But wait a minute…how is that ever possible given Cassandra and DynamoDB have different client libraries? The answer is simple: DynamoDB clients talk to DynamoDB servers in HTTP protocol.
By implementing a web service on top of Cassandra that behaves in the same way as the DynamoDB server, DynamoDB clients could continue to work without knowing it is actually talking to Cassandra. We implemented such a service as a new module in Stargate - the Dynamo API Service. A typical workflow is shown in the following diagram.
![\"Sequence](\"https://stargate.io/assets/images/towards-dynamodb-compatibility-for-cassandra/stargate-dynamodb-query.png\")
The sequence diagram above shows the workflow for DynamoDB PutItem API. Let’s ignore the first component AuthResource for now — all we need to know is that it helps with authentication. By using the DynamoDB client to put an item into the database, the client would send an HTTP request to the configured DynamoDB server endpoint. To use our system, users just need to change their endpoint from AWS to our service. It’s just one line of configuration change! Then everything is handled by our service and Cassandra.
Let’s get back to the sequence diagram. After receiving an HTTP request, DynamoResource, our REST API controller, will recognize the type of request, deserialize the parameters, and then dispatch them to an appropriate Proxy class, in this case, ItemProxy. The ItemProxy component takes the main responsibility of handling the request. Specifically, it needs to parse the request and transform it into a Stargate intermediate representation. You may ask, why an intermediate representation and not a Cassandra query directly? As we said earlier, Stargate is a middleware that sits on top of your database. Although the database we are using is Cassandra, it could be any other database as long as Stargate supports it.
By transforming DynamoDB request into Stargate intermediate representation, we leverage the Cassandra adapter that is already implemented by Stargate. After transforming the request to Stargate intermediate representation, ItemProxy sends it to the Stargate coordinator via StargateBridgeClient which is essentially a gRPC client. The Stargate coordinator then talks to Cassandra nodes and returns the results back.
Note that some requests are straightforward and only need one round-trip between Proxy — Stargate Coordinator — Cassandra cluster. Other requests are more complicated and may need multiple roundtrips. In the PutItem example shown in the sequence diagram, three round trips, at most, are needed. How come we need three round trips for a single write operation? That’s due to the schema difference between Cassandra and DynamoDB. Remember that in the beginning, we said DynamoDB is schemaless while Cassandra is not? That means you could insert an item to DynamoDB with new columns without pre-defining the schema (actually you cannot define a schema in DynamoDB), while you couldn’t do the same for Cassandra.
In Cassandra, if a write operation contains columns that are unknown, the request fails. Therefore, ItemProxy needs to first check whether the schema needs to be updated and if so, it must update the schema first before actually persisting the data. This sounds very slow, isn’t it? Luckily, Stargate has a caching mechanism, and most of the time, the schema will be cached and the overhead is small unless new columns appear frequently. \\
Now that you understand our motivation and the basic design for our service, stay tuned for our next blog in which I will talk about some interesting challenges we encountered and how we solved them.
Special thanks to all the members of the Stargate community who supported this effort including Prabhat Jha, Sebastian Estevez, Tatu Saloranta, and Jeff Carpenter.
","id":"acf50459-98a8-5747-8100-7948e6cb00ad","title":"Stargate: Towards DynamoDB Compatibility for Cassandra (Part 1)","origin_url":"https://stargate.io/2022/05/23/towards-dynamodb-compatibility-for-cassandra.html","url":"https://stargate.io/2022/05/23/towards-dynamodb-compatibility-for-cassandra.html","wallabag_created_at":"2022-06-10T19:56:31+00:00","published_at":"2022-05-23T06:00:00+00:00","published_by":null,"reading_time":6,"domain_name":"stargate.io","preview_picture":"https://stargate.io/assets/images/default-stargate.png","tags":["cassandra","dynamo","stargate"],"description":"To fulfill my capstone project requirement for the Carnegie Mellon University (CMU) Master of Computational Data Science (MCDS) program, I and two of my classmates, Ziyan Zhang and Xiang Yue, collabor..."},{"content":"In my previous blog post, I introduced the Dynamo API Service, which is our project to make Apache CassandraⓇ compatible with Amazon DynamoDB by building a middleware. In this blog post, I will share a few interesting challenges we encountered and how we tackled them.
This is the first problem we encountered. InDynamoDB (and many other AWS products), there are multiple ways to authenticate, among which a common way is to set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environmental variables.
Usually, you don’t need to worry about authentication after you have your Dynamo DB credentials set up. Stargate, on the other hand, requires you to provide a token in every request you make. We could let users follow the authentication guide and fetch their token either manually or programmatically, but the problem was how to make the DynamoDB client aware of this token and carry the token every time it makes an HTTP request. Of course, we could rewrite the DynamoDB client by ourselves, but we would like to avoid that if possible so that users don’t need to change their client library.
Luckily, we found a trick to tackle this problem. We found out that the DynamoDB client always puts an authorization header that contains an unencrypted AWS_ACCESS_KEY_ID in every HTTP request it makes. This makes sense because a DynamoDB client has to use the HTTP protocol to authenticate itself with the DynamoDB server. Now, in the Dynamo API Service for Stargate, we can easily read the token from this AWS_ACCESS_KEY_ID field. Problem solved. All the user needs to do is to put the Stargate token into their AWS_ACCESS_KEY_ID environment variable, and then they don’t need to worry about authentication while making requests!
Cassandra and DynamoDB have similar data models but they are not exactly the same. There is one-to-one mapping for basic types but not for the map, list, and set data types in DynamoDB. In DynamoDB, maps, lists, and sets can be deeply nested, meaning that you can have a list of maps of sets or even more complicated data structures. For example, you could insert an item whose goods attribute contains:
[\n {\n \"name\": \"apple\",\n \"price\": 4,\n \"images\": [\n \"apple1.jpg\",\n \"apple2.jpg\"\n ]\n },\n {\n \"name\": \"orange\",\n \"price\": 1.45,\n \"images\": \"orange.jpg\"\n }\n]\nAs we can see, the above data structure is a list of maps. Furthermore, the first map in the list contains images as a list, while the second map in the list contains images as a string. This heterogeneity might not be common in the real world but it is undoubtedly allowed as DynamoDB is schemaless. Cassandra, despite its support for nested collections, enforces a schema. For example, if you create a column goods with:
goods list<frozen<map<text,text>>\n
Then everything in the inner map must be of text datatype. The Cassandra native support fails this use case.
We don’t really know how nested collections are stored in DynamoDB (recall it’s not open-sourced!). But one (good) thing we know is that you can only create indices for top-level attributes with basic data types. In the previous example, you cannot index the name field because it’s a nested attribute under goods which is a top-level attribute. What does this mean for us? This means we can treat the whole nested collection as a BLOB (binary format) without sacrificing the ability to index.
In all, what we do is quite simple: whenever the user writes a collection data entry, we serialize it into a sequence of bytes and store it in Cassandra. Whenever the user needs to read it, we deserialize the collection from the sequence of bytes stored in Cassandra. And it works just fine! Right now we are using the Kryo library for serialization and deserialization, but we might write our own methods for better performance in the future.
If you are building a database, then you almost certainly need to write parsers to parse the queries. DynamoDB, at the first glance, seemed to be an exception because it uses JSON as a request payload — there are so many JSON libraries that can help us with the parsing (deserialization). This seems to suggest we don’t need to worry about writing parsers. Unfortunately, this is not actually the case. DynamoDB queries have fields like FilterExpression that allow users to define certain conditions in plain text format. For example, in a query, you can have a filter expression like the following:
(Debt = :debt OR Deposit <> :deposit) AND Sex = :s\n
In this example, the results are filtered out if they don’t satisfy the above expression. This FilterExpression supports different comparison operators and can be nested. You might think about using a regular expression to match the above text, but it is not feasible because REGEX generally cannot handle nested expressions. We actually tried writing regular expressions for simpler cases, but the code quickly turned obscure and we had to give up.
This is where ANTLR comes into play. ANTLR is a popular and powerful parser generator. By simply defining grammars, ANTLR helps us generate Java code that can parse the expressions into abstract parse trees. We then write code that visits the abstract parse tree and evaluates the expression. With the help of ANTLR, we keep our code concise and easy to maintain. It might be a bit challenging at the beginning to write clean grammar if you are not familiar with compiler knowledge, but it pays off!
This concludes our journey toward completion of our capstone project for our Spring semester in the Master of Computational Data Science (MCDS) program at Carnegie Mellon University (CMU). I would like to thank DataStax for the help and guidance throughout this journey. We will continue working on the project in the Fall semester to complete the rest of the APIs and do a thorough performance benchmark. Our hope is to deliver a complete product in the end!
Special thanks to all the members of the Stargate community who supported this effort including Prabhat Jha, Sebastian Estevez, Tatu Saloranta, and Jeff Carpenter.
","id":"20c49ad3-5420-53e4-bee1-a36b8e6a6103","title":"Stargate: Towards DynamoDB Compatibility for Cassandra (Part 2)","origin_url":"https://stargate.io/2022/05/31/towards-dynamodb-compatibility-for-cassandra-part-2.html","url":"https://stargate.io/2022/05/31/towards-dynamodb-compatibility-for-cassandra-part-2.html","wallabag_created_at":"2022-06-10T19:56:18+00:00","published_at":"2022-05-31T06:00:00+00:00","published_by":null,"reading_time":5,"domain_name":"stargate.io","preview_picture":"https://stargate.io/assets/images/default-stargate.png","tags":["cassandra","dynamo","stargate"],"description":"In my previous blog post, I introduced the Dynamo API Service, which is our project to make Apache CassandraⓇ compatible with Amazon DynamoDB by building a middleware. In this blog post, I will share ..."},{"content":"\n\t\t\t\t \n\t\t\t\tPost Views: \n\t\t\t\t2,532
Dear Readers,
\n![\"\"](\"https://www.ktexperts.com/wp-content/uploads/2016/10/1600-x-400-Cassandra-–-Types-of-NoSQL-Databases.jpg\")
There are four general types (most common categories) of NoSQL databases. Each of these categories has its own specific attributes and limitations. There is not a single solution which is better than all others; however there are some databases that are better to solve specific problems. To clarify the NoSQL databases, let’s discuss the most common categories:
\nWide Row Store:
\nAlso known as wide-column stores, these databases store data in rows and users are able to perform some query operations via column-based access. A wide-row store offers very high performance and a highly scalable architecture.
\nExamples: Cassandra, HBase, and Google BigTable.
\nKey-value stores:
\nThese NoSQL databases are some of the least complex as all of the data consists of an indexed key and a value. Designed to handle huge amounts of data, they allow developers to store schema less data. In the key-value storage, database stores data as hash table where each key is unique and the value can be string, JSON, BLOB (basic large object) etc. and a key may be strings, hashes, lists, sets, sorted sets.
\nExamples: Amazon DynamoDB, Riak, and Oracle NoSQL database.
\nDocument oriented:
\nExpands on the basic idea of key-value stores where “documents” are more complex, in that they contain data and each document is assigned a unique key, which is used to retrieve the document. These are designed for storing, retrieving, and managing document-oriented information, also known as semi-structured data.
\nExamples: MongoDB and CouchDB.
\nGraph:
\nDesigned for data whose relationships are well represented as a graph structure and has elements that are interconnected; with an undetermined number of relationships between them.
\nA graph database is a collection of nodes and edges. Each node represents an entity (such as a student or business) and each edge represents a connection or relationship between two nodes. Every node and edge is defined by a unique identifier. Each node knows its adjacent nodes. As the number of nodes increases, the cost of a local step (or hop) remains the same.
\nExamples: Neo4J and TitanDB.
\nThank you for giving your valuable time to read the above information. Please click here to subscribe for further updates
\nKTEXPERTS is always active on below social media platforms.
\nFacebook : https://www.facebook.com/ktexperts/
LinkedIn : https://www.linkedin.com/company/ktexperts/
Twitter : https://twitter.com/ktexpertsadmin
YouTube : https://www.youtube.com/c/ktexperts
Note: Please test scripts in Non Prod before trying in Production.
![\"Compare](\"https://docs.yugabyte.com/images/section_icons/index/comparisons.png\" "\\"Compare"){kind=icon}
\n >
See how YugabyteDB compares with other operational databases in the distributed SQL and NoSQL categories. For a detailed comparison, click the database name.
| Feature | \nCockroachDB | \nTiDB | \nVitess | \nAmazon Aurora | \nGoogle Cloud Spanner | \nYugabyteDB | \n
|---|---|---|---|---|---|---|
| Horizontal write scalability (with auto-sharding and rebalancing) | \n\n | \n | \n | \n | \n | \n |
| Automated failover & repair | \n\n | \n | \n | \n | \n | \n |
| Distributed ACID transactions | \n\n | \n | \n | \n | \n | \n |
| SQL Foreign Keys | \n\n | \n | \n | \n | \n | \n |
| SQL Joins | \n\n | \n | \n | \n | \n | \n |
| Serializable isolation level | \n\n | \n | \n | \n | \n | \n |
| Global consistency across multi-DC/regions | \n\n | \n | \n | \n | \n | \n |
| Follower reads | \n\n | \n | \n | \n | \n | \n |
| Built-in enterprise features (such as CDC) | \n\n | \n | \n | \n | \n | \n |
| SQL compatibility | \nPostgreSQL | \nMySQL | \nMySQL | \nMySQL, PostgreSQL | \nProprietary | \nPostgreSQL | \n
| Open Source | \n\n | Apache 2.0 | \nApache 2.0 | \n\n | \n | Apache 2.0 | \n
| Feature | \nMongoDB | \nFoundationDB | \nApache Cassandra | \nAmazon DynamoDB | \nMS Azure CosmosDB | \nYugabyteDB | \n
|---|---|---|---|---|---|---|
| Horizontal write scalability (with auto-sharding and rebalancing) | \n\n | \n | \n | \n | \n | \n |
| Automated failover & repair | \n\n | \n | \n | \n | \n | \n |
| Distributed ACID transactions | \n\n | \n | \n | \n | \n | \n |
| Consensus-driven, strongly-consistent replication | \n\n | \n | \n | \n | \n | \n |
| Strongly-consistent secondary indexes | \n\n | \n | \n | \n | \n | \n |
| Multiple read consistency levels | \n\n | \n | \n | \n | \n | \n |
| High data density | \n\n | \n | \n | \n | \n | \n |
| API | \nMongoDB QL | \nProprietary KV, MongoDB QL | \nCassandra QL | \nProprietary KV, Document | \nCassandra QL, MongoDB QL | \nYugabyte Cloud QL w/ native document modeling | \n
| Open Source | \n\n | Apache 2.0 | \nApache 2.0 | \n\n | \n | Apache 2.0 | \n
Note
Theorwith respect to any particular feature of a third-party database is based on our best effort understanding from publicly available information. Readers are always recommended to perform their own independent research to understand the finer details.![\"Cortex](\"https://github.com/cortexproject/cortex/blob/master/images/logo.png\"){kind=logo}
![\"CI\"](\"https://github.com/cortexproject/cortex/workflows/ci/badge.svg\"){kind=icon}
\n![\"GoDoc\"](\"https://godoc.org/github.com/cortexproject/cortex?status.svg\")
\n
Cortex provides horizontally scalable, highly available, multi-tenant, long term storage for\nPrometheus.
Cortex is a CNCF incubation project used in several production systems including Weave Cloud and Grafana Cloud.\nCortex is primarily used as a remote write destination for Prometheus, with a Prometheus-compatible query API.
Read the getting started guide if you're new to the\nproject. Before deploying Cortex with a permanent storage backend you\nshould read:
For a guide to contributing to Cortex, see the contributor guidelines.
To learn more about Cortex, consult the following talks and articles.
If you have any questions about Cortex:
Your feedback is always welcome.
For security issues see https://github.com/cortexproject/cortex/security/policy
The Cortex community call happens every three weeks on Thursday at 03:30pm – 04:15pm UTC to get calendar invite join the google groups.
Meeting notes are held here.
There are several commercial services where you can use Cortex\non-demand:
Weave Cloud from\nWeaveworks lets you deploy, manage, and monitor\ncontainer-based applications. Sign up at https://cloud.weave.works\nand follow the instructions there. Additional help can also be found\nin the Weave Cloud documentation.
Instrumenting Your App: Best Practices
To use Cortex as part of Grafana Cloud, sign up for Grafana Cloud\nby clicking \"Log In\" in the top right and then \"Sign Up Now\". Cortex is included\nas part of the Starter and Basic Hosted Grafana plans.
Amazon Managed Service for Prometheus (AMP) is a Prometheus-compatible monitoring service that makes it easy to monitor containerized applications at scale. It is a highly available, secure, and managed monitoring for your containers. Get started here. To learn more about the AMP, reference our documentation and Getting Started with AMP blog.
","id":"d4e68bc0-2b5a-53d7-a0ac-5bdf8e85e0b5","title":"cortexproject/cortex","origin_url":"https://github.com/cortexproject/cortex?spm=a2c65.11461447.0.0.748c6562AJ9SeR","url":"https://github.com/cortexproject/cortex?spm=a2c65.11461447.0.0.748c6562AJ9SeR","wallabag_created_at":"2021-02-03T22:57:13+00:00","published_at":null,"published_by":"['']","reading_time":4,"domain_name":"github.com","preview_picture":"https://opengraph.githubassets.com/98892693112b50cb330347db170b6a71c33e51f77a3fcc4751e8533b0e7e8837/cortexproject/cortex","tags":["aws.s3","bigtable","dynamo","prometheus","cassandra"],"description":"\n\n\nCortex provides horizontally scalable, highly available, multi-tenant, long term storage for\nPrometheus.Horizontally scalable: Cortex can run across multiple machines in a cluster, exceeding the th..."}],"skip":0,"limit":8,"totalCount":19,"currentPage":1,"url":"/tags/dynamo"}},"staticQueryHashes":[]}![\"Slack\"](\"https://img.shields.io/badge/join%20slack-%23cortex-brightgreen.svg\"){kind=icon}