Cassandra Python driver doesn't page large queries
Author: Rafi Adyatma
Originally Sourced from: https://stackoverflow.com/questions/73247050/cassandra-driver-doesnt-page-large-queries-on-python
It is said in the documentation that cassandra-driver does automatic paging when queries are large enough (with default_fetch_size being 5000 rows) and will return
I have tested reading data from my local Cassandra which contains 9999 rows with
SimpleStatement with my own fetch size, but it returned the ResultSet (9999 rows) instead of pages (instance of
PagedResult). Also, I tried to change the
Session.default_fetch_size but it didn't work as well.
Here's my code..
My first attempt: This is the
SimpleStatement code i have made to change the fetch size.
cluster = Cluster() session = cluster.connect(keyspace_name) query = "SELECT * FROM user" statement = SimpleStatement(query, fetch_size=10) rows = list(session.execute(statement)) print(len(rows))
It prints 9999 (all rows), not 10 rows as I already set the
My second attempt: I tried to change the query fetch size by changing session's default fetch size
cluster = Cluster() session = cluster.connect(keyspace_name) session.default_fetch_size = 10 query = "SELECT * FROM user" rows = list(session.execute(query)) print(len(rows))
It also prints 9999 rows instead of 10.
My goal is not to limit the rows from my fetch query, such as
SELECT * FROM user LIMIT 10. What I want is to fetch the rows page by page to avoid overload on memory.
So what actually happened?
Note: I am using Cassandra-Driver 3.25 for Python and using Python3.7
I am sorry if my additional information still doesn't make my question a good one. I never ask any questions before. So...any suggestions are welcome :)