News Feed

No results for 'undefined'
Powered by Algolia

Unable to create new Database and Collection at runtime in cosmodb while writing Spark Dataframe to Cosmodb (SQL Api) using Spark-CosmosDB Connector

Author: Rahul Chanda

Originally Sourced from:

I am using Apache Spark 2.3.2v. Below is my use case to perform with CosmosDB using Apache Spark. I am using azure-cosmosdb-spark_2.3.0_2.11_1.3.3 connector.

Use case - I am performing some complex aggregation in spark and my data is in CosmosDB(SQL). I want to read the data from CosmosDB in Apache Spark data-frame and perform some complex aggregations function and write back into CosmosDB storing in new Database and Collection which will create at Runtime.

I am able to read the data from CosmosDB successfully and performed my functions with data-frame. Now i want to write back into CosmosDB but in new Database and new Collection.

Below is my code to read data from CosmosDB -


val readConfig = Config(Map("Endpoint" -> "",
    "Masterkey" -> "",
    "Database" -> "Database1",
    "PreferredRegions" -> "",
    "Collection" -> "Collection1",
    "SamplingRatio" -> "1.0"))

val flights =
    flights.count()   // performed complex aggrgate functions

Below is code to write back to CosmosDB (new Database and Collection) -

val writeConfig = Config(Map(
    "Endpoint" -> "",
    "Masterkey" ->"",
    "Database" -> "Database2",
    "PreferredRegions" -> "",
    "Collection" -> "Collection2",
    "WritingBatchSize" -> "100",
    "Upsert" -> "true"))


When i try to give different database and new Collection name (not created in cosmosDb) it always give me below exception

Exception in thread "main" Message:      {"Errors":["Resource Not Found. Learn more: https:\/\/\/cosmosdb-tsg-not-found"]}

I looked in many links but all are giving the database name and collection name same to read write operation. Please let me know if you need any inputs.

Query -

  1. Can i create a Database and Collection in CosmosDB through Apache spark using Spark-cosmos Db connector?

Please suggest me any solution. Do i need to build an API as similar as it is in .NET SDK