Illustration Image

Cassandra.Link

The best knowledge base on Apache Cassandra®

Helping platform leaders, architects, engineers, and operators build scalable real time data platforms.

12/9/2023

Reading time:6 min

Using Oracle GoldenGate for Big Data

by John Doe

DBOPTIONS ENABLECASSANDRACPPDRIVERTRACE trueOptionalExtract parameter (.prm) file.falseUse only during initial load process.When set to true, the Cassandra driver logs all the API calls to a driver.log file. This file is created in the Oracle GoldenGate for Big Data installation directory. This is useful for debugging.DBOPTIONS FETCHBATCHSIZE numberOptionalExtract parameter (.prm) file.1000Minimum is 1Maximum is 100000Use only during initial load process.Specifies the number of rows of data the driver attempts to fetch on each request submitted to the database server.The parameter value should be lower than the database configuration parameter, tombstone_warn_threshold, in the database configuration file, cassandra.yaml. Otherwise the initial load process might fail.Oracle recommends that you set this parameter value to 5000 for initial load Extract optimum performance.TRANLOGOPTIONS CDCLOGDIRTEMPLATE pathRequiredExtract parameter (.prm) file.NoneThe CDC commit log directory path template. The template can optionally have the $nodeAddress meta field that is resolved to the respective node address.TRANLOGOPTIONS SFTP optionsOptionalExtract parameter (.prm) file.NoneThe secure file transfer protocol (SFTP) connection details to pull and transfer the commit log files. You can use one or more of these options:USER userThe SFTP user name.PASSWORD passwordThe SFTP password.KNOWNHOSTSFILE fileThe location of the Secure Shell (SSH)known hosts file.LANDINGDIR dirThe SFTP landing directory for the commit log files on the local machine.PRIVATEKEY fileThe SSH private key file.PASSPHRASE passwordThe SSH private key pass phrase.PORTNUMBER portnumberThe SSH port number.CLUSTERCONTACTPOINTS nodes USER dbuser PASSWORD dbpasswordOptionalGLOBALS parameter file127.0.0.1A comma separated list of nodes to be used for a connection to the Cassandra cluster. You should provide at least one node address. The parameter options are:USER dbuserNo defaultOptionalThe user name to use when connecting to the database.PASSWORD dbpasswordNo defaultRequired when USER is used.The user password to use when connecting to the database.PORT <port numberNo defaultOptionalThe port to use when connecting to the database.TRANLOGOPTIONS CDCREADERSDKVERSION versionOptionalExtract parameter (.prm) file.3.11The SDK Version for the CDC reader capture API.ABENDONMISSEDRECORD | NOABENDONMISSEDRECORDOptionalExtract parameter (.prm) file.trueWhen set to true and the possibility of a missing record is found, the process stops with the diagnostic information. This is generally detected when a node goes down and the CDC reader doesn't find a replica node with a matching last record from the dead node. You can set this parameter to false to continue processing. A warning message is logged about the scenario.TRANLOGOPTIONS CLEANUPCDCCOMMITLOGSOptionalExtract parameter (.prm) file.falsePurge CDC commit log files post extract processing. When the value is set to false, the CDC commit log files are moved to the cdc_raw_processed directory.JVMBOOTOPTIONS jvm_optionsOptionalGLOBALS parameter fileNoneThe boot options for the Java Virtual Machine. Multiple options are delimited by a space character.JVMCLASSPATH classpathRequiredGLOBALS parameter fileNoneThe classpath for the Java Virtual Machine. You can include an asterisk (*) wildcard to match all JAR files in any directory. Multiple paths should be delimited with a colon (:) character.OGGSOURCE sourceRequiredNoneThe source database for CDC capture or database queries. The valid value is CASSANDRA.SOURCEDB nodeaddress USERID dbuser PASSWORD dbpasswordRequiredExtract parameter (.prm) file.NoneA single Cassandra node address that is used for a connection to the Cassandra cluster and to query the metadata for the captured tables.USER dbuserNo defaultOptionalThe user name to use when connecting to the database.PASSWORD dbpasswordNo defaultRequired when USER is used.The user password to use when connecting to the database.ABENDONUPDATERECORDWITHMISSINGKEYS | NOABENDONUPDATERECORDWITHMISSINGKEYSOptionalExtract parameter (.prm) file.trueIf this value is true, anytime an UPDATE operation record with missing key columns is found, the process stops with the diagnostic information. You can set this property to false to continue processing and write this record to the trail file. A warning message is logged about the scenario. This operation is a partition update, see Partition Update or Insert of Static Columns.ABENDONDELETERECORDWITHMISSINGKEYS | NOABENDONDELETERECORDWITHMISSINGKEYSOptionalExtract parameter (.prm) file.trueIf this value is true, anytime an DELETE operation record with missing key columns is found, the process stops with the diagnostic information. You can set this property to false to continue processing and write this record to the trail file. A warning message is logged about the scenario. This operation is a partition update, see Partition Delete.MOVECOMMITLOGSTOSTAGINGDIR | NOMOVECOMMITLOGSTOSTAGINGDIROptionalExtract parameter (.prm) file.trueEnabled by default and this instructs the Extract group to move the commit log files in the cdc_raw directory on the Cassandra nodes to a staging directory for the commit log files. Only one Extract group can have movecommitlogstostagingdir enabled, and all the other Extract groups disable this by specifying nomovecommitlogstostagingdir.SSLOptionalGLOBALS or Extract parameter (.prm) file.falseUse for basic SSL support during connection. Additional JSSE configuration through Java System properties is expected when enabling this.CPPDRIVEROPTIONS SSL PEMPUBLICKEYFILE cassadra.pemOptionalGLOBALS or Extract parameter (.prm) file.String that indicates the absolute path with fully qualified name. This file is must for the SSL connection.None, unless the PEMPUBLICKEYFILE property is specified, then you must specify a value.Indicates that it is PEM formatted public key file used to verify the peer's certificate. This property is needed for one-way handshake or basic SSL connection.CPPDRIVEROPTIONS SSL ENABLECLIENTAUTH | DISABLECLIENTAUTHOptionalGLOBALS or Extract parameter (.prm) file.falseEnabled indicates a two-way SSL encryption between client and server. It is required to authenticate both the client and the server through PEM formatted certificates. This property also needs the pemclientpublickeyfile and pemclientprivatekeyfile properties to be set. The pemclientprivatekeypasswd property must be configured if the client private key is password protected. Setting this property to false disables client authentication for two-way handshake.CPPDRIVEROPTIONS SSL PEMCLIENTPUBLICKEYFILE public.pemOptionalGLOBALS or Extract parameter (.prm) file.String that indicates the absolute path with fully qualified name. This file is must for the SSL connection.None, unless the PEMCLIENTPUBLICKEYFILE property is specified, then you must specify a value.Use for a PEM formatted public key file name used to verify the client's certificate. This is must if you are using CPPDRIVEROPTIONS SSL ENABLECLIENTAUTH or for two-way handshake.CPPDRIVEROPTIONS SSL PEMCLIENTPRIVATEKEYFILE public.pemOptionalGLOBALS or Extract parameter (.prm) file.String that indicates the absolute path with fully qualified name. This file is must for the SSL connection.None, unless the PEMCLIENTPRIVATEKEYFILE property is specified, then you must specify a value.Use for a PEM formatted private key file name used to verify the client's certificate. This is must if you are using CPPDRIVEROPTIONS SSL ENABLECLIENTAUTH or for two-way handshake.CPPDRIVEROPTIONS SSL PEMCLIENTPRIVATEKEYPASSWD privateKeyPasswdOptionalGLOBALS or Extract parameter (.prm) file.A stringNone, unless the PEMCLIENTPRIVATEKEYPASSWD property is specified, then you must specify a value.Sets the password for the PEM formatted private key file used to verify the client's certificate. This is must if the private key file is protected with the password.CPPDRIVEROPTIONS SSL PEERCERTVERIFICATIONFLAG valueOptionalGLOBALS or Extract parameter (.prm) file.An integer0Sets the verification required on the peer's certificate. The range is 0–4:0–Disable certificate identity verification.1–Verify the peer certificate2–Verify the peer identity3– Not used so it is similar to disable certificate identity verification.4 –Verify the peer identity by its domain nameCPPDRIVEROPTIONS SSL ENABLEREVERSEDNSOptionalGLOBALS or Extract parameter (.prm) file.falseEnables retrieving host name for IP addresses using reverse IP lookup.

Illustration Image

DBOPTIONS ENABLECASSANDRACPPDRIVERTRACE true

Optional

Extract parameter (.prm) file.

false

Use only during initial load process.

When set to true, the Cassandra driver logs all the API calls to a driver.log file. This file is created in the Oracle GoldenGate for Big Data installation directory. This is useful for debugging.

DBOPTIONS FETCHBATCHSIZE number

Optional

Extract parameter (.prm) file.

1000

Minimum is 1

Maximum is 100000

Use only during initial load process.

Specifies the number of rows of data the driver attempts to fetch on each request submitted to the database server.

The parameter value should be lower than the database configuration parameter, tombstone_warn_threshold, in the database configuration file, cassandra.yaml. Otherwise the initial load process might fail.

Oracle recommends that you set this parameter value to 5000 for initial load Extract optimum performance.

TRANLOGOPTIONS CDCLOGDIRTEMPLATE path

Required

Extract parameter (.prm) file.

None

The CDC commit log directory path template. The template can optionally have the $nodeAddress meta field that is resolved to the respective node address.

TRANLOGOPTIONS SFTP options

Optional

Extract parameter (.prm) file.

None

The secure file transfer protocol (SFTP) connection details to pull and transfer the commit log files. You can use one or more of these options:

USER user

The SFTP user name.

PASSWORD password

The SFTP password.

KNOWNHOSTSFILE file

The location of the Secure Shell (SSH)known hosts file.

LANDINGDIR dir

The SFTP landing directory for the commit log files on the local machine.

PRIVATEKEY file

The SSH private key file.

PASSPHRASE password

The SSH private key pass phrase.

PORTNUMBER portnumber

The SSH port number.

CLUSTERCONTACTPOINTS nodes USER dbuser PASSWORD dbpassword

Optional

GLOBALS parameter file

127.0.0.1

A comma separated list of nodes to be used for a connection to the Cassandra cluster. You should provide at least one node address. The parameter options are:

USER dbuser

No default

Optional

The user name to use when connecting to the database.

PASSWORD dbpassword

No default

Required when USER is used.

The user password to use when connecting to the database.

PORT <port number

No default

Optional

The port to use when connecting to the database.

TRANLOGOPTIONS CDCREADERSDKVERSION version

Optional

Extract parameter (.prm) file.

3.11

The SDK Version for the CDC reader capture API.

ABENDONMISSEDRECORD | NOABENDONMISSEDRECORD

Optional

Extract parameter (.prm) file.

true

When set to true and the possibility of a missing record is found, the process stops with the diagnostic information. This is generally detected when a node goes down and the CDC reader doesn't find a replica node with a matching last record from the dead node. You can set this parameter to false to continue processing. A warning message is logged about the scenario.

TRANLOGOPTIONS CLEANUPCDCCOMMITLOGS

Optional

Extract parameter (.prm) file.

false

Purge CDC commit log files post extract processing. When the value is set to false, the CDC commit log files are moved to the cdc_raw_processed directory.

JVMBOOTOPTIONS jvm_options

Optional

GLOBALS parameter file

None

The boot options for the Java Virtual Machine. Multiple options are delimited by a space character.

JVMCLASSPATH classpath

Required

GLOBALS parameter file

None

The classpath for the Java Virtual Machine. You can include an asterisk (*) wildcard to match all JAR files in any directory. Multiple paths should be delimited with a colon (:) character.

OGGSOURCE source

Required

None

The source database for CDC capture or database queries. The valid value is CASSANDRA.

SOURCEDB nodeaddress USERID dbuser PASSWORD dbpassword

Required

Extract parameter (.prm) file.

None

A single Cassandra node address that is used for a connection to the Cassandra cluster and to query the metadata for the captured tables.

USER dbuser

No default

Optional

The user name to use when connecting to the database.

PASSWORD dbpassword

No default

Required when USER is used.

The user password to use when connecting to the database.

ABENDONUPDATERECORDWITHMISSINGKEYS | NOABENDONUPDATERECORDWITHMISSINGKEYS

Optional

Extract parameter (.prm) file.

true

If this value is true, anytime an UPDATE operation record with missing key columns is found, the process stops with the diagnostic information. You can set this property to false to continue processing and write this record to the trail file. A warning message is logged about the scenario. This operation is a partition update, see Partition Update or Insert of Static Columns.

ABENDONDELETERECORDWITHMISSINGKEYS | NOABENDONDELETERECORDWITHMISSINGKEYS

Optional

Extract parameter (.prm) file.

true

If this value is true, anytime an DELETE operation record with missing key columns is found, the process stops with the diagnostic information. You can set this property to false to continue processing and write this record to the trail file. A warning message is logged about the scenario. This operation is a partition update, see Partition Delete.

MOVECOMMITLOGSTOSTAGINGDIR | NOMOVECOMMITLOGSTOSTAGINGDIR

Optional

Extract parameter (.prm) file.

true

Enabled by default and this instructs the Extract group to move the commit log files in the cdc_raw directory on the Cassandra nodes to a staging directory for the commit log files. Only one Extract group can have movecommitlogstostagingdir enabled, and all the other Extract groups disable this by specifying nomovecommitlogstostagingdir.

SSL

Optional

GLOBALS or Extract parameter (.prm) file.

false

Use for basic SSL support during connection. Additional JSSE configuration through Java System properties is expected when enabling this.

CPPDRIVEROPTIONS SSL PEMPUBLICKEYFILE cassadra.pem

Optional

GLOBALS or Extract parameter (.prm) file.

String that indicates the absolute path with fully qualified name. This file is must for the SSL connection.

None, unless the PEMPUBLICKEYFILE property is specified, then you must specify a value.

Indicates that it is PEM formatted public key file used to verify the peer's certificate. This property is needed for one-way handshake or basic SSL connection.

CPPDRIVEROPTIONS SSL ENABLECLIENTAUTH | DISABLECLIENTAUTH

Optional

GLOBALS or Extract parameter (.prm) file.

false

Enabled indicates a two-way SSL encryption between client and server. It is required to authenticate both the client and the server through PEM formatted certificates. This property also needs the pemclientpublickeyfile and pemclientprivatekeyfile properties to be set. The pemclientprivatekeypasswd property must be configured if the client private key is password protected. Setting this property to false disables client authentication for two-way handshake.

CPPDRIVEROPTIONS SSL PEMCLIENTPUBLICKEYFILE public.pem

Optional

GLOBALS or Extract parameter (.prm) file.

String that indicates the absolute path with fully qualified name. This file is must for the SSL connection.

None, unless the PEMCLIENTPUBLICKEYFILE property is specified, then you must specify a value.

Use for a PEM formatted public key file name used to verify the client's certificate. This is must if you are using CPPDRIVEROPTIONS SSL ENABLECLIENTAUTH or for two-way handshake.

CPPDRIVEROPTIONS SSL PEMCLIENTPRIVATEKEYFILE public.pem

Optional

GLOBALS or Extract parameter (.prm) file.

String that indicates the absolute path with fully qualified name. This file is must for the SSL connection.

None, unless the PEMCLIENTPRIVATEKEYFILE property is specified, then you must specify a value.

Use for a PEM formatted private key file name used to verify the client's certificate. This is must if you are using CPPDRIVEROPTIONS SSL ENABLECLIENTAUTH or for two-way handshake.

CPPDRIVEROPTIONS SSL PEMCLIENTPRIVATEKEYPASSWD privateKeyPasswd

Optional

GLOBALS or Extract parameter (.prm) file.

A string

None, unless the PEMCLIENTPRIVATEKEYPASSWD property is specified, then you must specify a value.

Sets the password for the PEM formatted private key file used to verify the client's certificate. This is must if the private key file is protected with the password.

CPPDRIVEROPTIONS SSL PEERCERTVERIFICATIONFLAG value

Optional

GLOBALS or Extract parameter (.prm) file.

An integer

0

Sets the verification required on the peer's certificate. The range is 0–4:

0–Disable certificate identity verification.

1–Verify the peer certificate

2–Verify the peer identity

3– Not used so it is similar to disable certificate identity verification.

4 –Verify the peer identity by its domain name

CPPDRIVEROPTIONS SSL ENABLEREVERSEDNS

Optional

GLOBALS or Extract parameter (.prm) file.

false

Enables retrieving host name for IP addresses using reverse IP lookup.

Related Articles

hive
elasticsearch
cassandra

GitHub - embulk/embulk: Embulk: Pluggable Bulk Data Loader.

embulk

12/1/2023

Checkout Planet Cassandra

Claim Your Free Planet Cassandra Contributor T-shirt!

Make your contribution and score a FREE Planet Cassandra Contributor T-Shirt! 
We value our incredible Cassandra community, and we want to express our gratitude by sending an exclusive Planet Cassandra Contributor T-Shirt you can wear with pride.

Join Our Newsletter!

Sign up below to receive email updates and see what's going on with our company

Explore Related Topics

AllKafkaSparkScyllaSStableKubernetesApiGithubGraphQl

Explore Further

etl