ClickHouse Writer

ClickHouse Writer plugin is used to write data to ClickHouse.

Example

The following example demonstrates reading content from one table in ClickHouse and writing it to another table with the same table structure, to test the data structures supported by the plugin.

Table Structure and Data

Assume the table structure and data to be read are as follows:

CREATE TABLE ck_addax (
    c_int8 Int8,
    c_int16 Int16,
    c_int32 Int32,
    c_int64 Int64,
    c_uint8 UInt8,
    c_uint16 UInt16,
    c_uint32 UInt32,
    c_uint64 UInt64,
    c_float32 Float32,
    c_float64 Float64,
    c_decimal Decimal(38,10),
    c_string String,
    c_fixstr FixedString(36),
    c_uuid UUID,
    c_date Date,
    c_datetime DateTime('Asia/Chongqing'),
    c_datetime64 DateTime64(3, 'Asia/Chongqing'),
    c_enum Enum('hello' = 1, 'world'=2)
) ENGINE = MergeTree() ORDER BY (c_int8, c_int16) SETTINGS index_granularity = 8192;

insert into ck_addax values(
    127,
    -32768,
    2147483647,
    -9223372036854775808,
    255,
    65535,
    4294967295,
    18446744073709551615,
    0.9999998,
    0.999999999999998,
    1234567891234567891234567891.1234567891,
    'Hello String',
    '2c:16:db:a3:3a:4f',
    '5F042A36-5B0C-4F71-ADFD-4DF4FCA1B863',
    '2021-01-01',
    '2021-01-01 11:22:33',
    '2021-01-01 10:33:23.123',
    'hello'
);

The table to be written uses the same structure as the read table, with the following DDL statement:

create table ck_addax_writer as ck_addax;

Configuration

The following is the configuration file

{
  "job": {
    "setting": {
      "speed": {
        "channel": 1
      }
    },
    "content": {
      "writer": {
        "name": "clickhousewriter",
        "parameter": {
          "username": "default",
          "column": [
            "*"
          ],
          "connection": {
            "table": [
              "ck_addax_writer"
            ],
            "jdbcUrl": "jdbc:clickhouse://127.0.0.1:8123/default"
          },
          "preSql": [
            "alter table @table delete where 1=1"
          ]
        }
      },
      "reader": {
        "name": "clickhousereader",
        "parameter": {
          "username": "default",
          "column": [
            "*"
          ],
          "connection": {
            "jdbcUrl": "jdbc:clickhouse://127.0.0.1:8123/",
            "table": [
              "ck_addax"
            ]
          }
        }
      }
    }
  }
}

Save the above configuration file as job/clickhouse2clickhouse.json

Execute Collection Command

Execute the following command for data collection

bin/addax.sh job/clickhouse2clickhouse.json

Parameters

This plugin is based on RDBMS Writer, so you can refer to all configuration items of RDBMS Writer.