pytd.writer.BulkImportWriter

class pytd.writer.BulkImportWriter

A writer module that loads Python data to Treasure Data by using td-client-python’s bulk importer.

__init__()

Initialize self. See help(type(self)) for accurate signature.

Methods

__init__()	Initialize self.
close()
from_string(writer, **kwargs)
write_dataframe(dataframe, table, if_exists)	Write a given DataFrame to a Treasure Data table.

write_dataframe(dataframe, table, if_exists, fmt='csv')

Write a given DataFrame to a Treasure Data table.

This method internally converts a given pandas.DataFrame into a temporary CSV/msgpack file, and upload the file to Treasure Data via bulk import API.

Parameters

• dataframe (pandas.DataFrame) – Data loaded to a target table.

• table (pytd.table.Table) – Target table.

• if_exists ({'error', 'overwrite', 'append', 'ignore'}) –

  What happens when a target table already exists.

  • error: raise an exception.

  • overwrite: drop it, recreate it, and insert data.

  • append: insert data. Create if does not exist.

  • ignore: do nothing.

• fmt ({'csv', 'msgpack'}, default: 'csv') –

  Format for bulk_import.

  • csv

    Convert dataframe to temporary CSV file. Stable option but slower than msgpack option because pytd saves dataframe as temporary CSV file, then td-client converts it to msgpack.

  • msgpack

    Convert to temporary msgpack.gz file. Fast option but might be unstable because of skipping tdclient.API._prepare_file()