SQOOP-IMPORT(1)

NAME
sqoop-import - Import a table from an RDBMS to HDFS

SYNOPSIS

sqoop-import <generic-options> <tool-options>

sqoop import <generic-options> <tool-options>

DESCRIPTION

The import tool imports an individual table from an RDBMS to HDFS. Each row from a table is represented as a separate record in HDFS. Records can be stored as text files (one record per line), or in binary representation as Avro or SequenceFiles.

OPTIONS

The --connect and --table options are required.

  Database connection and common options

--connect (jdbc-uri)
    Specify JDBC connect string (required)

--connection-manager (class-name)
    Specify connection manager class name (optional)

--driver (class-name)
    Manually specify JDBC driver class to use

--connection-param-file (filename)
    Optional properties file that provides connection parameters

--hadoop-home (dir)
    Override $HADOOP_HOME

--help
    Print usage instructions

--password (password)
    Set authentication password (Note: This is very insecure. You should use -P instead.)

-P
    Prompt for user password

--username (username)
    Set authentication username

--verbose
    Print more information while working

  Import control options

--append
    Append data to an existing HDFS dataset

--as-avrodatafile
    Imports data to Avro Data Files

--as-sequencefile
    Imports data to SequenceFiles

--as-textfile
    Imports data as plain text (default)

--boundary-query (query)
    Using following query to select minimal and maximal value of --split-by column for creating splits

--columns (col,col,col...)
    Columns to export from table

--direct
    Use direct import fast path (MySQL and PostgreSQL)

--direct-split-size (n)
    Split the input stream every n bytes when importing in direct mode.

--inline-lob-limit (n)
    Set the maximum size for an inline LOB

--num-mappers (n), -m
    Use n map tasks to import in parallel

--query (statement)
    Imports the results of statement instead of a table

--split-by (column-name)
    Column of the table used to split the table for parallel import

--table (table-name)
    The table to import

--target-dir (dir)
    Explicit HDFS target directory for the import.

--warehouse-dir (dir)
    Tables are uploaded to the HDFS path /warehouse/dir/(tablename)/

--where (clause)
    Import only the rows for which clause is true. e.g.: --where "user_id > 400 AND hidden == 0"

--compress, -z
    Uses gzip (or, alternatively, the codec specified by --compression-codec, if set) to compress data as it is written to HDFS

--compression-codec (codec)
    Uses the Hadoop codec class to compress data as it is written to HDFS.

--null-string
    The string to be written for a null value for string columns

--null-non-string
    The string to be written for a null value for non-string columns

  Output line formatting options

The delimiters used to separate fields and records can be specified on the command line, as can a quoting character and an escape character (for quoting delimiters inside a values). Data imported with --as-textfile will be formatted according to these parameters. Classes generated by Sqoop will encode this information, so using toString() from a data record stored --as-sequencefile will reproduce your specified formatting.

The (char) argument for each argument in this section can be specified either as a normal character (e.g., --fields-terminated-by ,) or via an escape sequence. Arguments of the form \0xhhh will be interpreted as a hexidecimal representation of a character with hex number hhh. Arguments of the form \0ooo will be treated as an octal representation of a character represented by octal number ooo. The special escapes \n, \r, \", \b, \t, and \\ act as they do inside Java strings. \0 will be treated as NUL. This will insert NUL characters between fields or lines (if used for --fields-terminated-by or --lines-terminated-by), or will disable enclosing/escaping if used for one of the --enclosed-by, --optionally-enclosed-by, or --escaped-by arguments.

The default delimiters are , for fields, \n for records, no quote character, and no escape character. Note that this can lead to ambiguous/unparsible records if you import database records containing commas or newlines in the field data. For unambiguous parsing, both must be enabled, e.g., via --mysql-delimiters.

--fields-terminated-by (char)
    Sets the field separator character

--lines-terminated-by (char)
    Sets the end-of-line character

--optionally-enclosed-by (char)
    Sets a field-enclosing character which may be used if a value contains delimiter characters.

--enclosed-by (char)
    Sets a field-enclosing character which will be used for all fields.

--escaped-by (char)
    Sets the escape character

--mysql-delimiters
    Uses MySQL’s default delimiter set:

    fields: , lines: \n escaped-by: \ optionally-enclosed-by: '

  Input line parsing options

Record classes generated by Sqoop include both a toString() method that formats output records, and a parse() method that interprets text based on an input delimiter set. The input delimiters default to the same ones chosen for output delimiters, but you can override these settings to support converting from one set of delimiters to another.

--input-fields-terminated-by (char)
    Sets the input field separator

--input-lines-terminated-by (char)
    Sets the input end-of-line char

--input-optionally-enclosed-by (char)
    Sets an input field-enclosing character

--input-enclosed-by (char)
    Sets a required input field encloser

--input-escaped-by (char)
    Sets the input escape character

  Hive options

--hive-home (dir)
    Override $HIVE_HOME

--hive-import
    If set, then import the table into Hive

--hive-overwrite
    Overwrites existing data in the hive table if it exists.

--create-hive-table
    If set, then the job will fail if the target hive table exits. By default this property is false.

--hive-table (table-name)
    When used with --hive-import, overrides the destination table name

--map-column-hive (mapping)
    Override default mapping for SQL types into Hive types for configured columns

  HBase options

--column-family (family)
    Sets the target column family for the import

--hbase-create-table
    If specified, create missing HBase tables

--hbase-row-key (col)
    Specifies which input column to use as the row key

--hbase-table (table-name)
    Specifies an HBase table to use as the target instead of HDFS

  Code generation options

--bindir (dir)
    Output directory for compiled objects

--class-name (name)
    Sets the name of the class to generate. By default, classes are named after the table they represent. Using this parameters ignores --package-name.

--jar-file (filename)
    Disable code generation; use specified jar

--outdir (dir)
    Output directory for generated code

--package-name (package)
    Puts auto-generated classes in the named Java package

--map-column-java (mapping)
    Override default mapping from SQL type to Java type for configured columns

  Incremental import options

Sqoop can be configured to import only "new" data from the source database using the arguments in this section. While any import job can make use of these arguments, they are most powerful when used to initialize a recurring job with sqoop job --create .... After executing a saved job, the last observed value of the check column is updated in the saved job.

--incremental (mode)
    Specifies that this is an incremental import. Determines how Sqoop should discover new data. "mode" may be append, in which case new rows are expected to be added with increasing id values, or lastmodified, in which case new data is discovered by comparing a timestamp column with the timestamp at which the last import was performed.

--check-column (col)
    Specifies a column whose value should be compared to the last imported id or the last import timestamp to determine rows to import.

--last-value (value)
    Specifies the most recent id imported, or the timestamp of the most recent id. This argument is unnecessary for an initial import.

  Database-specific options

Additional arguments may be passed to the database manager after a lone -- on the command-line.

In MySQL direct mode, additional arguments are passed directly to mysqldump.

Note: When using MySQL direct mode, the MySQL bulk utilities mysqldump and mysqlimport should be available on the task nodes and present in the shell path of the task process.

Note: When using PostgreSQL direct mode, the PostgreSQL client utility psql should be available on the task nodes and present in the shell path of the task process.

ENVIRONMENT

See sqoop(1)

AUTHOR

Sqoop Team
    Author.