import

Summary

Imports JSON or CSV files into Fauna collections.

fauna import [OPTIONS]

Description

The import command imports the contents of a JSON or CSV file, or a directory of JSON or CSV files, into a Fauna collection. If you import a directory of source files, each file is imported into a separate collection.

JSON source files must contain valid JSON, and each JSON object in the file becomes a document in the target collection.

CSV source files must be comma-delimited, and each line in the CSV file becomes a document in the target collection. A CSV file must have a header line containing the field names to be used in the target collection.

If your CSV file has rows which contain fewer fields than the number of fields in the header line, you can use the --allow-short-rows option to permit the import to proceed. Otherwise the import fails with an error. If you use the --allow-short-rows option, the documents imported from short rows do not contain the missing fields.

If your CSV files has empty columns, you can use the treat-empty-csv-cells-as option to specify:

  • empty: The field exists in the imported document as an empty string.

  • null (default): The field does not exist in the imported document.

The target collection can be either an existing Fauna collection or a new one. If the target collection exists and is not empty, you must use the --append option, and the new documents are added to the existing collection. If you don’t specify a target collection, Fauna Shell uses the filename of the source file as the name of the target collection.

The --path option specifies the path to the source file or directory of source files, and it is required. If you don’t specify any other options, Fauna Shell uses the parameters specified in your fauna-shell configuration file.

Floating-point values that end in .0 are converted, by JavaScript, into integers.

Billing considerations

Importing data with the import command involves billable write and compute operations. One write operation accrues for each 1kb of data written, with a minimum of 1kb per imported record, so if you import a file of 5,000 small records, you accrue 5,000 write operations. If the records are larger, for example, 5kb each, you accrue 25,000 write operations (5 each for 5,000 records).

Additionally, importing data into collections with covering indexes causes index updates which incur write operations.

Compute operations accrue at a rate of approximately 1 compute operation for every 20 write operations. The rate may vary depending on several factors, including record size and retries for recoverable errors.

See Billing for more information.

To ensure the integrity of your data import, it is recommended that each record of your source file should include a unique identifying field. You can create a unique index on that field to ensure that no records are imported more than once, and you can query against the unique field to verify the completeness of your import.

Options

Option Description

--allow-short-rows

Optional - Allow CSV files in which some rows have fewer fields than the number of fields in the header row. Note that if import finds a row which has more fields than the number of fields in the header row, the import fails with an error.

--append

Optional - Append documents to an existing collection.

--collection=<collection>

Optional - Specifies the target collection to be created. Defaults to the filename of the source file.

--db=<db>

Optional - Specifies a child database in which to create a new target collection or append to an existing target collection. The parent database is always the database associated with the secret you specify for the command.

--dry-run

Optional - Performs all import operations, including record processing, type conversions, etc., except creating document in Fauna. This allows you to detect issues that might impact an import before writing documents to your collections.

--path=<path>

Required - Specifies the path to a source file or directory of source files.

Note that directories may only contain files, not additional directories. import does not recurse into sub-directories.

--domain=<domain>

Optional - The Fauna server domain, that is, the hostname where Fauna is running. Defaults to db.fauna.com.
Since the introduction of Region Groups, three cloud domains are available. You do not need to specify a domain, but connections work as expected if you do.
Classic (US and EU): db.fauna.com
United States (US): db.us.fauna.com
Europe (EU): db.eu.fauna.com

--endpoint=<endpoint>

Optional - The name of the endpoint to use for the command.

--port=<port>

Optional - The connection port. Defaults to 8443.

--scheme=<scheme>

Optional - The connection scheme. Must be one of https or http. Defaults to https.

--secret=<secret>

Optional - The secret to use. A secret authenticates your connection to Fauna, and connects you to a specific database.

--timeout=<timeout>

Optional - The connection timeout, an integer number of milliseconds. When the specified period has elapsed, fauna-shell stops waiting for a response and displays an error.

The default is zero, which means that fauna-shell waits until a response is received.

treat-empty-csv-cells-as=<empty|null>

Optional - Specifies how empty fields in a record should be handled:

  • empty: Empty fields should appear in the imported document containing an empty string.

  • null (default): Empty fields should not appear in the imported document.

--type=<column>::<type>

Optional - Specify a data type for a column.

This feature is only available while importing CSV files. Type conversions are ignored for JSON and JSONL files.

You can specify a column in your source file to be imported as one of the following data types:

  • bool: Converts true, t, yes, or 1 to true and all other values to false, except null which remains null.

  • number: Converts strings to numbers.

  • dateString: Converts an ISO-8601 or RFC-2022 date string into a Timestamp value. A best-effort attempt is made for other date string formats, and warnings are emitted when such date conversions are performed.

  • dateEpochMillis: Converts milliseconds since Unix epoch into a Timestamp value.

  • dateEpochSeconds: Converts seconds since Unix epoch into a Timestamp value.

Examples

Import a JSON file

A file named zipcodes.json contains the following:

{ "zipcode" : "01001", "city" : "AGAWAM", "pop" : 15338, "state" : "MA" }
{ "zipcode" : "01002", "city" : "CUSHMAN", "pop" : 36963, "state" : "MA" }
{ "zipcode" : "01005", "city" : "BARRE", "pop" : 4546, "state" : "MA" }
{ "zipcode" : "01007", "city" : "BELCHERTOWN", "pop" : 10579, "state" : "MA" }
{ "zipcode" : "01008", "city" : "BLANDFORD", "pop" : 1240, "state" : "MA" }

The following terminal command imports zipcodes.json:

fauna import --path=./zipcodes.json
Database connection established
Starting Import!
 ›   Success: Import from ./zipcodes.json to zipcodes completed. 5 rows/object imported.

In the above command, no --collection option is specified, so Fauna Shell creates a new collection called zipcodes.

Import a JSON file and append to an existing collection

A file named zipcodes2.json contains the following:

{ "zipcode" : "01010", "city" : "BRIMFIELD", "pop" : 3706, "state" : "MA" }
{ "zipcode" : "01011", "city" : "CHESTER", "pop" : 1688, "state" : "MA" }
{ "zipcode" : "01012", "city" : "CHESTERFIELD", "pop" : 177, "state" : "MA" }

The following terminal command imports zipcodes2.json and appends the documents to the existing collection zipcodes:

fauna import --path=./zipcodes2.json --collection=zipcodes --append
Database connection established
Starting Import!
 ›   Success: Import from ./zipcodes2.json to zipcodes completed. 3 rows/object imported.

Import a JSON file with configuration options

The following terminal command overrides any configuration options in the fauna-shell configuration file:

fauna import --path=./zipcodes.json --scheme=https --domain=db.us.fauna.com --port=443 --secret=secret
Database connection established
Starting Import!
 ›   Success: Import from ./zipcodes.json to zipcodes completed. 5 rows/object imported.

Import a CSV file with type conversions

The following CSV document contains three string values:

myDate,myBool,myNumber
"May 3, 2021",true,15338

To convert those string values to other types, you can use the --type option.

fauna import --path=./myFile.json --type=myDate::date --type=myBool::bool --type=myNumber::number
Database connection established
Starting Import!
Warning: the string 'May 3, 2021' is not valid ISO-8601 nor RFC_2822 date. Making a best-effort translation to 'Mon May 03 2021 00:00:00 GMT-0700 (Pacific Daylight Saving Time)'
 ›   Success: Import from ./myFile.csv to myFile completed. 1 rows/object imported.

The above JSON object becomes the following Fauna document:

{
  ref: Ref(Collection("myFile"), "328119865601688064"),
  ts: 1649178338530000,
  data: {
    myDate: Time("2021-05-03T07:00:00Z"),
    myBool: true,
    myNumber: 15338
  }
}

Import a directory of source files

A directory named source_files contains the following files:

myJSONfile1.json
myJSONfile2.json
myCSVfile1.csv
myCSVfile2.csv

The following command imports four files and creates four new collections:

fauna import --path=./source_files
Database connection established
Starting Import!
 ›   Success: Import from ./source_files/myCSVfile1.csv to myCSVfile1 completed. 8 rows/object imported.
Starting Import!
 ›   Success: Import from ./source_files/myCSVfile2.csv to myCSVfile2 completed. 14 rows/object imported.
Starting Import!
 ›   Success: Import from ./source_files/myJSONfile1.json to myJSONfile1 completed. 10 rows/object imported.
Starting Import!
 ›   Success: Import from ./source_files/myJSONfile2.json to myJSONfile2 completed. 10 rows/object imported.

Is this article helpful? 

Tell Fauna how the article can be improved:
 Visit Fauna's forums or email docs@fauna.com

Thank you for your feedback!