Skip to main content

FTP/SFTP Data Connector

The FTP/SFTP Data Connector enables federated SQL query across Parquet/CSV files stored in FTP/SFTP servers.

If a folder is provided, all child Parquet/CSV files will be loaded.

Configuration

Parameters

The connection to FTP can be configured by providing the following params:

  • file_format: Specifies the data file format. Required if the format cannot be inferred by from the from path.
    • parquet: Parquet file format.
    • csv: CSV file format.
  • ftp_port: Optional, specifies the port of the FTP server. Default is 21. E.g. ftp_port: 21
  • ftp_user: The username for the FTP server. E.g. ftp_user: my-ftp-user
  • ftp_pass: The password for the FTP server. E.g. ftp_pass: my-ftp-password
  • ftp_pass_key: The secret key container the password to connect with. E.g. ftp_pass_key: my-ftp-password-key

More CSV related parameters can be configured, see CSV Parameters

Examples

  - from: ftp://remote-ftp-server.com/path/to/folder/
name: my_dataset
params:
file_format: csv
ftp_user: my-ftp-user
ftp_pass_key: my-ftp-password