File Upload in Analytics

You can bring data into Envisio Analytics from almost any source by uploading CSV (comma-separated values) files. You can either upload your CSV file manually or set up a scheduled secured FTP connection. 

 

How File Upload Works

When a file is uploaded through the manual upload the columns will be created in your data source with a Data Type of Text. Once the upload completes the user will need to define the Data Type (Text, Number, Formula, etc.) for each column.

When you upload a file, you have the option to ‘Append’ or ‘Replace’ the data in your data source.

• Append will add to the bottom of the existing data
• Replace will wipe out the existing data set and replace it with the data in your file. 

Upload File Format

The majority of file upload error is caused by improper file formatting. So you should take care to ensure that your file has proper formatting. Probably the most common cause of content errors is improper use of separators, quote marks, and line break characters in the uploaded data. Understanding how file upload handles these can save you trouble down the road.

 

UTF-8 encoding

Your upload file should be in UTF-8 encoding. This is a standard encoding for most applications on the web. However, if you are exporting data from certain desktop product, such as Microsoft Excel™, you may need to convert your file to UTF-8 before uploading into Envisio Analytics. 

Related article: Create a UTF-8 encoded .CSV 

Separators

All the fields in your data must be separated from each other by commas. If there are commas within the actual data in a field you want to upload, that field must be surrounded by quotes. If your data includes double quotes, you can use a single quote character to surround the field.

Line breaks

Each line in the file must end with a line break. File upload does not support line breaks in your data even if these are escaped by quotes.

Header row

The first line in your file must be a header row. We will skip the header row when uploading data.

Limits of file upload

Each manual file upload has a file size limit of 50MB. There is no limit for SFTP upload.


 

Manually Upload CSV file

To manually upload your CSV file:

1. On your manual data source, click on Data Integration
2. From the Manual Upload tab, click on Choose File to select your .CSV file from your computer.
3. Select a Data Overwrite Option. 
   • Append will add to the bottom of the existing data
   • Replace will wipe out the existing data set and replace it with the data in your file. 
4. Toggle Override Column Name from CSV file, this option when toggled on will take the column names from your CSV and replace any columns manually created prior to upload. If toggled off column names will remain unchanged in your data source.
5. Click Upload to start uploading your file.

See below for a step by step walkthrough on uploading your data:

Upload File(s) Using Secure FTP

What is SFTP?

Secure File Transfer Protocol (SFTP) allows organizations to move data over a Secure Shell (SSH) data stream, providing excellent security over its FTP cousin. SFTP’s major selling point is its ability to prevent unauthorized access to sensitive information—including passwords—while data is in transit. The connection between the sender and receiver requires the user to be authenticated via a user ID and password, SSH keys, or a combination of the two.

To enable SFTP for your data source:

Pre-requisite: prior to setting up SFTP upload do an initial Manual CSV Upload to create the columns, define the data type and properties. Once complete, the SFTP upload can be enabled.

On your manual data source, click on Data Integration

1. From the SFTP tab, click on Enable SFTP toggle.
   When enabling SFTP, a set of SFTP credentials will be automatically generated for your account and data source. You can use your Envisio Password or use your SSH Key.
2. Select a Data Overwrite Option. 
   • Append will add to the bottom of the existing data
   • Replace will wipe out the existing data set and replace it with the data in your file. 
3. Set a schedule for the SFTP sync. 
4. Select your notification preference. 

Follow the provided link for a video walkthrough of how to configure and upload using SFTP.

You can upload multiple files to the SFTP folder. Envisio will compare the modified date of the files in the folder with the last sync date on the data source and it will process the new files as a batch when syncing, from oldest to newest. The data overwrite option will be honored at this time. This makes it easy to update your data as time goes on.

SFTP Credentials 

What is an SSH Key?

Setting up SSH keys allows users to be authenticated without the use of passwords. SSH key authentication works with key-pairs, namely, a public key and a private key. In order to use SSH keys with SFTP, you must generate an SSH key-pair on your computer and save the public key in Envisio. The private key should not be shared and should remain on the computer being used to access the SFTP server.

 

How to generate SSH key on my computer? 

Linux/Mac

1. Open your Terminal.
2. Paste the text below.

ssh-keygen -P "" -t rsa -b 4096

1. When you’re prompted to “Enter a file in which to save the key”, press Enter to accept the default file location or specify where you would like the file to be saved.
   
    

Windows

1. Install PuTTYgen as part of the PuTTY package (https://www.ssh.com/ssh/putty/download).
2. Open PuTTYgen and change “Number of bits in a generated key” to 4096.
3. Click the Generate button.
4. After following the steps to fully generate the key, save both the private key and the public key.

 

File Upload Log

What causes file upload to error? 

A file upload will be flagged with an error in the case when an unrecoverable error was encountered while processing. For manual upload, it could be a server communication or code error. For SFTP upload, server communication or the file being deleted from the SFTP folder while processing could be the cause of errors.

File upload error codes

Pending - A log entry has been created for the file but the data has not yet been processed and imported into your data source.

 

Success - The file was successfully uploaded and imported into your data source.

Warning - The file was uploaded and imported into your data source but there were issues with processing certain lines (eg. encoding incompatibility), which were skipped during the import process.

 Error - An unrecoverable error was encountered while processing the file. A retry button will be offered for the first failure or you can choose to cancel the upload. After one retry, only the cancel action will be available. On errors, the data source will still be locked from being edited and can only be unlocked from a successful retry or cancelling the upload.

Cancelled - The upload has been cancelled.

Notifications

You can configure whether to be notified by email for your SFTP sync. The email will notify the data source owner whether the SFTP sync has completed successfully, completed with warnings or encountered an error.

Identify Data Integration Type

On the data source list page, if a data integration has been enabled, the type will be included in the 'Data Integrations' column.