This article will serve as a guideline when interacting with the Service Delivery Team/Agent who will assist the customer with the ingestion request. This will be broken into the following sections.
- How the Dropsuite Free Ingestion Service Works
- Providing Amazon S3 Bucket Credentials
- How to Upload Data Using the Dropsuite Free Ingestion Service
- Important Points to Keep in Mind
- Overview of Ingestion Types
- Mapping File Guidelines
How the Dropsuite Free Ingestion Service Works
Once your request has been forwarded to the Service Delivery Team, you will be contacted through the ticketing system by a team member. They will introduce themselves and let you know which time zone they are working in, so you can have an idea of how often to expect updates.
Typically, for the Free Ingestion Service, you can expect a response within 2 business days after your ticket has been transferred to the Service Delivery Team.
Providing Amazon S3 Bucket Credentials
The team member will request the necessary Amazon S3 bucket credentials from the internal team. These credentials will be sent to you through a separate email (out-of-band) to the email address associated with your account.
NOTE: If you need the credentials sent to a different contact person, please let the support team know when you first reach out.
Along with the credentials, you will receive a short guide and a download link for the third-party S3 Browser tool.
Please be aware that Dropsuite does not provide support for the S3 Browser, as it is third-party software. We have no control over factors such as upload speed or file size limits when using this tool.
However, we recommend using S3 Browser because it is a freeware tool that offers an easy-to-use graphical interface for accessing and interacting with S3 Buckets.
How to Upload Data Using the Dropsuite Free Ingestion Service
Once you receive the Amazon S3 credentials in an out-of-band email, follow these steps to upload your data:
- Download S3 Browser
- Visit S3 Browser Download and download the application.
- Install it on your system.
- Note: if you have a significant amount of data to upload to S3 then you may wish to purchase the S3 Browser Pro version which offers concurrent upload capability.
- Launch the S3 Browser Application
- Open the S3 Browser app. When prompted, enter the Access Key ID and Secret Access Key provided in the email. Be sure to input the details exactly as they appear, without any extra characters.
- Enter the Bucket Name
- In the next step, you will be asked to enter the external bucket name. Enter it as "REPLACEWITHBUCKETNAME" (without the quotes).
- Upload Files
- Navigate to the uploads folder in S3 Browser. You can now start uploading the necessary zip files for ingestion.
Please note that the credentials link provided in the email will expire in 7 days, so it’s recommended that you securely store the credentials locally for future use.
These screenshots can guide you through the process.
Important Points to Keep in Mind
- Folder Structure Retention
- If your EML/PST files are uploaded with an existing folder structure, the ingestion system will retain that structure.
- Ingestion Process
- The system will ingest files on an "as-is" basis. This means that if your files are uploaded within additional top-level directories (e.g., if they are zipped in a particular folder format), the system will ingest them in that exact structure.
- Recommendation: To avoid any folder/sub-folder structure being retained, it is best to upload the EML/PST files directly to the upload folder in the Amazon S3 bucket.
- Only zipped EML files or PST files are accepted for ingestion. Any other file types will be disregarded by the ingestion system.
- Non-EML items included in the files, such as contacts (.vcf) or calendar appointments/meetings (.ical), will not be ingested. These item types, even if present within PST files, will be ignored during the process.
- Dropsuite makes use of de-duplication technologies during the ingestion process. This can reduce the amount of data ingested because some of the data may already exists in our databases. Due to this activity the count/volume of data ingested cannot always be directly compared to the data source provided as proof that the ingestion completed successfully.
-
Compression software
- Please note we only support the use of Zip/Gzip and not 7z software. Our application currently does not support 7z compression.
To ensure a smooth ingestion process, please make sure that your data contains only EML or PST files. Other formats will not be processed by the system.
Overview of Ingestion Types
Archiver Ingestion
Archiver Ingestions are typically used when the email data cannot be directly linked to a specific user. This often happens when the source files are a mix of EML files from multiple accounts and the exact user association is unknown.
In such cases, a "catch-all" email account is usually created, and all the data is imported into this single account. During the ingestion process, the system maps the EML files to the catch-all account or tries to determine the association with individual accounts based on the data provided.
Example:
If you have a large collection of unsorted EML files and cannot determine which file belongs to which user, you can upload all the data into a single account (the catch-all) to preserve the information.
Non-Archiver Ingestion
In a Non-Archiver Ingestion, the mapping between the source files and email accounts is already predetermined. This means you know exactly which files correspond to which email accounts before starting the ingestion process.
The system will directly ingest the files into their designated email accounts without needing to sort them.
Example:
You have a folder of PST files, and each file is labeled according to the user account it belongs to. This allows for a straightforward ingestion into the respective accounts.
Mapping File Guidelines
For the ingestion process to function correctly, you will be required to provide a mapping file in order to associate the source files with the respective email accounts. Dropsuite supports two types of ingestions, each requiring a different type of mapping file.
Mapping File Responsibility:
Please note that Dropsuite cannot build, upload, or manage mapping files on your behalf. We cannot be held responsible for potential mistakes made with data being ingested into the incorrect destination.
The ingestion process is fully automated, and Dropsuite personnel do not have access to the system during this process.
Important Note: Make sure to use the comma ( , ) delimiter in your CSV file. If you use any other delimiter, the ingestion will not work. If this happens, you will need to fix your mapping file and upload it again.
Archiver Ingestion Mapping File
As mentioned earlier, Archiver Ingestions are typically used when the email data cannot be directly linked to a specific user. This usually happens when the source files are a mix of EML files from various accounts, and the exact association with users is unknown. As such the mapping file would be more simple to fill in.
The sample mapping file you will receive for an Archiver Ingestion will look like this:
|
File Name |
|
micro_file_sample.zip |
|
sample.pst |
|
micro_sample.pst |
The below table will explain the fields in the mapping file and what data should be input into them
|
Field Name |
Explanation |
|
file_name |
-This field should contain the filenames of the email data to be ingested. -It must include the full filename with the correct extension. If the file is in subfolders, include the full file path. |
Non-Archiver Ingestion Mapping File
In a Non-Archiver Ingestion, the mapping between the source files and email accounts is already predetermined. This means you know exactly which files correspond to which email accounts before starting the ingestion process. As such the mapping file would require a bit more information.
The sample mapping file you will receive for an Non-Archiver Ingestion will look like this:
| account_id | email_account_id | file_name | |
| XXXXXX |
test_emls/test_sample.pst |
||
| XXXXXX |
test_emls/test_eml.zip |
The below table will explain the fields in the mapping file and what data should be input into them:
| Field Name | Explanation |
|
account_id |
-This is the account ID assigned to your Dropsuite organization. |
| email_account_id | -This field is only necessary when the user's email does not match the email on the tenant or if there are multiple users with the same email address. |
|
|
-This field should contain the user’s email address for which the data should be ingested. |
|
file_name |
-This field should contain the filenames of the email data to be ingested. -It must include the full filename with the correct extension. If the file is in subfolders, include the full file path. |
This section applies to both archiver and non-archiver ingestions.
As detailed above the file_name section of both mapping files should contain the filenames of the email data to be ingested and include the full filename with the correct extension. If the file is in subfolders, include the full file path.
The examples below should explain the difference:
- File uploaded directly to the S3 "uploads" folder (no folders):
| account_id | email_account_id | file_name | |
|
XXXXXX |
|
sample.pst |
- File enclosed in multiple folders within the S3 bucket:
| account_id | email_account_id | file_name | |
|
XXXXXX |
|
FOLDER-NAME1/FOLDER-NAME2/sample.pst |