NEMOServices 1.0.0.6

About

The NEMOServices Package contains a set of classes and functionalities provided by NEMO GmbH and allows to

• Upload and import files into NEMO
• Retrieve NEMO reports as CSV
• Connect to NEMO APIs

How to upload data to NEMO

NEMO generally uses a secure token-based authentication. Before starting an upload it is necessary to have NEMO credentials for authentication.

The following steps describe how to upload and import data to NEMO.

1. Authentication

The first step of NEMO authentication is based on username and password. With the combination of username and password it is possible to retrieve a token tuple from the NEMO user pool. These tokens can be used for authentication to the NEMO APIs. The NEMO class Authentication.cs offers the possibility to pass username and password directly or to read them from the Windows Credentials Manager.

Use Windows Credentials Manager to retrieve the Username and Password

First, create an instance of NEMOServices.Authentication:

Authentication auth = new Authentication( 
	"<region>", // String, the region where the userpool is located, default eu-central-1
	"<userpool-id>", // String, the Userpool ID (this information will be provided by NEMO DevOps)
	"<appclient-id>",  // String, the Appclient ID (this information will be provided by NEMO DevOps)
	"<name of the WIN secret>",  // String, The name of the secret in the Windows credential store. Currently only "Windows Generic Credentials are provided". 
	"<local error log path>"); // String, local file path to the error log

Forward Username and Password directly to the instance of NEMOServices.Authentication

Authentication auth = new Authentication( 
	"<region>", // String, the region where the userpool is located, default eu-central-1
	"<userpool-id>", // String, the Userpool ID (this information will be provided by NEMO DevOps)
	"<appclient-id>",  // String, the Appclient ID (this information will be provided by NEMO DevOps)
	"<username>",  // String, Name of the NEMO User
	"<password>", // String, Password of the NEMO User
	"<local error log path>"); // String, local file path to the error log

Use the login function of NEMOServices.Authentication to log in to the user pool:

await auth.LogInAsync();

Now the user has been successfully authenticated. The instance of NEMOServices.Authentication must be made known to the other classes of NEMOServices. They will use the Authentication instance to read the required tokens and pass them to the NEMO APIs.

For longer processes NEMOServices.Authentication provides a function to refresh the token tuple:

await auth.RefreshAuthenticationAsync();

There is also a logout function to log out of NEMO after finishing the upload process:

auth.LogOut();

2. Upload a file to the NEMO Cloud

Currently NEMO only supports CSV files with a specific format. Furthermore, CSV files have to be gzip-compressed before being imported into the HANA database. The gzip compression can be done by the NEMOServices.FileUpload class.

The restrictions are:

• international Date format 'YYYY-MM-DD'
• international Timestamp format 'YYYY-MM-DD HH:MM:SS.MS'
• String values should be enclosed by quotes
• No special characters in String values
• Decimal point
• Boolean values should be TRUE/FALSE or 1/0
• Record Delimiter is '\n'
• Column Delimiter is ';'

2.1 The upload process

The NEMOServices.FileUpload class requires additional credentials to upload a local file to Amazon.S3. Therefore it is necessary to create an instance of NEMOServiceTokenVendor, which can provide temporary credentials to access S3.

NEMOServiceTokenVendor tvm = new NEMOServiceTokenVendor(
	auth, // Instance of the NEMOServices.Authentication class, created in a previous step
	"<API Endpoint Url>", // String, Url of the API Endpoint, defaul is production "enter.nemo-ai.com/api/nemo-tokenvendor" 
	"<local error log path>"); // String, local file path to the error log

Next it is necessary to create an instance of NEMOServices.FileUpload and pass the NEMOServiceTokenVendor instance to it.

FileUpload uploader = new FileUpload(
	tvm, // Instance of the NEMOServiceTokenVendor class, created above
	"<local error log path>"); // String, local file path to the error log

The upload can be started with a function call. It will return the full S3 object path, which is needed for the next step of importing the data into the database.

string sUploadFile = uploader.startUpload(
	"<S3 Bucket Name>", // String, Name of the S3 Bucket (this information will be provided by NEMO DevOps)
	"<Region>", // The region where the userpool is located, default eu-central-1
	"tenant ID", // String, the tenant ID (this information will be provided by NEMO DevOps)
	"<path to the local file>", // String, absolute local path to the file to upload
	<true/false>); // Boolean, should be true if the file is not already gzip compressed

2.2 The import process

After the upload has been successfully completed, the resulting S3 object path can be used to import the uploaded file into a NEMO project. The import functionality in NEMOServiceQueue needs the project ID to import the CSV file into the corresponding table in the database. NEMOServicePersistence provides a function to get additional information about a project by its project display name.

NEMOServicePersistence persistence = new NEMOServicePersistence(
	auth, // Instance of the NEMOServices.Authentication class, created in a previous step
	"<API Endpoint Url>", // String, Url of the API Endpoint, defaul is production "enter.nemo-ai.com/api/nemo-persistence" 
	"<local error log path>"); // String, local file path to the error log

JObject project = persistence.GetProjectByDisplayName("MyTestProject");

GetProjectByDisplayName returns a JObject instance with additional information about the project, such as the project ID, which is needed for data import.

string projectid = project.GetValue("id").ToString();

The resulting project ID can be used in the next step of importing the file into the database.

There are two ways to import data into NEMO:

1. Import data and completely replace the target table. This means that the target table is deleted and replaced with the new data.
2. Import data and replace the values in the target table, identified by a datasource id, to import additional data into a large table containing data from other datasources.

Option V2 of dataimport (import data and completely replace the target):

NEMOServiceQueue queue = new NEMOServiceQueue(
	auth, // Instance of the NEMOServices.Authentication class, created in a previous step
	"<API Endpoint Url>", // String, Url of the API Endpoint, defaul is production "enter.nemo-ai.com/api/nemo-queue" 
	"<local error log path>"); // String, local file path to the error log

string task_id = queue.startNEMOImportProcessV2(
	sUploadFile, // String, absolute S3 Object path, has been returned after the FileUpload step from the previous step. 
	projectid); // String, project UUID, was determined with NEMOServicePersistence in the previous step above.  

Version 3 of DataImport allows it to import data into a big table by simply replacing the data from the same DataSource ID. With this option it is necessary to define a data source identifier, which is a key to identify the data source records in the big data table. A data source identifier can be a set of fields or just one field. When included in a query, it should provide a key-value pair that represents the name of the column in the target table, and the value to be set when the data is imported.

JArray datasource_id = new JArray 
{ 
	new JObject{{ "key", "datasource_id"}, {"value","CUSTOMER_MASTER_DATA" }} 
};

It is mandatory that each JObject contains the key and value properties in the following format

[
    {
      "key": "<column name>", // Name of the column in the big data table (column field that represents the data source identifier)
      "value": "<column value>" // Value of the column in the big data table (column value of the data source identifier field)
    }
]

Furthermore, it is possible to map fields from the source file to already existing fields in the target table. So if another data source has a similar field, it could be mapped to have a larger subset between the data sources. For example, identifier or master data fields.

JArray fieldmapping = new JArray 
{
    new JObject{ { "file_source_column", "CustomerStreet" },{ "target_column", "address_street" } },
};

Each JObject representing a mapping must contain the properties "file_source_column" and "target_column" in the following format

[
    {
      "file_source_column": "<column name in the CSV>", // Name of the column in the CSV Header
      "target_column": "<target column internal name>" // Internal Name of the column in the target table in NEMO
    }
]

The value of "file_source_column" must exactly match the column name in the header of the CSV file. "target_column" represents the internal name of the column in the target table in NEMO. Each project in NEMO contains imported columns in the Metadata Editor. The internal names of columns in a NEMO project can be found at https://enter.nemo-ai.com/nemo/metadata/importedColumn.

Once the data source identifier and optionally the column mapping have been defined, the import process can be triggered:

string result = queue.startNEMOImportProcessV3(
	sUploadFile, // String, absolute S3 Object path, has been returned after the FileUpload step from the previous step 
	projectid,  // String, project UUID, was determined with NEMOServicePersistence in a previous step above.  
	datasource_id, // JArray, contains instances of JObject to represent the name and values of the data source identifiers
	fieldmapping); // JArray, contains instances of JObject with the information of source and target columns to map fields

Both functions, startNEMOImportProcessV2 and startNEMOImportProcessV3, start an import job, perform the import asynchronously and return a string with the task ID. This ID can be used to find the corresponding task in the NEMO UI. The status of the import job can be checked in the NEMO UI → Admin Panel → Task Runs.