ETL Operations Guide

Use this skill when migrating data between datasources, creating entities, or running ETL copy scripts.

Core Types

• ETLEditor: orchestrates ETL operations
• ETLScriptHDR: script header
• ETLScriptDet: individual steps (create, copy)

Workflow

1. Create ETLEditor with editor.
2. Call CreateScriptHeader(sourceDs, progress, token).
3. Optionally filter or edit ScriptDetails.
4. Run RunCreateScript(progress, token, copydata, useEntityStructure).

Validation

• Check returned IErrorsInfo.Flag and LoadDataLogs for failures.
• Use ETLValidator when validating preconditions.

Pitfalls

• Copying data without creating entities first will fail.
• Large copy without batching can be slow or memory heavy.
• Forgetting to update destination datasource name in scripts can copy back to source.

File Locations

• DataManagementEngineStandard/Editor/ETL/ETLEditor.cs
• DataManagementEngineStandard/Editor/ETL/ETLScriptBuilder.cs
• DataManagementEngineStandard/Editor/ETL/ETLScriptManager.cs

Example

csharp
1var etl = new ETLEditor(editor);
2var progress = new Progress<PassedArgs>(p => Console.WriteLine(p.Messege));
3var token = CancellationToken.None;
4
5etl.CreateScriptHeader(sourceDs, progress, token);
6
7// Optional: keep only some entities
8etl.Script.ScriptDetails = etl.Script.ScriptDetails
9    .Where(s => s.SourceEntityName == "Customers")
10    .ToList();
11
12await etl.RunCreateScript(progress, token, copydata: true, useEntityStructure: true);

Task-Specific Examples

Migrate Specific Entities Only

csharp
1var etl = new ETLEditor(editor);
2etl.CreateScriptHeader(sourceDs, progress, token);
3
4etl.Script.ScriptDetails = etl.Script.ScriptDetails
5    .Where(s => s.SourceEntityName == "Customers" || s.SourceEntityName == "Orders")
6    .ToList();
7
8await etl.RunCreateScript(progress, token, copydata: true, useEntityStructure: true);

Import Using Mapping

csharp
1var etl = new ETLEditor(editor);
2var (errorInfo, map) = MappingManager.CreateEntityMap(
3    editor,
4    "LegacyCustomers",
5    "LegacyDB",
6    "Customers",
7    "MainDB");
8
9if (errorInfo.Flag == Errors.Ok)
10{
11    var selected = map.MappedEntities.First();
12    etl.CreateImportScript(map, selected);
13    await etl.RunImportScript(progress, token);
14}