FileDataSource Guide

Use this skill when working on DataManagementEngineStandard/FileManager to either:

• use FileDataSource correctly as an IDataSource, or
• add a new file-format reader extension.

Use this skill when

• Implementing or fixing FileDataSource behavior (connection, schema, query, CRUD).
• Adding support for a new file type via IFileFormatReader.
• Registering custom readers in FileReaderFactory.
• Debugging format-specific parse behavior, diagnostics, or schema inference.

Do not use this skill when

• The task is only generic IDataSource behavior. Use idatasource.
• The task is only connection config modeling. Use connectionproperties and connection.
• The task is primarily ingestion orchestration policies. Use importing.

Responsibilities

• Keep FileDataSource format-agnostic and delegate parsing/writing to _reader.
• Implement reader-specific logic only in IFileFormatReader implementations.
• Keep schema inference and row parsing consistent with existing file readers.
• Preserve IDataSource contract behavior (ErrorObject, ConnectionStatus, Entities, EntitiesNames).

Core API Surface

• FileDataSource:
  • Openconnection(), Closeconnection()
  • GetEntityStructure(...), GetEntity(...), CRUD methods
  • ResolveEntityFilePath(...)
• Reader contract:
  • IFileFormatReader (Configure, ReadHeaders, ReadRows, GetEntityStructure, write methods)
• Reader registry:
  • FileReaderFactory.Register(...)
  • FileReaderFactory.RegisterDefaults()
  • FileReaderFactory.GetReader(...)

Typical Usage Pattern

1. Configure datasource connection (FilePath, FileName, delimiter/props).
2. Ensure the reader type is registered for the target DataSourceType.
3. Call Openconnection() so FileDataSource resolves _reader via FileReaderFactory.
4. Use GetEntityStructure/GetEntity/CRUD through FileDataSource.
5. For new formats, implement IFileFormatReader, register it, then use FileDataSource unchanged.

Creating a Reader Extension

1. Create a class under FileManager/Readers implementing IFileFormatReader.
2. Decorate the class with FileReaderAttribute so FileReaderRegistry.Discover() can build a FileReaderDescriptor.
   • Example:

csharp
1[FileReader(DataSourceType.CSV, "CSV", "csv")]
2public sealed class CsvFileReader : IFileFormatReader
3{
4    public DataSourceType SupportedType => DataSourceType.CSV;
5    public string GetDefaultExtension() => "csv";
6    // ... implement contract
7}

3. Set SupportedType and GetDefaultExtension() for the format. Keep them consistent with the attribute.
4. Implement Configure(IConnectionProperties) for delimiter/encoding/flags.
5. Implement schema (ReadHeaders, GetEntityStructure) and row streaming (ReadRows).
6. Implement write operations (CreateFile, AppendRow, RewriteFile).
7. Register the reader:
   • Static path: FileReaderFactory.Register(new YourReader())
   • Discovery path: ensure assembly is loaded, then run new FileReaderRegistry(editor).Discover().

Validation and Safety

• Honor ParseMode (Strict vs Lenient) and populate LastDiagnostics.
• Keep row streaming forward-only (IEnumerable<string[]>) for large files.
• Use atomic rewrite semantics for update/delete style operations.
• Normalize/retain source column names consistently with FileReaderEntityHelper.
• Avoid embedding format-specific logic in FileDataSource partials.
• If using registry discovery, do not omit [FileReader(...)]; without it, the reader will not produce a FileReaderDescriptor.

Pitfalls

• Forgetting to register the reader before Openconnection().
• Returning inconsistent column counts from ReadRows.
• Not handling empty files and header-less files deterministically.
• Throwing on every malformed row when ParseMode.Lenient is selected.
• Breaking Entities/EntitiesNames refresh after schema inference.

File Locations

• DataManagementEngineStandard/FileManager/FileDataSource.cs
• DataManagementEngineStandard/FileManager/FileDataSource.Connection.cs
• DataManagementEngineStandard/FileManager/FileDataSource.Schema.cs
• DataManagementEngineStandard/FileManager/FileReaderFactory.cs
• DataManagementEngineStandard/FileManager/Readers/IFileFormatReader.cs
• DataManagementEngineStandard/FileManager/Readers/CsvFileReader.cs

Example

csharp
1public sealed class NdjsonFileReader : IFileFormatReader
2{
3    public DataSourceType SupportedType => DataSourceType.Json;
4    public bool HasHeader { get; set; } = false;
5    public ParseMode ParseMode { get; set; } = ParseMode.Lenient;
6    public IReadOnlyList<RowDiagnostic> LastDiagnostics => _diagnostics;
7    private readonly List<RowDiagnostic> _diagnostics = new();
8
9    public string GetDefaultExtension() => "ndjson";
10    public void Configure(IConnectionProperties props) { }
11    public void ClearDiagnostics() => _diagnostics.Clear();
12    public string[] ReadHeaders(string filePath) => Array.Empty<string>();
13    public EntityStructure GetEntityStructure(string filePath) { /* infer fields */ return null; }
14    public IEnumerable<string[]> ReadRows(string filePath) { /* stream json lines */ yield break; }
15    public string InferFieldType(string current, string rawValue) => TypeInferenceHelper.Widen(current, rawValue);
16    public bool CreateFile(string filePath, IReadOnlyList<string> headers) => true;
17    public bool AppendRow(string filePath, IReadOnlyList<string> headers, IReadOnlyList<string> values) => true;
18    public bool RewriteFile(string filePath, IReadOnlyList<string> headers, IEnumerable<IReadOnlyList<string>> rows) => true;
19}
20
21// Startup/bootstrap
22FileReaderFactory.Register(new NdjsonFileReader());

Related Skills

• idatasource
• connectionproperties
• importing
• beepdm

Detailed Reference

Use reference.md for a complete extension checklist and implementation template.