Support Protocol: SFTP, FTP
Importer:
Read data, filter and create data snapshot with SparkSQL
Support Protocols: S3 file, Data-as-a-service platform Hudi
Processor:
Process data from multiple importers with SparkSQL
Custom process logic(Spark, Scala) can be defined
Comparator:
Compare input data with SparkSQL
- save compared result in S3
- save metadata of the result in DB
-
Uploader:
Upload the results from Importer, Processor
Support Protocols: S3, SFTP