Data ETL
This page provides an overview of our data ingestion process
Overview
Our data processes contains a sequential chain of process/events to ETL client data to FHIR specifications. The implementation data process starts with working alongside the client to create a functional workbook from our mapping template. This is a Google Sheets document which contains all the data mappings. This sheet is imported into the client PostgresDB. Then, NiFi reads from this table as well as from the raw client's files, either from S3 or from the PostgresDB to generate FHIR resources using a custom processor. The FHIR resources are then loaded into DynamoDB and events stored in ElasticSearch.
NiFi
Each client's NiFi flow may differ slightly from others, as there will often be nuances which need accommodation. For example, one client may sent two files which are use in one resource which must be joined together before processing in NiFi. Another client may send JSON files as opposed to CSV files which must use a different processor.
Custom NiFi Processors
There are a number of custom NiFi processors written by the team in order to accommodate various client's data formats. The custom processors in use are
- CCDAtoFHIR Processor
- IGValidator Processor (Implementation Guide Validator)
- JSONtoFHIR Processor
- HL7toFHIR Processor
- FHIRXMLConverter Processor
Last modified 8mo ago
Copy link