Data ETL
This page provides an overview of our data ingestion process

Overview

Our data processes contains a sequential chain of process/events to ETL client data to FHIR specifications. The implementation data process starts with working alongside the client to create a functional workbook from our mapping template. This is a Google Sheets document which contains all the data mappings. This sheet is imported into the client PostgresDB. Then, NiFi reads from this table as well as from the raw client's files, either from S3 or from the PostgresDB to generate FHIR resources using a custom processor. The FHIR resources are then loaded into DynamoDB and events stored in ElasticSearch.

NiFi

Each client's NiFi flow may differ slightly from others, as there will often be nuances which need accommodation. For example, one client may sent two files which are use in one resource which must be joined together before processing in NiFi. Another client may send JSON files as opposed to CSV files which must use a different processor.

Custom NiFi Processors

There are a number of custom NiFi processors written by the team in order to accommodate various client's data formats. The custom processors in use are
    CCDAtoFHIR Processor
    IGValidator Processor (Implementation Guide Validator)
    JSONtoFHIR Processor
    HL7toFHIR Processor
    FHIRXMLConverter Processor
Last modified 5mo ago