Data ETL

Overview
Our data processes contains a sequential chain of process/events to ETL client data to FHIR specifications. The implementation data process starts with working alongside the client to create a functional workbook from our mapping template. This is a Google Sheets document which contains all the data mappings. This sheet is imported into the client PostgresDB. Then, NiFi reads from this table as well as from the raw client's files, either from S3 or from the PostgresDB to generate FHIR resources using a custom processor. The FHIR resources are then loaded into DynamoDB and events stored in ElasticSearch. 
NiFi
Each client's NiFi flow may differ slightly from others, as there will often be nuances which need accommodation. For example, one client may sent two files which are use in one resource which must be joined together before processing in NiFi. Another client may send JSON files as opposed to CSV files which must use a different processor.  
Custom NiFi Processors
There are a number of custom NiFi processors written by the team in order to accommodate various client's data formats. The custom processors in use are
CCDAtoFHIR Processor
IGValidator Processor (Implementation Guide Validator)
JSONtoFHIR Processor
HL7toFHIR Processor
FHIRXMLConverter Processor

Data Pipeline

ETL Tool

Last modified 5mo ago

Copy link

Contents

Overview

NiFi

Custom NiFi Processors