1up Admin
1up HomepageSupportDev Console
Search…
1up Platform Administration
Architecture
Deployment
Data ETL
ETL Tool
Provenance
Data Ingestion Guide
DIMA Flat File Extraction Guide
Data Ingestion Updates
Unstructured Data
Sensitive Resource Tagging
Operation
Change/Configuration Management
Validation
3rd Party App Vetting
Kibana Reporting
Member Services Training Material
Implementations
Powered By GitBook
Data ETL
This page provides an overview of our data ingestion process

Overview

Typically we receive customer claims, clinical, administrative, provider directory, and/or formulary data via flat files that are mapped, transformed to FHIR, and loaded to our FHIR server via the Data ETL tooling described below. The frequency with which we update data in our FHIR server depends on the frequency we receive incremental updates (e.g., daily, weekly, monthly, etc.). During implementation we work with customers to meet provisions in the CMS mandate for data being updated in a timely manner (claims one business day after they are processed, encounters one business day after being received, clinical data one business data after being received, provider directory updates 30 days after being received).
Our data processes contains a sequential chain of process/events to ETL client data to FHIR specifications. The implementation data process starts with working alongside the client to create a functional workbook from our mapping template. This is a Google Sheets document which contains all the data mappings. This sheet is imported into the client PostgresDB. Then, NiFi reads from this table as well as from the raw client's files, either from S3 or from the PostgresDB to generate FHIR resources using a custom processor. The FHIR resources are then loaded into DynamoDB and events stored in ElasticSearch.

NiFi

Each client's NiFi flow may differ slightly from others, as there will often be nuances which need accommodation. For example, one client may sent two files which are use in one resource which must be joined together before processing in NiFi. Another client may send JSON files as opposed to CSV files which must use a different processor.

Custom NiFi Processors

There are a number of custom NiFi processors written by the team in order to accommodate various client's data formats. The custom processors in use are
  • CCDAtoFHIR Processor
  • IGValidator Processor (Implementation Guide Validator)
  • JSONtoFHIR Processor
  • HL7toFHIR Processor
  • FHIRXMLConverter Processor
Previous
Data Pipeline
Next
ETL Tool
Last modified 3mo ago
Copy link
Outline
Overview
NiFi
Custom NiFi Processors