Validation
1upHealth performs five categories of tests to validate client deliverables. This section describes the purpose and frequency of each testing category, and provides guidance on interpreting results.

FHIR API Conformance

Purpose

Ensure the APIs 1upHealth has established in each client environment conform to CMS implementation guidance for the Interoperability and Patient Access final rule

Testing Frequency

    Initial: once after the environment is established
    Pre-release: once during user acceptance testing
    Ongoing: before each subsequent software release to the client Production environment

Overview

We use Touchstone, a product from Aegis, an independent third-party testing organization. Aegis is a member of the FHIR Business Alliance. Their Touchstone product integrates the latest health IT standards. Testing with Touchstone helps us ensure our updates and your environments always meet the most current FHIR requirements.

Interpretation

We use several standard testing sets using Touchstone:
    CARIN Blue Button Test
    Da Vinci FHIR 4.0.1 Plan Network (i.e., provider directory implementation guide)
    Da Vinci FHIR 4.0.1 PDEX Formulary
    FHIR 4.0.1 Basic (overall FHIR release 4 testing)
The FHIR 4.0.1 tests will run on the central 1UpHealth production server, not individual client environments.
Key Information in Header
    When did the collection of tests start and end?
    What’s the status of all tests collectively? (Passed / Failed)
    How many total tests ran?
    On which environment did the tests run?
Additional Key Information in Detail
    What did the individual test examine?
    What’s the status of the individual test? (Passed / Failed)
Sample Test Report Output: FHIR API Conformance

API Status and Uptime Monitoring

Purpose

Ensure the production client environment is actively and correctly handling FHIR server uptime, FHIR APIs, user management, and OAuth 2.0 authorization management APIs

Testing Frequency

    Initial: set up after the production environment is established and the 1UpHealth platform is deployed
    Pre-release and ongoing: beginning with user acceptance testing, once every 10 minutes for authorization & user management and uptime; every 30 minutes for FHIR APIs

Overview

We use Checkly, an independent third-party testing organization. Checkly is designed to actively monitor APIs and user interfaces. This makes it a natural fit to ensure your production environment is always available to developers, members, and other users.

Interpretation

Key Information
    At the last test run, did the specified API respond properly when called?
    What was the response time?
    What percent of the time did the API response performance conform to our expectations over the past day and past week?
Sample Test Report Output: Application Validation and Consent Management

Data Validation

Purpose

Ensure your data are received at 1upHealth consistent with our agreed-upon data intake specifications

Testing Frequency

    Performed with every data update after final specifications are in place

Overview

Our in-house data-validation tests ensure the data we receive match the content and quality of the data you send. These tests also highlight any potential trouble with FHIR resource creation. This helps us quickly trace problems to our data-handling activities and/or to the source data.

Interpretation

Key Information
    How many files, members, and resources are associated with the update?
    Which specific files were updated, and did we detect any invalid data?
    How many FHIR resources were created, by type, and did we detect any invalid resources?
Sample Test Report Output: Data Validation

Performance

Purpose

Ensure performance meets appropriate standards for responsiveness and stability under a particular workload

Testing Frequency

    Continuous monitoring after each client environment is established

Overview

We use Apache JMeter to test functional behavior and measure performance under the load of ongoing FHIR server and environment activity. JMeter tests both static and dynamic resources, discovers concurrent users, and provides a variety of graphical analyses for performance testing. JMeter performance testing includes:
    Load testing: modeling the expected usage by simulating multiple users accessing the APIs concurrently
    Stress testing: finding the maximum load a server can handle before responding slowly or producing errors
This testing ensures your members have the best possible user experience while retrieving their data.

Interpretation

Key Summary Information
    What percent of OAuth requests are fulfilled within the toleration threshold?
Key Detail Information
    How many requests were made?
    How many failed?
    Of those that completed successfully, how were response times distributed?
    Does performance degrade over time?
Sample Test Report Output: Performance

Security

Purpose

Ensure environments remain protected against unauthorized penetration and other threats to privacy and security

Testing Frequency

    Continuous monitoring after each client environment is established

Overview

We use OWASP ZAP (Open Web Application Security Project Zed Attack Proxy) to monitor and categorize risks, and to generate notifications when certain risk levels or types are triggered. In addition to OWASP ZAP, we use Rapid7 to support penetration and network security testing. Rapid7 is a web scanning tool that includes CSS injection, LDAP injection testing, SQL injection, cross-site request forgery scripting, Insecure Direct Object References testing, and missing function level access control testing. We have a shared interest in maintaining the highest standards of security and privacy for your members and their data.
Interpretation
Key Summary Information
    How many alerts were triggered, and at which risk levels?
Key Detail Information
    What specific alerts were triggered?
    What additional reference information is available to help isolate and, if appropriate, address this alert?
Sample Test Report Output: Security
Last modified 8mo ago