With vast amounts of data from multiple sources to contend with, healthcare organisations often have to deal with complex and expensive technology options for handling and storing large volumes of data securely. Reliability is also a key concern, especially when data integrations are part of mission critical workflows. Integration engines are trying to bridge the gap and provide translation between multiple systems, it is a challenging task and too often error prone. When errors occur, it may be necessary to access and re-process original data again. The potential use of data is also constantly evolving – data that is of little interest today may provide the key to crucial insights tomorrow, and having to return to source for that data to re-process can be a costly overhead.
How can healthcare organisations solve these issues?
The re-use of data for issue resolution and new requirements
An original source data store where all incoming data is stored exactly as they were received from the source system can be a great asset. Messages that become malformed, or is otherwise incorrectly processed into a downstream system, can be replayed from this source data store without the expense of retrieval from the original upstream system. This requires clear identification of the relevant messages and a strategy to replace all the previously corrupted information. In a worst-case scenario where an entire system becomes corrupted or unavailable and needs to be replaced or rebuilt, the source data store serves as an ideal backup or source of data for replaying.
In addition, a source data store accelerates an environments deployment rollout, by making the same test data available for verification against development, stage and production environments, making the whole QA process more reliable.
As new data requirements emerge, the stored data can also be used to seed new systems or can be processed in new ways to extract additional insights. An example is the emergence of predictive analytics and machine learning, which often need a large bank of historical data for training; sourcing years’ worth of data for this retrospectively would be expensive, if at all possible.
Offsite storage for reliability and compliance with retention policies
Organisations dealing with healthcare and related data need to be able to apply the necessary retention policies according to legal requirements, quality principles and business needs. HIPAA and GDPR define a set of principles for retaining healthcare data, from keeping the documents for at least 6 years to holding on to them for the shortest time possible. Organisations might have other specific needs for keeping data for a determined amount of time, therefore adding more variables to the rules for retaining the data.
In general, retention policies are defined according to: data types, personally identifiable data, special personal data, purpose of processing, legal obligations, time limits or costs.
The correct classification of processed data and metadata association are fundamental procedures in the application of the retention policies. For data that needs to undergo long-term retention but is not actively used, the archiving option is available. This brings down the cost of storage while searches can be more effective amongst the active data. Archiving can be adopted as an intermediate step towards data erasure.
A commonly used mechanism to improve the reliability of a data system is to store (redundant) copies of data at one or more different physical locations. This ensures that there is always data available for disaster recovery should the primary data store be compromised in some way; reasons can range from cyber-attacks, through to floods, fires, earthquakes, or burglary, which is why highly secure, offsite storage is important to guard against both security and physical threats.
How does Amadeus CORE provide re-use of data and offsite storage
Offsite storage
Amadeus CORE is a HIPAA compliant and HITRUST certifiable data store hosted in an AWS environment. Structured, semi-structured or unstructured data can be persisted in a variety of data stores; Source data can be stored with minimal pre-processing in the Source Data Store, and modelled data can be housed in custom data domains or in pre-configured clinical domains. Amadeus CORE leverages the many security protections and high availability mechanisms of AWS to provide a robust and scalable offsite storage option.
Replay capability
Amadeus CORE acts as an original message store where data has been tagged and organised based on its nature (e.g. format, type, version, etc) and provenance (e.g. source organisation, source application, author, etc). This categorisation, together with crucial characteristics such as time of ingestion and creation date of the documents, enable effective identification of the data that needs to be re-run. As an example, it is possible to trigger a previously processed set of documents to seed a brand-new system, or data may be replayed to existing ones to correct integration mistakes or to resolve issues in downstream systems. Old data can also be a valuable asset when used to extract new information or to leverage new capabilities such as machine learning, therefore when stored and made available over the long term, maximises flexibility and potential of the data system going forward.
Complies with retention policies
When dealing with different types of information, a number of retention policies might apply according to the nature of the data and organisation. A thorough classification of the documents allows the application of the appropriate logic for keeping the right data for the correct timeframe. The policy rules are configurable to satisfy the needs of the business.
Sometimes legal requirements force organisations to keep in storage documents that are not actively used. Under these circumstances, the retention policy could be layered with an archiving option of cold storage, resulting in a more cost-effective solution.
Data is easily purged when no longer required
Organisations can be asked to purge data for a variety of reasons and with retention policies requirements such as in GDPR, data erasure can be requested on demand. Systematic problems or unintended duplication of data, such as in software upgrade scenarios, requires deletion of documents to return the system to a correct state. Data can be erased from Amadeus CORE upon request, and the resulting activities are traceable and audited. Once the data has been erased, it is no longer available. A strategy that encompasses temporary archiving before final deletion, is recommended.
With Amadeus CORE historical information is available for re-use without the expense of retrieving the data a second time from the source. When data is no longer required, it is easily purged from the solution.
Check out the data sheet for more information.
