Dynamic Entity Attribute Value Model

Creating a database model that supports dynamic entity creation requires a flexible and scalable design. The goal is to design tables that can accommodate the creation of new entities  on the fly, without the need for structural database changes. 

Applications  built on a microservices architecture, where each microservice manages its own database. This architecture enhances service independence and scalability. Several of these microservices also implement the dynamic entity model in their databases, offering flexibility for managing dynamic and diverse data types and relationships.

Proposed dynamic entity design allows for individual microservices to be updated, scaled, and maintained independently, promoting agility and robustness in our system's overall functionality.

This model is Ideal for scenarios where entity attributes are numerous and varied, and where new attributes might be frequently added

Components:

1. BaseEntity: Represents generic entity types in our system.

                Example ; Study, Parcel, Sample etc

     2. EntityInstance: Each instance of an entity is recorded here, storing unique occurrences of BaseEntity types. 

                Example, a specific study or a particular task.

     3. EntityAttribute: Defines the set of attributes applicable to each entity type.

•         Attribute can contain validations like Minlength, max length, validations etc
• -metadata for attributes added to attributes table

     4. EntityAttributeValue: Stores values for these attributes for each entity instance.  

     

     5. EntityMetadata: Provides additional descriptive information or configuration settings for each entity type or instance, enhancing the contextual understanding of the entities. 

    6. EntityRelationship: Manages the relationships between different entities, crucial for representing complex associations such as the linkage between studies and samples or tasks and their parent entities.

• The EntityRelationship table essentially serves as a cross-reference (XRef) table, especially in a many-to-many relationship scenario. 
• EntityInstance records are joined through EntityRelationship to find all related child entities for a given parent entity.

Managing Dynamic Attributes

Key Challenges

• Attributes differ significantly across entity types and instances, requiring a system that can accommodate a wide range of data structure
• As the number of entities and attributes grows, the system must scale efficiently without compromising performance.
• Ensuring accuracy and consistency of data across various entities with diverse attributes.

Strategies for Managing Dynamic Attributes

• DB model allows our microservices to define and modify attributes without restructuring the database schema.
• Attribute Metadata Management: Utilize EntityMetadata to store additional information about attributes, such as data validation rules, which helps in maintaining data quality and integrity.
• APIs will be designed to dynamically generate responses based on the attributes of the requested entities, ensuring flexibility and relevance in data delivery.

Use cases

• When a new attribute is introduced, it will be registered in the EntityAttribute table, making it immediately available for association with entity instances.
• Values for these attributes are stored in the EntityAttributeValue table, allowing for efficient retrieval and manipulation as per business logic requirements.
• The frontend dynamically adjusts to display and interact with these attributes, providing a seamless user experience regardless of the underlying data 

Design Considerations

UI Structure  

• Generic page that can adapt to display any entity type. This is already in place
• Dynamic form to render different types of inputs/display fields based on the data type of each attribute ( text fields, data pickers, dropdowns etc)
• metadata can be used to add additional information or influence the rendering of the entity (e.g., adding tooltips, conditional formatting).
• UI to call  endpoints passing relevant entity and instance ID’s

Fetching Data

• API endpoints to fetch data for a specific entity including its attributes and metadata

Example 

• To fetch a specific Study instance, Query Should join baseentity, EntityInstance, EntityAttribute and EntityAttributeValue tables
• If these  tables have large number of rows, Performance will be extremely slow. To mitigate that we can consider indexes on frequently queried columns like entityid

Data Integrity and Consistency –  

• Enforce data type consistency. For instance, if an attribute 'StartDate' is designed to store dates, the schema restricts this field to date data types only.
• Data Validation :before inserting or updating data, the application layer checks if a 'StartDate' is indeed a valid date and not just a random string or number.
• When a new 'Study' entity is created, and multiple attributes are added, this process is wrapped in a transaction. If adding any attribute fails, the entire operation is rolled back to avoid partial updates.

Optimization and Performance:

• Leveraging indexing on the EntityAttributeValue table for faster query execution.
• Dynamic model queries can be complex and might impact performance. We should consider query optimization techniques, caching, or even indexed views if the performance becomes a concern.

Barcode Printing Solution

In the labs, there will be different types of barcode label printers. When designing a solution for barcode systems, it is important to streamline the processes, protocols, and network connectivity to optimize operations and maintenance costs effectively.

1. Network Connectivity 

Transition USB Printers to Ethernet connections (TCP )

Printers that require USB connections and are currently connected to desktops in Zone 3 should gradually transition to Ethernet connections. 

Users can print from different locations with out physically connected printer 

- More secure (firewall protection)

- Queue Management

2. Printer Communications and Protocols

Implement IPP (Internet Printing Protocol) for networked printers, facilitating secure and standardized communication over IP networks. IPP is supported by most modern printers and provides features like encryption and authentication.

Ensure ZPL II-compatible label printers are used  and Print Management Microservice can generate and send ZPL II commands. 

3. Local Network Printing Solutions

To access printers on an  onpremise network

Establish VPN Connection to AWS to the network or via AWS direct Connect

Network Print server – Print server act as intermediary recovering print jobs from print microservice over VPN or Direct Connect and forwarding these jobs to isolated printers

4. Cloud printing Solution

Cloud Printing solution simplifies the architecture, it has direct IP printing/queue management, driver management. 

Reduces the need for on-premise print servers

Authentication, and authorization are integral parts to ensure that only authorized users can execute print jobs, and sensitive documents are handled securely.  Cloud printing provider will provide necessary API keys 

Cloud Printing Solution

With Cloud Printing service, on premise print servers may not be required.  Cloud   printing service typically manages queue management, job distribution and driver management

Options 

If Cloud  Services has to send print jobs directly to Printer behind firewall, then we need to open port 632 to allow incoming IPP traffic to onpremise printers and  VPN connection between cloud service abd Onpremise network 

Printers can be configured to make outbound connection to cloud provider over the network. In such case we just need to configure only the outbound rules

Local Network Printing Solution

API Gateway: Acts as the entry point for job submissions from the eLIMS UI.

Print Microservice: Processes the print jobs and interacts with SNS and SQS for messaging and queue management.

SNS/SQS Queue: Holds print jobs sent by the microservice. It manages the delivery of these jobs to the print servers in a scalable and fault-tolerant manner.

Local and Isolated Print Servers: Push print jobs from the SQS Queue and manage the actual printing process on their respective printers.

UI Printing

If there is no connectivity issues with the printers from the UI, then UI printing can also be used

Multi Level search in Microservice environment

 In a microservices architecture, data is typically distributed among different services, each with its own database. Implementing a multi-level search across these microservices can be challenging. The goal is to efficiently search and aggregate data from different microservices while ensuring optimal performance, scalability, and cost-effectiveness.

• Solutions

Selective Replication pattern:

In this approach, we replicate the data needed from other microservices into the database of our microservice.

• Selective Replication: Only replicate the data that is frequently accessed together. For example, if we often need to know the status of tasks associated with an instrument but not other details, only replicate the task status.
• Event-Driven Updates: data will be kept in  sync using an event-driven approach. For example, when a task is created or updated, it could publish an event that the Instrument service listens to and then updates its own records accordingly.

Example

Tasks Table: In the Instrument module, We could have a table that stores task details associated with an instrument. This could include the task ID, status, type, etc.

Pros

• Optimized Performance: By replicating frequently used data, we reduce latency and improve user experience.
• Reduced Network Calls: This approach minimizes the number of cross-service calls for common operations.
• Flexibility: We have the flexibility to fetch detailed data when necessary.

Cons

• Cross Service Calls The idea is to replicate only the most frequently accessed and critical attributes to optimize performance and reduce cross-service calls. However, for attributes that are less frequently used or for details that require a more comprehensive view, we might still need to make calls across microservices.

API Composition Pattern

In this approach, ApiGateway(or a composite microservice) aggregates data from multiple services and returns a unified response. This pattern is useful when we want to fetch data from different microservices in a single API call without the need for data replication. For Example -  Details of an instrument along with the tasks associated with that instrument

Flow

• The client requests details about an instrument, including its associated tasks.
• The request is received by the API Gateway or a Composite Microservice, which acts as an orchestrator.
• The API Gateway first calls the Instrument Management microservice to fetch details about the requested instrument.
• After obtaining the instrument details, the API Gateway calls the Task microservice to fetch tasks associated with the instrument.
• The API Gateway aggregates the instrument details and the associated tasks into a unified response.
• The aggregated response is sent back to the client.

Pros

• Single API Call: The client can fetch data from multiple microservices with a single API call, simplifying client-side logic
• Flexibility: The API Gateway can tailor responses to the specific needs of different clients.
• Decoupling: Microservices remain independent and can evolve without affecting the client-side code.

Cons

• Memory: It might not be suitable for complex queries and large datasets that require in-memory joins. 
• Complexity: The API Gateway may need to implement complex orchestration and error-handling logic.
• Dependency: If one microservice is down, it could potentially affect the entire operation.
• Error Propagation: Errors from one microservice need to be gracefully handled and communicated to the client.

Search Microservice

The Search Database typically contains a subset of the data from the other microservices, optimized for search operations. It may not have all the data but instead holds indexed, denormalized, or transformed data that is required to fulfill search requests quickly and efficiently.

• The microservices publishes events (SNS/SQS) when data changes.
• The Search Microservice subscribes to these events and updates its data in real-time or near-real-time.
• The Search Microservice processes and indexes the fetched data in a database

Flow

• A client (e.g., a web application) initiates a search request with specific criteria.
• The request may first go through an API Gateway, which routes the request to the appropriate microservice.
• The Search Microservice processes the search criteria and queries the indexed data.
• If necessary, the Search Microservice aggregates or transforms the data to match the desired response format.
• The Search Microservice sends the search results back to the client.
• The client displays the search results to the user.

Database Aggregator

A central database is used to aggregate and store data from different microservices, providing a unified point for querying and retrieving data.

Each microservice owns its data and performs its business operations as usual. Data from these microservices is then aggregated into a central database.

Event-Driven Updates:

• Microservices can publish events when data changes (Create, Update, Delete operations).
• The central database subscribes to these events and updates its records accordingly.

Search and Query:

• When a multi-level search request is made, the system queries the central database, which contains aggregated data from all microservices.
• The central database can be optimized for search operations, facilitating complex and cross-cutting queries.
• Query: The central database can also be  queried directly to fetch and search data.
  
  
  
• Pros
• Simplified Queries
• Performance
• Loose Coupling
• Cons
• Complexity
• Data Consistency:
• Storage Overhead

Two-Tier Authentication in Microservices Architecture

 

Microservices are decoupled, self-contained units, which makes security pivotal. Two-tier authentication can offer an extra layer of protection. By integrating both AWS Cognito (for cloud-based authentication) and FAMS (an on-premises solution), we can create a robust authentication mechanism for such architectures.

It's a clear separation of concerns, with FAMS focusing on user identity and Cognito securing your API. This is a valid and robust approach, particularly if you want to leverage Cognito's capabilities for managing API access without intertwining it with FAMS.

Two-Tier Authentication in Microservices

Microservices often communicate through APIs. The two tiers in this setup are:

On-Premises Authentication (e.g., FAMS): Before accessing cloud-based microservices, authentication through on-prem systems like FAMS ensures that the initial user or service is validated.

Cloud-Based Authentication (e.g., AWS Cognito): After the initial validation, Cognito facilitates the subsequent authentication steps, providing tokens that are required to access microservices' endpoints.

Benefits of Using AWS Cognito with FAMS

·        Seamless Integration: AWS Cognito integrates well with AWS services and can work in tandem with FAMS for initial authentication.

·        Token-based security: After initial authentication with FAMS, Cognito handles token-based authentication for cloud resources.

·        Flexibility: Offers the ability to switch between different authentication providers.

Sequence Flow in a Microservices Environment

 

Conclusion

Two-tier authentication using both FAMS and AWS Cognito offers a comprehensive authentication strategy for microservices, bridging on-premises systems and cloud architectures. It ensures that microservices are only accessed by authenticated clients and services, upholding the principles of security and integrity.