Digibee Integration Platform architecture on Dedicated Saas model

Learn about the Digibee Platform architecture for Dedicated SaaS customers

The Digibee Integration Platform runs on a Kubernetes cluster that is managed by major cloud providers such as Google, AWS, and Azure. This architecture documentation aims to outline the key components and objectives of the platform, including, but not limited to, high availability, scalability, and logging.

High Availability

The Digibee Integration Platform is designed to ensure high availability by using availability zones. Availability zones are physically separate locations within a cloud provider's data center that provide independent power, cooling, and networking.

By using availability zones, the Digibee Integration Platform ensures that the cluster remains available even in the event of a single zone failure. This is achieved by distributing the cluster's resources across multiple availability zones and using load balancers to distribute traffic and k8s placement strategies to distribute resources.

In case of a zonal failure, Digibee implements automatic failover in all its applications, ensuring the Digibee Integration Platform remains available to users. This availability strategy ensures that the resources are used efficiently when scaling to a high volume.

Different cloud providers may treat availability zones differently.

Scalability

The Digibee Integration Platform is designed to be scalable, allowing it to handle an increasing number of users and requests. This is achieved through the use of Kubernetes and the separation of base components and applications.

Base components

The base components of the Digibee Integration Platform, such as the Kubernetes cluster, are designed to be horizontally scalable. This means that more resources can be added to these components as needed to handle increased traffic.

The Kubernetes nodes, which are the individual machines or virtual machines that make up the cluster, can scale based on different criteria. One common method for scaling is through the use of an autoscaler. An autoscaler is a Kubernetes controller that automatically increases or decreases the number of nodes in a cluster based on predefined rules and thresholds.

The autoscaler can be configured to use different metrics like memory, network, and other custom metrics to trigger the scaling of the cluster.

This approach allows the platform to be highly available and responsive to users, while also minimizing costs by only using the resources needed to handle the current traffic.

Applications

The applications running on the Digibee Integration Platform are also designed to be horizontally scalable. Kubernetes is used to automatically spin up new instances of an application as needed to handle increased traffic. This is achieved through the use of a feature called Horizontal Pod Autoscaler (HPA).

HPA is a Kubernetes controller that automatically increases or decreases the number of replicas (instances) of a pod (a group of one or more containers) based on predefined rules and thresholds. The HPA controller continuously monitors metrics like CPU and memory usage of the pod and based on the defined rules, it will increase or decrease the number of replicas.

This allows for the platform to automatically adjust its resources to meet the demands of the applications running on it. In situations of high traffic, the HPA will increase the number of replicas to handle the additional load. This ensures that the application remains responsive to users even during periods of high traffic.

Logging

The Digibee platform includes a centralized logging system that allows for the collection and analysis of logs from all components of the platform. This allows for easy debugging and troubleshooting.

The logging stack is divided into two main categories: platform logs and pipeline logs.

Platform logs refer to the logs generated by the infrastructure that we maintain, such as Kubernetes and the database. These logs are used for monitoring and alarm purposes, and we use Prometheus and Datadog for this purpose. Prometheus is an open-source monitoring system and time series database that allows for the collection and analysis of metrics from different sources. Datadog is a monitoring and analytics platform that allows for near real-time visibility into the performance of the platform.

Pipeline logs are stored in Elasticsearch, which is a search and analytics engine that allows for the indexing and querying of large amounts of data. These logs provide a simplified and objective view of each transaction and can be accessed through the platform's interface.

The logging system provides valuable insights into the performance and behavior of the platform, allowing for quick identification and resolution of any issues.

Elasticsearch is not accessible outside Digibee Integration Platform infraestructure.

Digibee Integration Platform components

The Digibee Integration Platform ecosystem is built with a variety of components, including both off-the-shelf and custom-built components. These components work together to provide a stable and responsive experience for users while also being easily customizable and adaptable to the needs of each organization.

The following components are distributed across namespaces to provide high availability for the Platform. The namespaces are organized to ensure that each component is isolated and can function independently of the others. This allows for easy scaling and maintenance of the platform, as well as the ability to quickly identify and resolve any issues that may arise.

• Off-the-shelf components:

  • Redis: in-memory data structure store used as a cache. It stores data in a key-value format and is used to speed up data retrieval.

  • Minio: open-source object storage server used for storing static files. It allows for the storage of large amounts of data and is easily scalable.

  • Kong: open-source API gateway used to manage and secure API traffic. It provides features such as authentication, rate limiting, and monitoring.

  • Postgres: open-source relational database management system used by Kong and FusionAuth.

  • FusionAuth: open-source authentication and user management platform used to handle user authentication and authorization.

  • ElasticSearch: search and analytics engine used to index and query large amounts of data.

  • MongoDB: open-source document-oriented database used to store and retrieve data in a flexible and scalable way.

  • RabbitMQ: open-source message broker used to send and receive messages between different components of the Platform.

• Custom-built components:

  • Triggers HTTP: custom-built component that processes HTTP requests.

  • Triggers HTTP-File: custom-built component that processes files via HTTP requests.

  • Triggers REST: custom-built component that processes REST requests.

  • Triggers Email: custom-built component that processes emails.

  • Triggers Scheduler: custom-built component that processes scheduled tasks using cron.

  • Portal: custom-built UI for users.

  • Coordinator: custom-built backend for administration APIs.

  • Controller: is a custom-built backend that controls and creates objects within Kubernetes.

Digibee Platform Communication Process Sequence

The following diagram provides a graphical representation of the sequence of operations and communications during the installation and upgrade of the Digibee Integration Platform.