Understanding application flows

These flows allow the proper functioning of any computer system; they must be rigorously packaged in order to guarantee a renewed execution of the processes they involve.

‍

‍

Data security and format

‍

Application flows therefore allow the exchange of data between different places, for example the product catalog of a retail site is managed by a flow that searches permanently or programmed in a database for the items that are present in this catalog.

By convention, the data transferred is in JSON format, more rarely you can use XML, TXT or CSV formats.

The security of this data is conditioned by the security of the flow itself, so care must be taken to accurately authorize the accesses and permissions granted to a third party concerning a given flow. The security mechanisms can be:

• by IP address recognition: allows you to authorize only the IP of one or more servers to use a developed API.
• by various authentication mechanisms using individual “keys”.
• by more secure protocols to guarantee the authentication of a specific user (e.g.: Oauth 2, JWT)

‍

Useful lexicon

‍

To make sure you understand the explanations provided in this article, let's take a quick look at some of the inerrant terms on the subject.

• FTP or SFTP: It is a file transfer protocol, this term also simply refers to the platform used to manage files on the remote server where they are stored.
• S3: It is a protocol similar to FTP but more modern because S3 is based on the cloud, so the data is more secure because it is replicated in several areas and not only stored physically in one place like those of a traditional remote server used by an FTP protocol. And an SDK is available for developers to easily interact with the files.
• CRON: It is a program that allows you to configure the scheduling of scripts, and therefore automated tasks, to be executed at specific times.
• API: Programming interface allowing the exchange of information between several systems.
• Forehead: User side interface of an application.

‍

The types of flows and how they work

‍

These examples are based on a situation where we would need to retrieve a list of items.

File uploading by FTP or SFTP

In terms of the basic architecture of a flow, it is easily understandable: an application queries the SFTP at regular intervals using a CRON where the data is recorded in order to detect changes in existing data or new data. Once this data is retrieved, the API processes it and stores it in a database.

On the user side, the Internet user will connect to a site, for example an online store, their browser will query the API to retrieve the list of items, the API therefore displays what they find in their database.

This type of flow architecture is the simplest that exists. It has the advantage of making it master of its data, however it does not allow to have data in real time and will overload the network by perhaps running a CRON unnecessarily if new data is not present in the SFTP.

‍

The file repository by S3

‍

A file repository by S3 represents a slightly more advanced architecture. The data present on the SFTP will be synchronized in the cloud, if a change is observed in the cloud, an application function will create a route to the API which will record this change in its database.

‍

For the user, no change has been observed, it is simply the path taken by the information that is changing more efficiently.

Unlike a repository by SFTP, the repository by S3 makes it possible to avoid network overload, to have data updated in real time while remaining in control of your data.

‍

Calling other APIs

‍

In some cases where we are not in control of the data we need, we can design an API that requests the retrieval of the list of items from third-party APIs that would follow a traditional path developed above.

For example, an e-commerce site such as a “marketplace” that groups together catalogs from different providers will have an API that will call those of its partners to display their items in its catalog.

This system allows you to have up-to-date data all the time and is very micro-services oriented, it is however totally dependent on the functioning of third-party APIs.

‍

The Queuing message

‍

In order to replace this dependency on third party APIs, we can use the Queuing message. This refers to a channel in which an application drops a “message”, a bit like a kind of inbox, when it has a new article in its database. This message can be retrieved by another system. So, even if the third-party system no longer works, our API can still retrieve its data and store it in our database.

‍

This method therefore makes it possible to have data that is always up to date, without overloading the network and without depending on the operation of the third-party API.

‍

Orchestration and flow monitoring

‍

When a system has several flows that operate simultaneously, it is necessary to program their execution in a well-defined order to ensure the consistency of the information.

For example, in our article system, if a second feed exists to retrieve a “category”, if I want to retrieve an article and attach it to a category that does not exist in my data, my system will encounter a problem.

In case of problems, you must also always make sure you have tools that trace the various information on the good or bad execution of flows and ideally have simple mechanisms to restart them in case of errors.