Nowadays, we take cloud technologies for granted, but should we really? We have seen many projects that lack well-defined and established processes and infrastructure, but it doesn’t stop you too much as long as you have an army of managed services provided by one of the major cloud providers.
But what if it’s not the case? What if your options are limited, yet you have requirements and regulations to be followed? That’s what happened with us and one great KSA-based FinTech startup. You are about to hear about the peaks and valleys of this great journey, what obstacles we encountered, and how we have passed them. So please, take your seat!
Navigating the cloud landscape with limited options
Let’s take a look at what we had once we started. Well, you guessed it right - not too much. Microservices were scattered across different services and providers, some of them deployed manually. To be more specific, the main front end was running in Cloud Run service somewhere in Europe. The backend was running in a reserved virtual machine in Sahara Cloud (local KSA cloud provider that simply resells Azure Stack Hub), and a few other little things for which Firebase was used. Oh yes, and the MongoDB Atlas cluster didn’t have any KSA region back then.
Ok, so you have an idea what was going on. Considering all the mess, we started with planning. The first thing to consider was a cloud provider with a KSA region. Unfortunately, Sahara Cloud, as mentioned earlier, isn’t suitable - it is quite expensive, and Azure Stack Hub provides you with basic services only. Both Azure and AWS don’t have regions in KSA and thus were not considered. The only viable option we had was GCP - by strange and extremely convenient coincidence, they had recently opened the me-central2 region in Dammam.
But starting using GCP wasn’t as straightforward as you would expect. Registration was only available through the CNTXT reseller. Even though the region was formally operational, for KSA citizens, it wasn’t the case - CNTXT was expected to start online sales in a couple of months. Luckily, the CNTXT team was extremely open, so we had a way to register an account earlier.
But we are not there yet. As it usually happens in new regions, not all services are available. Not only that, but some basic resources may also be scarce; thus, sometimes, we can’t even reserve an external IP address in me-central2. But it has been getting more stable every week, and I doubt you will have the same issues we had. Nonetheless, it was time to make some decisions regarding architecture. Considering the microservices nature of the project, not an overwhelming amount of these microservices and services Dammam region could provide. It was either a bunch of Cloud Run services or a GKE cluster.
We decided to stick with Cloud Run due to its simplicity. As a plus, one of the services was already running in Cloud Run, which meant we had less work to do. Preparing other services wasn’t an issue, so we moved them rather quickly. Ok, but what’s with the database? Dammam region doesn’t have a managed service for MongoDB yet, so our only option was self-hosted MongoDB. It was not the most elegant solution, but the only option considering regulations. The Damma region also lacks Cloud Functions, but fortunately, existing functions were not operating with data itself, so it was safe to put them in Europe.
Streamlining development and deployment processes
Ok, now it’s clearer what direction for architecture was chosen, but I also mentioned some processes that, let’s say, used to lack maturity. The new development team is extremely pedantic when it comes to how its product has to be developed, built, and deployed, what stages for testing and reviews should take place, etc. Thus, reactive manual releases, as they used to be, were abandoned completely. Instead, we developed a robust CI/CD that addresses developers’ workflow in the best way possible.
It might look simple, but it has meaningfully increased the number of feature releases. Now, we have four environments compared to two prior to the changes. A dev environment is a convenient sandbox environment that lets you continue working on new features while a bundle of other features are being tested in QA. Once new features are approved in QA, they are promoted to the staging environment. Think of a staging environment as one that is as close to production as possible. Once staging is approved, a new release can be moved to production. It's a standard yet powerful and flexible workflow with some space for more improvements.
Great, it looks pretty solid - we know how and where to ship services, but what’s happening with observability? Due to the managed-service nature of infrastructure, we decided to stick with native GCP logging and monitoring solutions, which turned out to be a great solution. We had to install and configure Ops Agent on our MongoDB servers and configure alert policies. All the alerts are sent via both Telegram and Slack for extra reliability. A few alerts later, we started to see some patterns in how the application works, which can help with later improvements and optimization.
Even though we described the main issues and solutions, it was still quite a broad overview of what was done. Many things were behind the scenes and somehow committed into a pool of challenges. For instance, not too polished GCP provider for terraform, many custom automation due to lack of managed service, and many more things. Yet we are not complaining - scarcity of options leads to creativity and new paths to be discovered. Thus, we expect to grow this project and push all the available options to the absolute limit, optimizing every bit and corner of our infrastructure. Stay tuned for more updates, and if you are striving to build a reliable platform based on best practices that will meet the challenges of your business, do not hesitate to contact ITSyndicate; we are dedicated to ensuring a smooth and fast development process.
Cut your Kubernetes cloud bill with these 5 hacks for smarter scaling and resource tuning
PostgreSQL blends relational and NoSQL for modern app needs
Mutable vs immutable infra key perks drawbacks and Terraform hacks