> ## Content Index > Fetch the complete content index at: https://madewithlove.com/blog/llms.txt > Use this file to discover other available public pages before exploring further. # Cloudy with a chance of function calls - URL: https://madewithlove.com/blog/cloudy-with-a-chance-of-function-calls/ - Published: 2025-11-14T11:32:37.000Z - Updated: 2026-06-02T12:31:29.000Z - Description: In the first of a series exploring infrastructure fundamentals, Brenden addresses the most frequently asked questions about what's really happening under the hood with complex pipelines and AI/data systems, bringing the cloud to life. - Author: Brenden Cambier - Tags: AI, Tutorials, Infrastructure, AI development, Refactoring, Technical debt, Engineering, Architecture, Tooling I frequently get questions about what's really happening **under the hood** with [**complex pipelines and AI/data systems**](https://madewithlove.com/blog/integrating-ai-into-your-engineering-team/). I'm planning a series of posts to address this. ## Let's [talk about infrastructure](https://madewithlove.com/blog/there-is-a-time-and-place-for-everything-managed-services-vs-diy/) Before we dive into code and architecture choices, I think we need to talk about **infrastructure**. I've written about this in the past as well, start-up full-stack developers these days are also expected to dabble a bit in the **hosting** part of the application. ## Cloudy: the public cloud The public cloud is like IKEA for developers. Everything looks **simple**, but I wouldn't recommend it for couples therapy. The public cloud is a **mystical beast** for some, but it's the home for what they call **cloud-native applications**. For start-ups, there's a sweet spot to be found here for several reasons. **1 Money.** Every start-up needs its cash. Luckily, the cloud companies are fighting for your wallet like Tinder dates offering free drinks. It's not uncommon for a start-up to host for **free** for the first 2 years using **credits** and free tiers. Although that might sometimes mean to burn 10k of credits using the most expensive servers, just to get the new tier of credits (looking at you [Azure](https://azure.microsoft.com/)). **2 Managed.** Every start-up needs its **cash**. Remember when you needed to pay an Ops person to configure SSL, backups, and database clusters? Now it’s a **dropdown menu**. The cloud will do it for you. The only thing it won’t manage is your **expectations**. **3 Money & managed.** Wait, I just covered those. Yes, but **combined**, they make even more sense because the managed services can be significantly cheaper than the **vanilla** services. ## The cloud-native mindset Cloud-native requires a certain method or mindset. I've seen companies try to lift and shift their **on-prem** application to the **cloud** and expect a spiritual awakening. Afterwards, they complain that they pay **twice as much** for the same pain in a shinier console. It’s like upgrading from a Renault Twingo to a Ferrari, but insisting they keep the Twingo’s engine. Congratulations, you’ve built the world’s loudest lawnmower. That’s not a cloud migration, that’s a **midlife crisis** on wheels. ## With: containerization **With Docker**. It’s the seatbelt of **deployment**. Optional until the crash. If you’re not using Docker yet, I want you to go outside, take a long walk, and rethink your life choices. And that means something coming from a Python enthusiast. **Python** developers love their **virtual environments**. However, I'm here to say that **Docker** will help any start-up playing the **cloud game**. Create a new service? Dockerise it. **Always**. Docker gives you **portability** and runtime **sanity**. It’s your golden ticket out of vendor hell. Azure gives you a headache? Move your container. **Kubernetes** gives you an aneurysm? Move your container. Your local dev machine is a Mac, your server is Linux, your intern runs Windows on a microwave? **Docker** doesn’t care about your life's choices. **It just runs**. ## A chance: on a hands-off cloud Not a gift. The cloud gives you a chance to **focus** on the things that matter, like the features in your code, while it takes care of the specialised stuff like **scalability**, **high availability, security, data integrity**, and some other -y's. With great power comes great **invoice** detail. The cloud is notoriously known for its gateway to incur **crazy bills**. There's budget alerts, but there are not always hard stops. **Wrong configuration** can lead to immense costs. But that's not something that happens overnight, and you have to really **ignore all the warning signs** before reaching that. ## OF OF. Oh f\*ck. The database is down again. Before the age of the cloud, I remember my OF moments. Whenever the application went **down**, there was a 90% chance that it was due to the database. The database **wasn’t a service**; it was a personality test. We tried read replicas, great in theory until replication lag turned every query into a Russian roulette. We tried multi-primary clusters, any node could take writes, which just meant all of them were on fire at the same time. In the end, I had to realise we were duct-taping hope onto scaling issues. Now, I press a few **buttons** in the managed service menu, and I get the database I want, with all the **scalability** mentioned before, taken care of by the best industry experts in the world. - **For SQL databases** you can look at Amazon RDS, Azure SQL Database or Google Cloud SQL. - **For caches** like Redis; Amazon ElastiCache (also Memcache), Azure Cache for Redis, Google Memorystore for Redis/Memcache; are your friends. - **For files** there's Amazon S3, Azure Blob Storage and Google Cloud Storage. - **For documents** there's Amazon OpenSearch (ElasticSearch), Amazon DocumentDB (MongoDB), Amazon Keyspaces (Cassandra), Azure Managed OpenSearch (preview), MongoDB Atlas on Azure, Azure Managed Instance for Apache Cassandra, Google Elastic Cloud, Google MongoDB Atlas and Google AstraDB (Cassandra). - **For message queues** there's Amazon MSK (Kafka), Amazon MQ (RabbitMQ), Azure Managed Kafka, Azure Managed RabbitMQ, Google Confluent Cloud (Kafka) and Google Cloud RabbitMQ. - **For data pipelines** there's Amazon Managed Workflows for Apache Airflow, Amazon Glue (Spark), Azure Data Factory Managed Airflow, Azure Databricks (Spark), Google Cloud Composer (Airflow), Google Dataproc (Apache Spark, Hadoop, Flink). Those are all generic, but **managed services** giving you the same portability advantage as our **Docker yields**. You can use it and port it somewhere else. Now it's time to make things more complex. Each of the big players offers even cheaper, better, stronger, specialised managed services, but with [**vendor lock-in**](https://madewithlove.com/blog/there-is-a-time-and-place-for-everything-managed-services-vs-diy/). Removing the portability advantage, but giving an advantage in **time to market**. For example, Amazon DynamoDB, Step Functions, Redshift, SQS, Sagemaker, and Lambda. Azure CosmosDB, Data Factory, Event Grid, Service Bus, AI Search, Functions. Google BigQuery, Dataflow, Vertex AI and Cloud Functions. How do you make that puzzle? First of all, it's a business decision mainly related to **compliance**. Does the cloud provider offer the **regions** you need? Or do your customers have a veto for Microsoft? Enterprise love for Azure is a form of Stockholm Syndrome. But now with Teams integration for the kidnapper's daily stand-up. Hell just became cloud-ready. In many cases, it may be decided to stick to one (or two for specific services) **cloud provider(s)**. Just add Kubernetes for extra pain. Eternal suffering scales horizontally. ![Just add Kubernetes for extra pain. Eternal suffering scales horizontally.](https://madewithlove.com/blog/content/images/2026/05/devil.jpg) Just add Kubernetes for extra pain. Eternal suffering scales horizontally. Some vendor-specific services offer **Docker** (I told you to use it before, right?) **support**. For example, AWS Lambda and Sagemaker are very vendor-specific, but both can use your codebase through a Docker image, lowering the portability disadvantage by A LOT. Time to market. For start-ups **speed** is most important. Setup that AWS Step Functions, get your product out there. If you need to **migrate** away from it later, that's still a fair price to pay for the speed it has offered you before. ## Function calls: serverless FaaS We've talked about data storage solutions, but what **compute** options are there? **Serverless**. For start-ups, proof of concepts, MVPs, expeditions, start looking at serverless compute using **Docker images** (yes, again). You don't have to think about auto-scaling, container pods or other forms of digital self-harm. Upload the image, spin it up (AWS Lambda, Azure Functions, Google Cloud Functions) and the cloud provider will **scale** it for you. Cheap. To put cost into perspective: I recently created a multi-step pipeline (5 steps per bird) to ingest and process all 11000 birds of the world using AWS Lambda & Step Functions. Lambda used over a million gigabyte-seconds. After the monthly free-tier discounts, I paid $11 for Lambda and $2 for Step Functions. Yes, the internet is full of people complaining about serverless. - "Serverless has limitations" - "Memory capped" - "It's 1 function per instance, it's micro-service hell" - "Cold starts" - "No state" hurr durr. But those are just the kind of people who still name their servers after Pokémon. Yes, serverless is often **capped,** for example, 10GB memory per invocation. But I have scaled big data operations over AWS Lambda. I can do a lot with 10GB of memory. Yes, **serverless is stateless**. Stateless is **not a limitation**; it’s a lifestyle choice. State is where bugs go to retire. Yes, there's cold starts. But if you can develop a bit, you can make your Docker **lightweight** and only a fraction ever experiences a cold start. No, it's not a micro-service tool. You can fit an entire [FastAPI](https://fastapi.tiangolo.com/) app into a single function (hello [Mangum](https://pypi.org/project/mangum/)). Spin it up on AWS Lambda, Azure Functions, or Google Cloud Functions. Your Docker image (yes, Docker again) becomes a **portable**, **stateless** little **miracle**. Eventually, you’ll grow. You might **outgrow** serverless, just like you outgrew your monolith, your ORM, and your optimism. That’s fine. Take your same Docker image and throw it at a **container** service like ECS or whatever managed Kubernetes flavour (EKS/AKS) if you're a masochist. Just remember: the more **configuration** options a cloud service gives you, the more chances you have to ruin it. The cloud is both the best and worst thing to happen to startups. It gives you infinite power and infinite ways to shoot yourself in the foot. Use it wisely. Pick **managed services** where you can. Dockerise everything. And when someone tells you their on-rem, custom-created orchestration, AI pipeline is production-ready; smile kindly and prepare your “OF” face. ## Ready to bring AI into your engineering team? Infrastructure is only one piece. Our guide covers how to evaluate, integrate, and govern AI tools across the engineering organisation. [Read the AI integration guide](https://madewithlove.com/blog/integrating-ai-into-your-engineering-team/)