> ## Content Index > Fetch the complete content index at: https://madewithlove.com/blog/llms.txt > Use this file to discover other available public pages before exploring further. # Gradually migrating between mail providers in Laravel - URL: https://madewithlove.com/blog/gradually-migrating-between-mail-providers-in-laravel/ - Published: 2023-07-20T07:28:11.000Z - Updated: 2023-10-27T12:40:05.000Z - Description: Migrate between two email providers in a Laravel project with ease of mind while maintaining control. - Author: Michal Karnicki - Tags: Engineering, Architecture, Tutorials, Laravel Sending mail was at the core of our customer's business. A reliable email provider was necessary to send both transactional and non-transactional messages. Then, they needed to switch providers — and fast. The challenge was clear. I was tasked with routing our customer's mail traffic in a gradual manner from SendGrid to Mailgun. I had to do it in a way to ensure the IPs were warmed up so that we weren’t blocklisted by the service or network providers. To make it more interesting, the non-transactional type of mail was sent from a large number of different sending domains owned by our customer. How could I ensure proper mail delivery to the end users? ## The strategy for a mail provider migration in Laravel Because of the quickly approaching deadline, I was aiming for a simple solution that would be robust enough to support us in this transition period. When exploring options on how to approach the problem, I reviewed Laravel's `mail.php` config file. Among other entries, this file has some top level key-value pairs related to a sole SMTP configuration. If the `MAIL_DRIVER` environment variable was set to `smtp`, the SMTP configuration would be used. However, the updated format of that file has a `mailers` array that contains different mailer configurations, one of which can be selected using the `MAIL_MAILER` environment variable. A plan was forming, one where I could manually control which [Laravel](https://madewithlove.com/blog/tag/laravel/) mailer to use for the next outgoing mail based on assigned mailer weights. ## Step 1: Weights for mail providers Allowing for gradual control of how much mail has to be routed to one mail provider or the other necessitated assigning weights to these mail providers. So it felt very natural to leverage the existing `mail.php` config file `mailers` array and supplement it with a `weight` for each mailer. ```php use App\Mail\MailProvider; return [ 'mailers' => [ MailProvider::SENDGRID->value => [ // ... 'weight' => env('SENDGRID_MAILER_WEIGHT', 100), ], MailProvider::MAILGUN->value => [ // ... 'weight' => env('MAILGUN_MAILER_WEIGHT', 0), ], ], ]; ``` Now that each mailer could have a `weight` value assigned, it was a matter of making the mail go through a probabilistically chosen mailer. You can send mail [**via a specific mailer**](https://laravel.com/docs/10.x/mail#sending-mail-via-a-specific-mailer) like so: ```php Mail::mailer('mailgun') ->to($request->user()) ->send(new OrderShipped($order)); ``` We can use the `mailer` method to pass a mailer name, corresponding to a key from the `mailers` array in `mail.php`. Here's a class that we can use to pick a mailer: ```php namespace App\Mail\Mailer; use App\Infrastructure\Random\RandomInt; use Illuminate\Support\Collection; class ProbabilisticMailerChoice { public function __construct(private readonly RandomInt $randomInt) { } public function forDomain(string $domain): string { $possibleMailers = self::getMailersForDomain($domain); $maxRandomValue = $possibleMailers->sum('weight'); if ($maxRandomValue === 0) { throw FailedToChooseMailer::forDomain($domain); } $randomValue = $this->randomInt->fromRange(1, $maxRandomValue); $partialSum = 0; foreach ($possibleMailers as $mailer => $mailerConfig) { $weight = $mailerConfig['weight']; if ($randomValue <= $partialSum + $weight) { return $mailer; } $partialSum += $weight; } throw FailedToChooseMailer::forDomain($domain); } private function getMailersForDomain(string $domain): Collection { return collect(config('mail.mailers')) ->filter(fn ($mailer) => key_exists('weight', $mailer) && $mailer['weight'] > 0) ->sortByDesc('weight'); } } ``` This class has a `forDomain` method, which takes the domain name as an argument. For transactional mail, it would be the apex domain of our customer. But for non-transactional mail, there was a large pool of possible values to be passed. All these domains had to have DNS configured correctly to work as sending domains; more on this later. First, we retrieve the `mailers` from `mail.php` config, those with a defined weight (and the weight must be greater than 0). This means it's as easy as defining a (positive) `weight` on one of the `mailers` to include it in the pool. We then return them sorted in descending weight order. Formally, probability is expressed as a number in the range from 0 and 1\. However, our weights were simply integers, usually (but not necessarily) adding up to 100 across all mailers which defined a weight. Next, we sum the weights and make sure the sum is larger than 0\. Otherwise we can't reasonably choose a mailer to use, so we throw an exception in that case. Next, we pick a random integer in the range from 1 to {sum of weights} and assign it to a variable, `$randomValue`. Finally, we iterate over the pool and add the weight of the current mailer. When we exceed the `$randomValue`, we know we found **the** mailer to use and we return its name, which would in turn be passed to the `Mail::mailer` method. Because this code only depends on a config file value (read by the framework when it boots) and a random number, it executes nearly instantaneously. To be able to unit test the `ProbabilisticMailerChoice` class, we had to decouple from the element that provided the randomness, so we inject an instance bound to the following interface in the container: ```php namespace App\Infrastructure\Random; interface RandomInt { public function fromRange(int $min, int $max): int; } ``` Because of this slight abstraction, we can easily mock the `RandomInt` in our unit tests and have deterministic results. Here's the implementation we use in production: ```php namespace App\Infrastructure\Random; use Exception; class NativeRandomInt implements RandomInt { /** * @throws Exception */ public function fromRange(int $min, int $max): int { return random_int($min, $max); } } ``` Here's a sample `mail.php` config content before we started the migration. Note the weight keys added to the mailer configurations: ```php use App\Mail\MailProvider; return [ 'default' => env('MAIL_MAILER', MailProvider::SENDGRID->value), 'mailers' => [ MailProvider::SENDGRID->value => [ 'transport' => 'sendgrid', // ... 'weight' => env('SENDGRID_MAILER_WEIGHT', 100), ], MailProvider::MAILGUN->value => [ 'transport' => 'mailgun', // ... 'weight' => env('MAILGUN_MAILER_WEIGHT', 0), ], // ... ], ]; ``` This also relies on environment variables to set the values. You can use `SENDGRID_MAILER_WEIGHT` and `MAILGUN_MAILER_WEIGHT` to start gradually moving your transactional mail from one provider to another at a reasonable pace for your total sending volume. ## Migrating non-transactional emails using Laravel The non-transactional mail was just a tad more involved. As mentioned earlier, we had a large pool of sending domains to support. Part of the migration from SendGrid to Mailgun required other code which was responsible for configuring the DNS for these domains so that Mailgun could use them (SPF and DKIM for outbound mail, then MX for inbound mail). One fact related to domain configuration is worth mentioning: while you can have two mail providers configured for outbound mail for a given domain, you can only have one of the mail providers configured for inbound mail. Why was this case more involved? Because in Mailgun, SMTP credentials are created via their API for each domain separately, so we can't easily just put them in `mail.php` and call it a day. They have to be injected dynamically. For that reason, the transactional mail and non-transactional mail were sent from two completely separate services (which happen to live in two separate repositories). I decided to keep it simple and allowed myself to reuse this implementation (a.k.a. copy-paste) but freely supplement it to satisfy requirements. There was only one service that sent non-transactional mail, so creating a composer package (shared code!) did not make sense. Here's the "upgraded" implementation of the same class in the second service. We'll go through it together below. Do note that, while developed in test-driven style, I made some reasonable shortcuts in terms of code design due to the short deadline. ```php sum('weight'); if ($maxRandomValue === 0) { throw FailedToChooseMailer::forDomain($domain); } $randomValue = $this->randomInt->fromRange(1, $maxRandomValue); $partialSum = 0; foreach ($possibleMailers as $mailer => $mailerConfig) { $percentage = $mailerConfig['weight']; if ($randomValue <= $partialSum + $weight) { if ($this->mailerNeedsSmtpCredentialInjected($mailerConfig)) { // New condition here. $this->injectMailerSmtpCredentialsForDomain($mailer, $domain); } return $mailer; } $partialSum += $weight; } throw FailedToChooseMailer::forDomain($domain); } private function getMailersForDomain(string $domain): Collection { $providersWithTheDomainVerified = DB::table((new DomainSetupStatus())->getTable()) ->where([ 'domain_name' => $domain, 'outbound_status' => DomainVerificationStatus::VERIFIED->value, ]) ->pluck('provider'); return collect(config('mail.mailers')) ->filter(fn ($mailer) => key_exists('weight', $mailer) && $mailer['weight'] > 0) ->filter(fn ($mailer, $key) => $providersWithTheDomainVerified->contains($key)) // New filter here. ->sortByDesc('percentage'); } /** * @throws FailedToInjectSmtpCredentials */ private function injectMailerSmtpCredentialsForDomain( string $mailer, string $domainName ): void { // We could have a repository factory here, and internally // use the Eloquent model for the query. I think it's a bit // overkill at this point, considering this code is only called // when credential injection is required, so this service only. $table = sprintf('domain_%s_setup_trackers', Str::lower($mailer)); $result = DB::table($table) ->select(['smtp_username', 'smtp_password']) ->where('domain_name', $domainName) ->first(); if (empty($result->smtp_username) || empty($result->smtp_password)) { throw FailedToInjectSmtpCredentials::forDomain($domainName); } $usernameKey = sprintf('mail.mailers.%s.username', $mailer); config([$usernameKey => EncryptionService::decrypt($result->smtp_username)]); $passwordKey = sprintf('mail.mailers.%s.password', $mailer); config([$passwordKey => EncryptionService::decrypt($result->smtp_password)]); } private function mailerNeedsSmtpCredentialInjected(array $mailerConfig): bool { return key_exists('inject_credentials', $mailerConfig) && $mailerConfig['inject_credentials'] === true; } } ``` In the first line, a call to the `getMailersForDomain` method, we get a list of providers from a database that have our specific domain configured (the VERIFIED status). Then we filter the configurations by weight (positive number) and our verified list (matching a key from the `mailers` array). That is the reason why, as mentioned earlier, the `weights` don't have to add up to 100\. If there are 3 providers with weights (50, 30, 20), and the last provider isn’t configured for this particular domain, we need to pick a random number in the range of 1...80. In the main method, `forDomain`, while finding our mailer, we also check if an `inject_credentials` configuration key is also defined and has a value of true. If so, we dynamically inject the SMTP username and password into the mail config, based on the credentials retrieved for the domain from the database. This way, it’s possible to ensure that mail goes through a correctly configured mailer with a simple one liner: `Mail::mailer(MailerChoice::forDomain($sendingDomain))->...` Note that `MailerChoice` here is simply a Laravel facade pointing at an instance bound to the `MailerChoice` interface in the container. You're free to use dependency injection in your code to call it like so instead: `Mail::mailer($mailerChoice::forDomain($sendingDomain))->...` ## Improving Laravel mail providers in the future A fun exercise would be to automate the migration process based on specific criteria (e.g. "no more than x number of mails sent for a given domain via a given provider per day"). This would allow for a transition between mail providers over a designated period of time without human intervention. That's it! Hopefully this article has inspired you with an idea how you can gradually move across mail providers in a controlled fashion as well as how to dynamically inject credentials into the Laravel `mail` config when a sending domain requires a unique set of credentials.