Email is an open system, right? Anyone can send a message to anyone... unless they are on Gmail!
- vmailer1.vig.co.nz 69.48.142.156
- vmailer3.vig.co.nz 190.92.153.150
It goes without saying that our messages are not spam, and we do everything right to prevent them ending up in spam folders. Both email servers have PTR records set up, and SPF records only allow email emanating from these two servers. Every message is cryptographically signed with DKIM, and DMARC records tell receiving servers to apply all these restrictions strictly.
Neither server is a open relay, and both send only our messages - no-one else is sending spam from these servers.
All this effort pays off. We check our servers aren't on spam blacklists every day, and our Google Postmaster Tools account shows practically perfect scores on all metrics:
Can anyone hear a "but" coming?
But... Gmail is rate-limiting our messages daily.
Those blue dots show over 3,500 Gmail customers having the booking confirmation email they asked for delayed by up to 12 hours. Our support people get several calls a day asking about missing confirmation emails, and they wearily explain that Gmail is delaying delivery for no good reason.
Our system has detected an unusual rate of 421-4.7.28 unsolicited mail originating from your IP address. To protect our 421-4.7.28 users from spam, mail sent from your IP address has been temporarily 421-4.7.28 rate limited.
"An unusual rate of unsolicited mail"? We've established we don't send spam, and Google's own Postmaster Tools can't detect any spam, so it seems to me that Gmail's detection algorithm is broken.
And it has been broken for a while, because we've been rate-limited for months. We've done everything we can to alert Google to the problem, and this blog post is a last-ditch attempt to get the message across. If you know anyone at Google, please send them a link.
Wait, I think I hear another "but"!
BUT... Problems like this are killing independant email.
This rate-limiting issue is not the only thorn in the side of email server admins. If you search the forums, you'll come across hundreds of reports of the big three email providers (Google, Microsoft and Yahoo) making email delivery so difficult that independant email servers are becoming untenable - keeping messages flowing is just too hard.
And this is happening after SPF, DKIM and DMARC provided a solution to the spam problem.
Any mail system can remove practically all spam by insisting messages conform to those three standards, so Gmail could - and should - be accepting our squeaky-clean messages without restriction.
Instead, the big three are making independant email delivery so difficult that we give up and move our inboxes to their services, and use one of their partners (like MailChimp or SendGrid) to deliver our system-generated messages.
I really hope I can write a follow-up saying that fixing the rate-limiting problem proves that Gmail is still committed to the open email standard. The alternative is that email will be subsumed into giant corporation's proprietary systems, and the wonderfully open and extensible message service we've enjoyed since the dawn of the internet will be gone.
Well.... welcome to the club. I have been setup independent email server for people for the last 10 years and making EVERYTHING you mention to be out of the spam jail but Google will ALWAYS put message in the spam folder for no reason. Microsoft do the same btw.
ReplyDeleteAt least until web servers start refusing creating an account from all 3 MMP (mafia mail provider), this isn't going to change. Keep all the evidences that they acted in an unethical, not respecting standard and adding an obvious commercial burden on your standard email service, so when they'll fight back, you are covered.
ReplyDeleteThe message on your sign up page could be "[ ] I understand that Google GMail is unreliable with unexplained 12H offset for mail delivery. And I still want to use that rubbish email service to sign up. YOURSERVICE can not be claimed for any damage due to a third party service not respecting email's delivery standard."
I'm pretty sure that, as soon as theses messages contaminate the internet, Gmail will change its policy.
The saddest aspect of this situation is that the malicious behavior is not a targeted attack on independent mail servers, but rather a result of the ignorance and carelessness of the employees involved. This lack of knowledge and thoughtlessness is a direct consequence of belonging to an enormous organization with abundant resources and numerous clients. Regrettably, this issue seems nearly insurmountable unless there will be a way to run a free email service provider with an exceptional user interface.
ReplyDeleteI don't give the benefit of the doubt to Google anymore. It has been proven already that they are willing to prey on open standards
DeleteIf you're EU resident please consider pursuing another Google fine for monopoly behaviour.
ReplyDeleteTake them to court...
ReplyDeleteI read this in Hozier's voice.
DeleteI'm a System Engineer who has set up email systems for many years. These issues are just as you describe, and have been longstanding issues. The decision makers at google have no visibility or feedback mechanisms by design, about the only thing you can do is take google out of the equation, file FTC complaints, and reach out to congressional representatives. Seeing as its a school, I don't see a class action happening. Setup a server you control that has notifications after logging in, and have the parents check and confirm. Free email service has been made unreliable.
ReplyDeleteIt all reminds me of Embrace Extend Extinguish. Emails have been embraced by corporatocracy, and then turned into walled garden.
ReplyDeleteThis is interesting, but instead of drawing the conclusion that google is destroying an open standard by doing so, consider the amount of nefarious sh*t google has to deal with on a regular basis. A 12h delay on a small school email, compared to tools made to deal with bots from all over the world, is almost a miracle. When you’re equipped to deal with the same amount of nefarious traffic, you can still cant make the argument, because they havent changed the standards
ReplyDeleteFound the boot licker!
DeleteThis is 100% by design, there are other examples of this behaviour from Google, see the case of xmpp
DeleteYou're writing that last section as if that weren't exactly what Google and Microsoft work towards.
ReplyDeleteEverything you said is right, but your mailers are on a shared hosting network. Those server for sure do not spam, but the IP or hell, the AS, may exist in some Google naughty list.
ReplyDeleteI'm dreaming of antitrust busting on this Microsoft-Google-Amazon racket and a DANS-like verification to allow unfiltered mail transit.
Did you ever get any luck with those forms https://support.google.com/mail/contact/gmail_bulk_sender_escalation?rd=2&visit_id=638236101332243775-606097502
ReplyDeleteI never get anything back.
Hi there... I have a similar issue with Microsoft: every few months emails to hotmail, outlook and such sent by my properly configured mail server get being rejected due to "possible spam", "rate limit" or whatever, and I always fill out an online form, disucss things by email with some MS support staff, and get whitelisted again a few days later, not for good though, for a few months again. This is so ridicolous!
ReplyDeleteI have seen that with the Microsoft domains on a server I maintained for some years (but stopped doing so at the end of last year). The information I got was: Yes, Microsoft does block that whole provider at will, just because they can. That virtual server was hosted with NetCup (Germany). The mail server I had configured on that server had never sent spam before.
ReplyDeleteOn my own mail server (hosting: dedicated server, Hetzner, also Germany) I never had that kind of problems, so it seems Microsoft „likes“ Hetzner well enough to not just block emails from their customer's servers.
Never had trouble with GMail, though. At least … not yet.
Same experience here. Years ago I used to run the web and email for my tiny domain off my personal computer and things were great for a long time. Very low traffic, just a few email accounts and a small book club listserv. Then Cox started blocking ports on my residential connection, forcing me to either pay for a business connection or move to a hosting provider.
ReplyDeleteI moved to a shared hosting provider and all was good for web, but email was still a frequent hassle due to the issues in your article. Hotmail addresses were the worst for me despite not being on any spam lists.
I ended up changing hosting providers *again* and using a third party for the listserv and things finally seemed to have calmed down and most of the time people don't get the emails they find them in their spam folders.
It still drives me crazy when I hear "so and so says they didn't get the email" because it has become such an absolute nightmare to troubleshoot and deal with these big companies who have no humans you can contact.
We've seen this too recently. No change in our sending rate but messages randomly got temporarily bounced.
ReplyDeletehttps://blog.feedmail.org/2023/05/deliver-delays-to-gmail.html
You seem to be hosting your INDEPENDENT project on a shared server and it makes 0 sense that you're complaining about this.
ReplyDelete