February 23, 2022
CloudFront and mysterious OriginCommError
Table of Contents
If you are running production workloads on AWS there’s a good chance you’re using Amazon CloudFront - Content Delivery Network (CDN) that helps you improve performance, security, and reliability of your website:
performance - caching assets at edge locations near to your users
security - DDoS mitigation and numerous other safety checks, especially when combined with AWS WAF
reliability - relates to (1) and (2): putting less load on your servers and preventing common attacks
The typical deployment
You have a CloudFront distribution with perhaps Beanstalk or Application Load Balancer (ALB) as your origin server.
To reap the benefits mentioned above, you need to restrict access to your load balancers - otherwise, an attacker can bypass all the CloudFront/WAF protections you have in place and perform a DDoS (or another) attack.
There are multiple options to do this, but a common and effective one is to allow access only from CloudFront IPs.<1>
And here comes the problem: how do you know which IPs should you whitelist?
Restricting Access
Let’s say you are an administrator of cool-website.com. You heard about this technique (described above) to improving the security of your website.
You attach a security group to your ALB which restricts access only from CloudFront IPs.
It appears to work - nice and safe, right?
The mystery
The original administrator left the company a few years ago and you are now responsible for keeping the website up and running.
You suddenly get a report about intermittent timeouts affecting some of your visitors. They all look pretty random and there’s nothing in your Beanstalk app logs. Moreover, the requests don’t seem to be reaching your proxy running on the web nodes.
What the hell is going on?
Logs, logs, and logs
You keep searching through the Cloudwatch logs but cannot find anything. It’s a mystery…
Then you finally bite the bullet and start downloading CloudFront logs.
It’s a real pain to work with them because they are stored in a S3 bucket.
When you look up some of the failing requests in the logs,
you notice an interesting pattern: all of them are failing with this mysterious OriginCommError:
(Tip: open the picture in a new browser tab)
OriginCommError - who are you?
But the first two links, serverfault and reddit, are actually useful. They don’t give you a clear answer but hint at something: that is, the requests are probably not reaching your load Beanstalk load balancers at all!
Ok, but why?
If something cannot connect to your network, it likely means there’s a firewall blocking access. On AWS, we have security groups.
You vaguelly remember, there was a Lambda function that the original administrator set up. It was supposed to update load balancer’s security groups to make sure the CloudFront IPs are whitelisted and the list of the IPs stay up to date. You check it and, of course, the lambda has been failing for a long time.
Now you are onto something: the CloudFront IPs must have changed!
Resolution
With the CloudFront managed prefix list, you don’t need to read or maintain a list of IP address ranges yourself. If you remove all other inbound rules from the security group, you prevent any non-CloudFront traffic from reaching the instance. Check Managed prefix lists in AWS VPC console, Use an AWS-managed prefix list and Reference prefix lists in your AWS resources.
The problem is solved, the users are happy again and you can go home.
References
Footnotes:
<1>: Additionally, you can make CloudFront attach a secret value in an HTTP header and configure ALB to only forward requests containing this header