99designs, like many websites that take privacy & security seriously, runs a bug bounty program; we pay “good hackers” to hack us and inform us of vulnerabilities that we then fix. As a global creative platform predominantly delivered via the web, most of the issues raised are fairly unremarkable CSRF and the occasional XSS vulnerabilities, but recently we had a report that stood out and took us on an interesting journey:
What is request smuggling?
—
Request smuggling is a type of attack whereby a bad actor crafts a HTTP request in such a way that they can cause disagreement (desynchronisation) between intermediate servers in how the request should be processed, resulting in their request being interpreted as the start of the next (probably valid) request on the connection.
In the case of this vulnerability, the disagreement arose between AWS Application Load Balancers and the HTTP library in Go, a language we love at 99designs:
How request smuggling can be exploited
—
Because the attacker has complete control over the beginning of the request, there are two basic types of attacks they can perform:
The first forces random users to take actions that they didn’t initiate, e.g. liking a design or accepting a friend request. This is done by ignoring the user’s original HTTP Request-Line by turning it into a header:
POST /anything HTTP/1.1 Content-Length: 4 Transfer-Encoding : chunked 0 GET /like-my-design HTTP/1.1 Ignore: XGET /any-user-request Cookie: UsersSessionCookie
When used like this, the cookies of the original request are kept intact, but the attacker no longer has control of the body. This limits the scope of the attack to simple actions that don’t require a body.
The second form of this attack takes control of the body of a request while using headers provided by the attacker. It can be much harder to find an endpoint that is vulnerable to this kind of attack, but if found it can be catastrophic to the end-users security.
POST /anything HTTP/1.1 Content-Length: 4 Transfer-Encoding : chunked 0 POST /send/private/message HTTP/1.1 Cookie: AttackersSessionCookie Content-Type: application/x-www-form-encoded msg=GET /any-user-request HTTP/1.1 Cookie: UserSessionCookie
In this example, the attacker has leveraged form encoding to send themselves a private message containing the user’s cookies. This would allow them to then take any action on behalf of a user. Form encoding will continue up until the first “&” in the body, which is enough to capture headers and a large portion of the body if the request was using json.
How we discovered the AWS/Go vulnerability
—
We learned about the AWS/Go request smuggling vulnerability on September 1st from a report by a security researcher at HackerOne (the host of our bounty program).
Internal triage and assessment
We spent the first week of September trying to understand the vulnerability and assess it’s possible impact.
We found we could not reproduce the issue in a local development environment—this lead us to believe the AWS Application Load Balancer might be involved. There were a few possible locations across our infrastructure that could have been causing the issue, so we started digging.
We tracked the issue to the Go-based web proxy / router we use to split traffic across all of our various services and realized it was an issue between that and its load balancer. At that point, we realized that this likely meant that all Go services using the standard library with an ALB were vulnerable (I believe the exact words were “half the internet”).
This seemed like a serious issue so we escalated the problem to both the Go and AWS security teams via email. Both companies replied promptly and committed to fixing the issue: Go said they were rolling out a security fix release and AWS promised to add a setting to ALBs.
Where are we now?
As of the publication of this article, this vulnerability between AWS and Go is no longer an issue on 99designs:
On September 26th: The Go team released Go 1.13.1 with a fix and submitted a CVE to MITRE.
September 30th: The next business day, we deployed the fix across our 20-something Go services.
October 1st: We paid a bounty to the security researcher who reported the vulnerability.
A huge thanks to the security researcher at HackerOne who discovered the vulnerability. Totally unsolicited plug: at 99designs, we take privacy and security very seriously, and we’ve been incredibly satisfied with the bug bounty program through HackerOne and highly recommend it.
The post Request smuggling: the Amazon is burning… your Go services appeared first on 99designs.
No comments:
Post a Comment