Do you speak SAML? Google Apps, Salesforce and SAP Hana Cloud tested
In this article we compare the SAML service provider implementation of three popular cloud services:
- Google Apps (which includes GMail, Google Drive and Docs, Calendar)
- Salesforce
- SAP Hana Cloud
Our testing procedure includes verification of the service provider compliance to the SAML 2.0 specification and checking the handling of signature validations.
Abstract:
Secure Assertion Markup Language (SAML) today is the main standard used for signing in to Cloud Services with a single authentication procedure (typically username/password). A correct implementation of the standard is crucial for security. Failing to do so may compromise security and lead to information loss.
Unfortunately SAML 2.0 is very complex and probably over-engineered. Leaving the developer too many degrees of freedom to implement only parts of the security measures envisaged by the standard. The risk even aggravates as the implementation might look like they are functioning correctly: single sign on works and some of the checks against signature or timestamps are processed. But on diving a little deeper security issues or nonconformity will become evident.
In our research we tested Google Apps, Salesforce CRM and SAP Hana Cloud as representatives of modern Cloud Service providers which provide Single Sign On integration with SAML 2.0.
Research method: Tool to run automated tests leveraging the SSOCheck API.
Test cases were divided into different testing areas:
- Replay
- General XML
- SAML Response Message
- SAML Assertion
- Digital Signature
Whereas the tests of area 3 and 4 typically refer to the components of the SAML documents as illustrated in the following picture.
Results:
SAP performed best in all categories. Salesforce ranked second. Google was vulnerable to assertion replay and almost completely ignored the response part of the SAML message and several attributes of the assertion.
We informed the security teams of the tested companies about the results before publishing the article. All companies replied in acceptable time. Some involved their development departments which tried to reproduce the tests and some were arguing with risk based approaches. Salesforce being the fastest and most communicative respondent. SAP’s answer was the slowest but the most meticulous. Google took some time to respond but over time a very interesting discussion evolved with participation of several members of the security and product team which leads to the enrollment of product patches. Most parties leveraged SSOCheck tool to understand and reproduce the findings.
The following table summarizes the results found.
Summary Table (% passed tests)
| Test | Google Apps | Salesforce CRM | SAP Hana Cloud | |||||
|---|---|---|---|---|---|---|---|---|
| Replay | 0 | 100 | 100 | |||||
| General XML | 100 | 100 | 100 | |||||
| SAML Response | 16.7 | 66.7 | 83.3 | |||||
| SAML Assertion | 50.0 | 69.2 | 76.9 | |||||
| Digital Signature | 100 | 100 | 100 (*) | |||||
| Total | 48.5 | 82.7 | 88.5 |
*) SAP Hana Cloud was the only service provider who accepted a SAML response with an evil assertion inserted before the valid assertion. We rated the test as passed since the SAP implementation seemed to totally ignore the evil assertion and therefore could not be used to attack the service.
Total results were calculated as a weighted average of the group results. Giving the SAML assertion tests a weight of 2, general XML tests a weight of 0.5 and the rest a weight of 1.
Detailed test result table:
| Test | Google Apps | Salesforce CRM | SAP Hana Cloud | |
|---|---|---|---|---|
| 1 | Unmodified SAML – as a positive protocol test |  |
|
|
| 2 | Replay Attack – SAML protocol message replayed | |
|
|
| 3 | Invalid SAML Protocol Namespace |  |
|
|
| 4 | Invalid SAML Assertion Namespace | |
|
|
| 5 | SAML Response Status Code is set to RequestDenied | |
|
|
| 6 | SAML Response Issuer is invalid | |
|
|
| 7 | SAML Response IssueInstant is set to a value in the future | |
|
|
| 8 | SAML Response InResponseTo is invalid | |
|
|
| 9 | SAML Response Destination is invalid | |
|
|
| 10 | SAML Response Version is invalid | |
|
|
| 11 | SAML Assertion Issuer invalid | |
|
|
| 12 | SAML Assertion IssueInstant is set to a value in the future | |
|
|
| 13 | SAML Assertion Version is invalid | |
|
|
| 14 | SAML Assertion Subject without NameID | |
|
|
| 15 | SAML Assertion subject NameId format set to an unknown value |  |
|
|
| 16 | SAML Assertion SubjectConfirmation Method invalid | |
|
|
| 17 | No SubjectConfirmationData element in the SAML Assertion sent | |
|
|
| 18 | SAML Assertion InResponseTo is invalid | |
|
|
| 19 | Recipient in SAML Assertion SubjectConfirmationData is invalid | |
|
|
| 20 | Address in SAML Assertion SubjectConfirmationData is invalid | |
|
|
| 21 | NotOnOrAfter in SAML Assertion SubjectConfirmationData is set to a value 1h into the past | |
|
|
| 22 | Two Assertion SubjectConfirmationData elements whereas the first is the valid one and the second is a wrong value. | |
|
|
| 23 | Two Assertion Subject Confirmation Data elements whereas the first is the wrong one and the second has the correct value. | |
|
|
| 24 | SAML Assertion Condition is inserted which is unknown to the service provider | |
|
|
| 25 | SAML Assertion Condition NotBefore is set to a value of 1h in advance. | |
|
|
| 26 | SAML Assertion Condition NotOnOrAfter set to 1h in the past. | |
|
|
| 27 | Syntax test to check that the SP supports the OneTimeUse element. | |
|
|
| 28 | AudienceRestriction element in SAML Assertion Condition is empty | |
|
|
| 29 | AudienceRestriction element in SAML Assertion Condition is set to a wrong value | |
|
|
| 30 | Two values in one SAML Assertion AudienceRestriction element. The wrong value is the first | |
|
|
| 31 | Two values in one SAML Assertion AudienceRestriction element. The wrong value is second. | |
|
|
| 32 | Two AudienceRestriction elements in SAML Assertion. The first elment holds the wrong value | |
|
|
| 33 | Two AudienceRestriction elements in SAML Assertion. The second elment holds the wrong value | |
|
|
| 34 | Two AudienceRestriction elements in SAML Assertion. Both hold two audience values in different ordering | |
|
|
| 35 | AuthnStatement is missing in SAML Assertion | |
|
|
| 36 | Sets the SubjectLocality of AuthnStatement to a non valid IP address | |
|
|
| 37 | AuthnInstant timstamp of Assertion AuthNStatement is moved one day into the future. | |
|
|
| 38 | AuthnInstant timstamp of Assertion AuthNStatement is moved one day back in time. | |
|
|
| 39 | SessionNotOnOrAfter timstamp of Assertion AuthNStatement is set one day in the past. | |
|
|
| 40 | AuthnContextClassRef of Assertion AuthNStatement is set to “unsepcified” and should be declined by the service provider. | |
|
|
| 41 | Multiple Signature tests: signature exclusion | |
|
|
| 42 | Multiple Signature tests: mangled signature | |
|
|
| 43 | Multiple Signature tests: wrong signature key | |
|
|
| 44 | signature wrapping variants | |
|
|
Conclusion:
All tested Cloud Services did not fully comply with the SAML standard.
SAP and Salesforce did not disclose any severe problems which could lead to a significant exploit. Non conformity to the specification might lead to the non-functioning of specific use cases but can be justified in order to achieve broader compatibility with IDP products or might be argued with risk based approaches.
Google Apps SAML implementation revealed several issues which could be leveraged by an attack scenario. The good news is that Google has rolled out fixes for these findings which we were able to verify.
We especially thank the Google team for a valuable interaction and cooperation.
If you have questions or comments please let me know. We are also looking for other SaaS services, which might be of general interest to run the tests against.
