Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-7235

S3 Credentials are too restrictive


    • Type: Task
    • Status: Resolved
    • Priority: High
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Storage Volumes
    • Labels:


      Currently, it appears that we pretty much force the credentials to be in the URI. Instead, the following mechanism should be probably be used:


      • No credentials specified: depend on the machine role if its in AWS as that may have its own credentials
      • our reference.conf - remember the user should be able to override it with their own application.conf which is merged with our reference.conf, specifically, akka.stream.alpakka.s3.aws.{access-key-id, secret-access-key, default-region} which we should set to (as this makes us very consistent with http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html:
        • access-key-id: ${?AWS_ACCESS_KEY_ID}
        • secret-access-key: ${?AWS_SECRET_ACCESS_KEY}
        • default-region: ${?MARATHON_BACKUP_S3_DEFAULT_REGION} 
      • Loading the credentials from ~/.aws/credentials.
      • Loading the credentials from EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is set.
      • Instance profile credentials delivered through the Amazon EC2 metadata service.


      I didn't look at how alpakka handles all of this, but using the AWS class will do almost all of this for us (with the exception of application.conf, but the above description will set it up consistent with the expectations of the DefaultAWSCredentialsProviderChain). DefaultAWSCredentialsProviderChain has methods to fetch the credentials to put it into the S3 client.


      Finally, we should also consider taking in the URL as part of the HTTP Body instead of query parameters as URIs can be long and generally the total uri is limited to 2047 characters.




            • Assignee:
              matthias Matthias Veit (Inactive)
              jgilanfarr Jason Gilanfarr (Inactive)
              Orchestration Team
              James DeFelice, Jason Gilanfarr (Inactive)
            • Watchers:
              2 Start watching this issue


              • Created: