Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Full AWS Serverless

ProdOps
September 12, 2017

Full AWS Serverless

https://www.meetup.com/preview/San-Francisco-Cloud-Mafia/events/242901762

How an entirely serverless architecture can be achieved using a collection of AWS services, show a case study of an application using S3, Cognito, Lambda, DynamoDB, SNS, SQS, SES, IAM and other AWS services working together. Share decisions were made to build a scalable platform with the ability to add features within days.

ProdOps

September 12, 2017
Tweet

More Decks by ProdOps

Other Decks in Technology

Transcript

  1. .co.il www. cpu memory I/O time keeping customer box Why

    am I doing repetitive work here? :( I will be rich soon!
  2. .co.il www. JavaScript / ECMAScript 6 + AWS SDK ◦

    Usable in browser, backend, lambda ◦ Plenty of programmers and freelancers ◦ Avoid variability
  3. .co.il www. Deploy Web Assets ◦ Atlassian BitBucket Pipelines (free,

    fast) ◦ minimal babel compile ◦ execute transpiled tests ◦ Upload to S3
  4. .co.il www. # package.json { ... "scripts": { "build": "babel

    src -d dist/src && babel test -d dist/test", "build:test": "tap -Rspec --cov \"dist/test/**/*.test.js\"", }, } CI / CD
  5. .co.il www. CI / CD # bitbucket-pipelines.yml --- image: kopterio/node-yarn:7.7

    pipelines: default: - step: script: - yarn - npm run -s build - npm run -s build:test
  6. .co.il www. aws apigateway put-rest-api \ --rest-api-id 1234123412 \ --mode

    overwrite \ --body 'file:///path/to/API_Swagger_template.json' \ --region us-west-2 aws apigateway put-rest-api \ --rest-api-id 1234123412 \ --mode merge \ --body 'file:///path/to/API_Swagger_template.json' \ --region us-west-2
  7. .co.il www. /: &path_with_cors options: summary: CORS Response responses: 200:

    description: Stub CORS response headers: &cors_headers Access-Control-Allow-Origin: { type: string } Access-Control-Allow-Methods: { type: string } Access-Control-Allow-Headers: { type: string } schema: $ref: "#/definitions/Empty" x-amazon-apigateway-integration: passthroughBehavior: "never" type: "mock" responses: default: statusCode: 200 responseParameters: &cors_response_params method.response.header.Access-Control-Allow-Methods: "'GET,OPTIONS'" method.response.header.Access-Control-Allow-Headers: "'Content-Type, ... '" method.response.header.Access-Control-Allow-Origin: "'*'" requestTemplates: application/json: '{"statusCode": 200}' none of that nonsense ... docs.aws.amazon.com /apigateway/latest/ developerguide/how-to-cors.html
  8. .co.il www. /v1/audit/aws: <<: *path_with_cors get: summary: Get the audit

    log. security: - kopter.io: [] parameters: - $ref: '#/parameters/authorization' - $ref: '#/parameters/identityId' - $ref: '#/parameters/requestId' responses: 200: description: An array of audit log events headers: <<: *cors_headers schema: $ref: '#/definitions/AuditLogEvents' 520: description: Error response schema: $ref: '#/definitions/Error' x-amazon-apigateway-integration: requestParameters: integration.request.header.authorization: "'Basic $nginx_auth'" integration.request.header.identityId: "context.identity.cognitoIdentityId" integration.request.header.requestId: "context.requestId" responses:
  9. .co.il www. Kopter AWS account Process Customer AWS Events CloudTrail

    S3 Customer AWS account SNS SQS ECS DynamoDB Elastic Beanstalk
  10. .co.il www. app.post('/', sqsHandler); app.post('/cron', cronHandler); export async function sqsHandler(req,

    res) { const queueName = req.get('X-Aws-Sqsd-Queue'); try { switch (queueName) { case process.env.AGGREGATOR_QUEUE_NAME: await new Aggregator(req.body).start(); break; case process.env.CLOUD_TRAIL_INGEST_QUEUE_NAME: await new CloudTrailIngest(req.body).start(); break; default: throw new Error(`no worker found with X-Aws-Sqsd-Queue: ${queueName}`); } } catch (error) { log.error(error); res.status(500).send({}); } res.status(200).send({}); }
  11. .co.il www. cron.yaml version: 1 cron: - name: "backup-job" url:

    "/cron/backup" schedule: "0 */12 * * *" - name: "audit" url: "/cron/audit" schedule: "0 23 * * *"
  12. .co.il www. EC2 Events Queue kopter workers Describer Cron EC2

    Describer Aggregator Cron SNS *Hourly* CRON Aggregator *Hourly* CRON EC2 Pricing *Hourly* CRON EC2 Describer ? ? CloudTrail Ingest sqsd AWS Accounts sqsd Aggregator Queue sqsd Aggregator sqsd EC2 Pricing Audit Log Queue sqsd Audit Logs Frontend Notifications sqsd Frontend Notifications CloudTrail Ingest EC2 Snapshots EC2 Pricing Backend API sqsd Describer Queue sqsd EC2 Events Price Graph Data Audit Logs Frontend Notifications Frontend Notifications Global ? Legend DB Table Queue Worker sqsd kopter workers kopter workers ᵣ function
  13. .co.il www. export class Handler { static async start() {

    const queueUrl = process.env.AGGREGATOR_QUEUE_URL; const message = await sqsCheck(queueUrl); await new Aggregator(message).start(); await sqsDeleteMessage(queueUrl, message); } } export async function handler(event, context, callback) { try { await Handler.start(); callback(); } catch (error) { callback(error); } }
  14. .co.il www. Version: '2012-10-17' Statement: - Sid: AllowAccessToOnlyItemsMatchingUserID Effect: Allow

    Action: - dynamodb:GetItem - dynamodb:Query Resource: - arn:aws:dynamodb:us-west-2:123456789012:table/GameScores Condition: ForAllValues:StringEquals: dynamodb:LeadingKeys: - "${www.amazon.com:user_id}" dynamodb:Attributes: - UserId - GameTitle - Wins - Losses - TopScore - TopScoreDateTime StringEqualsIfExists: dynamodb:Select: SPECIFIC_ATTRIBUTES
  15. .co.il www. Version: '2012-10-17' Statement: - Sid: LimitAccessToSpecificAttributes Effect: Allow

    Action: - dynamodb:UpdateItem - dynamodb:GetItem - dynamodb:Query - dynamodb:BatchGetItem - dynamodb:Scan Resource: - arn:aws:dynamodb:us-west-2:123456789012:table/GameScores Condition: ForAllValues:StringEquals: dynamodb:Attributes: - UserId - TopScore StringEqualsIfExists: dynamodb:Select: SPECIFIC_ATTRIBUTES dynamodb:ReturnValues: - NONE - UPDATED_OLD - UPDATED_NEW
  16. .co.il www. --- Version: '2012-10-17' Statement: - Action: - s3:ListBucket

    Effect: Allow Resource: - arn:aws:s3:::mybucket Condition: StringLike: s3:prefix: - "${cognito-identity.amazonaws.com:sub}/*" - Action: - s3:GetObject - s3:PutObject Effect: Allow Resource: - arn:aws:s3:::mybucket/${cognito-identity.amazonaws.com:sub}/* S3 permissions
  17. .co.il www. ElastiCache "FNR4GZ675EW5UXJW" : { "FNR4GZ675EW5UXJW.JRTCKXETXF" : { "offerTermCode"

    : "JRTCKXETXF", "sku" : "FNR4GZ675EW5UXJW", "effectiveDate" : "2015-10-01T00:00:00Z", "priceDimensions" : { "FNR4GZ675EW5UXJW.JRTCKXETXF.6YS6EN2CT7" : { "rateCode" : "FNR4GZ675EW5UXJW.JRTCKXETXF.6YS6EN2CT7", "rateType" : "Fixed", "description" : "$0.044 per RDS T2 Small Instance hour (or partial hour) running SQL Server EX - LI", "beginRange" : "0", "endRange" : "Inf", "unit" : "Hrs", "pricePerUnit" : { "USD" : "0.0440000000" AWS EC2 Pricing API ~ 100mb aws.amazon.com/blogs/aws/new-aws-price-list-api/