10 Tips for Your Journey to the Public Cloud презентация

of Product Development, Intuit Director of Quality and Operations, Intuit

May 28, 2015

…

dumb pass-through
Routing logic to support bulk-head pattern (Pods) too complex for current ELBs
Developed a proxy layer to:
Terminate SSL
Implement routing logic
Access audit logging

1

with periodic key rotation:
Application encryption of sensitive data
Encryption in flight
File level encryption at rest
Reviewed fields to identify sensitive data to be “application level” encrypted
Dropping of clear text columns before data ready to ship
>50TB of data encrypted

2

encryption
Mint planned for 30% degradation
Continuous measurement of TP50, TP90, TP99 for critical features
Weekly review of TPs to drive improvements to reduce latency
Constant tuning of code and single page architecture
Able to maintain TP50 & TP90 SLAs
Create a culture of continuous focus on TPs to drive improvements

3

failing to deploy and requiring rollback
What we learned:
In AWS, operations spends a lot of time writing code: CloudFormation templates, deployment automation, monitors
Development rigor was new to the operations team
Needed to adopt development practices within operations: designs, code reviews, testing, validation, formal release processes for infrastructure

4

of secure data “over the wire”
Plan for data transport to AWS:
Encrypted drives physically secure shipped to AWS; 3 days to ship backup copy to AWS and upload
Catch up replication
Final drive shipment needs to be timed so that replication can catch up to the shipment window and sustain data growth prior to production cutover

5

a service to operation
Recovery Point Objective (RPO): amount of data acceptable to lose
Solve for availability first with Multi-AZ
Determine acceptable RTO/RPO and solve for regional failures second
Balance lower RTO/RPO against increased cost and complexity
Recognize the technology you use to handle regional failures will add complexity that could increase outages

Region US-EAST

Availability Zone

Availability Zone

Availability Zone

Region US-WEST

Availability Zone

Availability Zone

Availability Zone

6

dynamic
Number of instances are constantly changing
When an instance has issues it can be “blown away”
Build resilient and self-healing infrastructure
Monitoring should then be built to compliment this
If you alert on failure, have the courtesy to alert on healing

7

environment, you also need to validate:
Build, config, deploy, and validation infrastructure
Logging, Monitoring, etc system that ensure the environment is healthy
Access controls and security
Auto-Scaling

Continuous synthetic testing in the production environment
provide an end-to-end test to ensure the customer experience doesn’t degrade

8

9 hours per day, reserved will save money
On-demand for seasonal workloads and rare peaks
Reaper scripts; shutdown unused instances
Snapshots drove significant cost savings
Storage is cheap
A lot of work that yields a small return
IOPS are not
Optimizing IOPS per shard saved a lot of money

9

everything (blue-green)
We can always go back to N-1, ALWAYS!!

10

encryption, AWS Key Management
Low latency: TP50, TP90, TP99 measure and improve
Infrastructure as code: Design, review, test templates
Migrating large volumes of data: Encrypted drives
HA/DR: Multi-AZ, multi-region
Monitoring and diagnostics: Disassociate with IP addresses
End-to-end testing: Don’t forget to test auto-scaling
Managing costs: Compute is more expensive than storage
Release operations: Rollback-ready, blue-green