- Главная
- Разное
- Дизайн
- Бизнес и предпринимательство
- Аналитика
- Образование
- Развлечения
- Красота и здоровье
- Финансы
- Государство
- Путешествия
- Спорт
- Недвижимость
- Армия
- Графика
- Культурология
- Еда и кулинария
- Лингвистика
- Английский язык
- Астрономия
- Алгебра
- Биология
- География
- Детские презентации
- Информатика
- История
- Литература
- Маркетинг
- Математика
- Медицина
- Менеджмент
- Музыка
- МХК
- Немецкий язык
- ОБЖ
- Обществознание
- Окружающий мир
- Педагогика
- Русский язык
- Технология
- Физика
- Философия
- Химия
- Шаблоны, картинки для презентаций
- Экология
- Экономика
- Юриспруденция
Big Data Platform at interest презентация
Содержание
- 1. Big Data Platform at interest
- 3. Data Architecture
- 4. Data at Pinterest 60 Billion Pins
- 5. Pinterest Data Architecture App
- 6. Pinterest Data Architecture App
- 7. Pinterest Data Architecture App
- 8. Pinterest Data Architecture App
- 9. Design Choices for Hadoop Platform
- 10. Ephemeral clusters Access control layer Shared data
- 11. Decoupling compute & storage Hadoop Cluster 1
- 12. Centralized Hive Metastore Hive Metastore Pig Cascading Hive HDFS/S3 Data Metadata
- 13. Multi-layered Packaging Mapreduce Jobs Hadoop Jars/Libs Job/User
- 14. Executor Abstraction Layer Hive Metastore
- 15. API for simplified executor abstraction Advanced support
- 16. Pinball for Workflow Management
- 17. Scale: 60 Billion Pins Hundreds of workflows
- 18. Why Pinball? Requirements Simple abstractions Extensible in
- 19. Pinball Design Master
- 20. Workflow A directed graph of nodes
- 21. Job State Job state is captured in
- 22. Job State Machine RUNNABLE RUNNING WAITING
- 23. Master keeps the state
- 24. Master Entire state is kept in
- 25. Worker
- 26. Open Source Git repo:
- 27. Thank You
Data Architecture
Слайд 4Data at Pinterest
60 Billion Pins
1 Billion boards
100M MAU
60 PB of data
on S3
3 PB processed every day
2000 node Hadoop cluster
250 engineers
3 PB processed every day
2000 node Hadoop cluster
250 engineers
Слайд 8
Pinterest Data Architecture
App
events
Kafka
Secor
Skyline
Pinball
Redshift
Pinalytics
Features
Qubole (Hadoop)
Singer
Слайд 10Ephemeral clusters
Access control layer
Shared data store
Easy deployment
Hadoop Platform Requirements
Isolated multi-tenancy
Elasticity
Support multiple
clusters
Слайд 11Decoupling compute & storage
Hadoop Cluster 1
Transient HDFS
Hadoop Cluster 2
Transient HDFS
S3 Persistent
Store
Слайд 13Multi-layered Packaging
Mapreduce Jobs
Hadoop Jars/Libs
Job/User level Configs
Software Packages/Libs
Configs (OS/Hadoop)
Misc Sys Admin
OS
Bootstrap Script
Core
SW
Runtime Staging
(on S3)
Automated Configuration
(Masterless Puppet)
Baked AMI
Слайд 14Executor Abstraction Layer
Hive Metastore
HDFS/S3
Qubole
Managed Hadoop
EMR
Executor
Pinball
Dev Server
Слайд 15API for simplified executor abstraction
Advanced support for spot instances
Baked AMI customization
Why
Qubole?
Hadoop & Spark as managed services
Tight integration with Hive
Graceful cluster scaling
Слайд 17Scale:
60 Billion Pins
Hundreds of workflows
Thousands of jobs
500+ jobs in a workflow
3
petabytes processed daily
Support:
Hadoop, Cascading, Hive, Spark …
Support:
Hadoop, Cascading, Hive, Spark …
Scale of Processing
job
workflow
Слайд 18Why Pinball?
Requirements
Simple abstractions
Extensible in future
Reliable stateless computing
Easy to debug
Scales horizontally
Can be
upgraded w/o aborting workflows
Rich features like auto-retries, per-job emails, overrun policies…
Options
Apache Oozie, Azkaban, Luigi
Rich features like auto-retries, per-job emails, overrun policies…
Options
Apache Oozie, Azkaban, Luigi
Слайд 20Workflow
A directed graph of nodes called jobs
Edge
Run after dependence
Node
Job is
a node
Workflow Model
Слайд 21Job State
Job state is captured in a token
Tokens are named hierarchically
Master
Job
Token
version: 123
name: /workflow/w1/job
owner: worker_0
expiration: 1234567
data: JobTemplate(....)
version: 123
name: /workflow/w1/job
owner: worker_0
expiration: 1234567
data: JobTemplate(....)
Слайд 23
Master keeps the state
Workers claim and execute tasks
Horizontally scalable
Master Worker Interaction
Worker
Master
Persistent
Store
1: request
2: update
3: ack
Слайд 24Master
Entire state is kept in memory
Each state update is synchronously persisted
before master replies to client
Master runs on a single thread – no concurrency issues
Master runs on a single thread – no concurrency issues
Слайд 26Open Source
Git repo:
https://github.com/pinterest/pinball
Mailing list:
https://groups.google.com/forum/#!forum/pinball-users
