…And Metrics For All презентация

Содержание

1. …And Metrics For All
2. About Yelp Founded: 2004 Monthly Active Users:
3. What are metrics? Name Value
4. What are metrics? Name Value Timestamp
5. What are metrics? Name Value Timestamp server1.load.1m 28.826667 1431950640
6. What are metrics? Name Value Timestamp server1.load.1m
7. What are metrics? Name Value Timestamp server1.load.1m
8. Graphite Components Carbon: relay cache aggregator Whisper Web app
9. Carbon Relay Deals with 2 things Replication Sharding
10. Relay Methods Rules [replicate] pattern = ^services\.ads\..+
11. Carbon Cache Receives metrics and persists them to disk Writes based on storage schemas
12. Storage Schemas Details retention rates for storing
13. Storage Aggregation Rules for aggregating data to
14. Carbon Aggregator Buffers metrics before forwarding to carbon cache Roll up metrics based on rules
15. Aggregation Rules Not to be confused with
16. Whisper Fixed size database Allows for roll ups Allows for backfilling data
17. Web App Django based app for rendering graphs
18. Putting it all together Carbon cache listening
19. Getting more complicated Carbon relay using consistent
20. More Relays Use HAProxy to load balance between relays Use more relays to use CPU
21. Even more relays Useful for sending metrics to other locations
22. Replicate the metrics Duplicate your metrics for backup, and redundancy
23. More caches instead Consistent hash across multiple nodes
24. Where does the aggregator fit? Aggregator uses
25. Scaling further Use nodes for particular functions:
28. Getting your data back out Graphite Dashboard
30. Tips Aggregate before ingestion Control the metrics
31. Optimize your dashboard queries services.biz_app.*.*.timers.pyramid_uwsgi_metrics_tweens_*.p99 2154 results
33. What’s the Future? InfluxDB Cassandra Third party
34. We’re hiring! http://www.yelp.com/careers Hiring SREs in Dublin, London, New York, San Francisco

About Yelp Founded: 2004 Monthly Active Users: ~142 Million Non-US Monthly Users: ~31 Million Review: ~77 Million Local Businesses: 2.1 Million Territories: Available in 31 countries

Слайд 1…And Metrics For All
Paul O’Connor
github.com/pauloconnor
2015-05-19

Слайд 2About Yelp
Founded: 2004
Monthly Active Users: ~142 Million
Non-US Monthly Users: ~31 Million
Review:

~77 Million
Local Businesses: 2.1 Million
Territories: Available in 31 countries

Слайд 3What are metrics?
Name Value

Слайд 4What are metrics?
Name Value Timestamp

Слайд 5What are metrics?
Name Value Timestamp
server1.load.1m 28.826667 1431950640

Слайд 6What are metrics?
Name Value Timestamp
server1.load.1m 28.826667 1431950640
server1.load.1m 29.188333 1431950700
server1.load.1m 29.231667 1431950760
server1.load.1m

29.083333 1431950820
server1.load.1m 29.710000 1431950880

Слайд 7What are metrics?
Name Value Timestamp
server1.load.1m 28.826667 1431950640
server1.load.1m 29.188333 1431950700
server1.load.1m 29.231667 1431950760
server1.load.1m

29.083333 1431950820
server1.load.1m 29.710000 1431950880

Слайд 8Graphite Components
Carbon:
relay
cache
aggregator
Whisper
Web app

Слайд 9Carbon Relay
Deals with 2 things
Replication
Sharding

Слайд 10Relay Methods
Rules
[replicate]
pattern = ^services\.ads\..+
servers = 10.1.2.3, 10.2.2.3
continue = true
Consistent Hashing
Defines a

sharding strategy across multiple backends

$Relay MethodsRules[replicate]pattern = ^services\.ads\..+servers = 10.1.2.3, 10.2.2.3continue = trueConsistent HashingDefines a sharding strategy across multiple backends$

Слайд 11Carbon Cache
Receives metrics and persists them to disk
Writes based on storage

schemas

Слайд 12Storage Schemas
Details retention rates for storing metrics

[databases_10sec_1year]
pattern = ^servers\.db.*$
retentions = 10s:7d,1m:30d,5m:90d,30m:365d

$Storage SchemasDetails retention rates for storing metrics[databases_10sec_1year]pattern = ^servers\.db.*$retentions = 10s:7d,1m:30d,5m:90d,30m:365d$

Слайд 13Storage Aggregation
Rules for aggregating data to lower-precision retentions

[all_min]
pattern = \.min$
xFilesFactor =

0.1
aggregationMethod = min

$Storage AggregationRules for aggregating data to lower-precision retentions[all_min]pattern = \.min$xFilesFactor = 0.1aggregationMethod = min$

Слайд 14Carbon Aggregator
Buffers metrics before forwarding to carbon cache
Roll up metrics based

on rules

Слайд 15Aggregation Rules
Not to be confused with storage aggregation
Tells the carbon aggregator

what to aggregate and how

output_template (frequency) = method input_pattern

.applications..all.requests (60) = sum .applications..*.requests

prod.applications.apache.www01.requests
prod.applications.apache.www02.requests
prod.applications.apache.www03.requests
prod.applications.apache.www04.requests
prod.applications.apache.www05.requests

prod.applications.apache.all.requests

Слайд 16Whisper
Fixed size database
Allows for roll ups
Allows for backfilling data

Слайд 17Web App
Django based app for rendering graphs

Слайд 18Putting it all together
Carbon cache listening on port 2003
Write to disk
Listen

with web

Слайд 19Getting more complicated
Carbon relay using consistent hashing to multiple caches
Individual caches

responsible for specific metrics

Слайд 20More Relays
Use HAProxy to load balance between relays
Use more relays to

use CPU

Слайд 21Even more relays
Useful for sending metrics to other locations

Слайд 22Replicate the metrics
Duplicate your metrics for backup, and redundancy

Слайд 23More caches instead
Consistent hash across multiple nodes

Слайд 24Where does the aggregator fit?
Aggregator uses a lot of CPU. Put

it on it’s own node

Слайд 25Scaling further
Use nodes for particular functions:
Use forwarding relay nodes solely to

forward
Have consistent hashing nodes
Have aggregation nodes

Слайд 26

Слайд 27

Слайд 28Getting your data back out
Graphite Dashboard
Third Party Dashboard
We use Grafana http://grafana.org/
Graphite-api

https://github.com/brutasse/graphite-api

Слайд 29

Слайд 30Tips
Aggregate before ingestion
Control the metrics that can be sent
Metrics are a

gas - they expand to fill all available room
Use C implementation of carbon
Use the latest webapp.

Слайд 31Optimize your dashboard queries
services.biz_app...timers.pyramid_uwsgi_metrics_tweens_*.p99
2154 results
35 seconds to just find these files

on disk
Running functions against these results
Timeout after a minute
Dashboard automatically refreshing every 10 seconds

Слайд 32

Слайд 33What’s the Future?
InfluxDB
Cassandra
Third party

Слайд 34We’re hiring!
http://www.yelp.com/careers
Hiring SREs in Dublin, London, New York, San Francisco

Скачать презентацию

…And Metrics For All презентация

Содержание

Слайд 1…And Metrics For All
Paul O’Connor
github.com/pauloconnor
2015-05-19

Слайд 2About Yelp
Founded: 2004
Monthly Active Users: ~142 Million
Non-US Monthly Users: ~31 Million
Review:

Слайд 3What are metrics?
Name Value

Слайд 4What are metrics?
Name Value Timestamp

Слайд 5What are metrics?
Name Value Timestamp
server1.load.1m 28.826667 1431950640

Слайд 6What are metrics?
Name Value Timestamp
server1.load.1m 28.826667 1431950640
server1.load.1m 29.188333 1431950700
server1.load.1m 29.231667 1431950760
server1.load.1m

Слайд 7What are metrics?
Name Value Timestamp
server1.load.1m 28.826667 1431950640
server1.load.1m 29.188333 1431950700
server1.load.1m 29.231667 1431950760
server1.load.1m

Слайд 8Graphite Components
Carbon:
relay
cache
aggregator
Whisper
Web app

Слайд 9Carbon Relay
Deals with 2 things
Replication
Sharding

Слайд 10Relay Methods
Rules
[replicate]
pattern = ^services\.ads\..+
servers = 10.1.2.3, 10.2.2.3
continue = true
Consistent Hashing
Defines a

Слайд 11Carbon Cache
Receives metrics and persists them to disk
Writes based on storage

Слайд 12Storage Schemas
Details retention rates for storing metrics

[databases_10sec_1year]
pattern = ^servers\.db.*$
retentions = 10s:7d,1m:30d,5m:90d,30m:365d

Слайд 13Storage Aggregation
Rules for aggregating data to lower-precision retentions

[all_min]
pattern = \.min$
xFilesFactor =

Слайд 14Carbon Aggregator
Buffers metrics before forwarding to carbon cache
Roll up metrics based

Слайд 15Aggregation Rules
Not to be confused with storage aggregation
Tells the carbon aggregator

Слайд 16Whisper
Fixed size database
Allows for roll ups
Allows for backfilling data

Слайд 17Web App
Django based app for rendering graphs

Слайд 18Putting it all together
Carbon cache listening on port 2003
Write to disk
Listen

Слайд 19Getting more complicated
Carbon relay using consistent hashing to multiple caches
Individual caches

Слайд 20More Relays
Use HAProxy to load balance between relays
Use more relays to

Слайд 21Even more relays
Useful for sending metrics to other locations

Слайд 22Replicate the metrics
Duplicate your metrics for backup, and redundancy

Слайд 23More caches instead
Consistent hash across multiple nodes

Слайд 24Where does the aggregator fit?
Aggregator uses a lot of CPU. Put

Слайд 25Scaling further
Use nodes for particular functions:
Use forwarding relay nodes solely to

Слайд 26

Слайд 27

Слайд 28Getting your data back out
Graphite Dashboard
Third Party Dashboard
We use Grafana http://grafana.org/
Graphite-api

Слайд 29

Слайд 30Tips
Aggregate before ingestion
Control the metrics that can be sent
Metrics are a

Слайд 31Optimize your dashboard queries
services.biz_app...timers.pyramid_uwsgi_metrics_tweens_*.p99
2154 results
35 seconds to just find these files

Слайд 32

Слайд 33What’s the Future?
InfluxDB
Cassandra
Third party

Слайд 34We’re hiring!
http://www.yelp.com/careers
Hiring SREs in Dublin, London, New York, San Francisco

Обратная связь

Что такое ThePresentation.ru?

…And Metrics For All презентация

Содержание

Слайд 1…And Metrics For AllPaul O’Connorgithub.com/pauloconnor2015-05-19

Слайд 2About YelpFounded: 2004Monthly Active Users: ~142 MillionNon-US Monthly Users: ~31 MillionReview:

Слайд 3What are metrics?Name Value

Слайд 4What are metrics?Name Value Timestamp

Слайд 5What are metrics?Name Value Timestampserver1.load.1m 28.826667 1431950640

Слайд 6What are metrics?Name Value Timestampserver1.load.1m 28.826667 1431950640server1.load.1m 29.188333 1431950700server1.load.1m 29.231667 1431950760server1.load.1m

Слайд 7What are metrics?Name Value Timestampserver1.load.1m 28.826667 1431950640server1.load.1m 29.188333 1431950700server1.load.1m 29.231667 1431950760server1.load.1m

Слайд 8Graphite ComponentsCarbon:relaycacheaggregatorWhisperWeb app

Слайд 9Carbon RelayDeals with 2 thingsReplicationSharding

Слайд 10Relay MethodsRules[replicate]pattern = ^services\.ads\..+servers = 10.1.2.3, 10.2.2.3continue = trueConsistent HashingDefines a

Слайд 11Carbon CacheReceives metrics and persists them to diskWrites based on storage

Слайд 12Storage SchemasDetails retention rates for storing metrics[databases_10sec_1year]pattern = ^servers\.db.*$retentions = 10s:7d,1m:30d,5m:90d,30m:365d

Слайд 13Storage AggregationRules for aggregating data to lower-precision retentions[all_min]pattern = \.min$xFilesFactor =

Слайд 14Carbon AggregatorBuffers metrics before forwarding to carbon cacheRoll up metrics based

Слайд 15Aggregation RulesNot to be confused with storage aggregationTells the carbon aggregator

Слайд 16WhisperFixed size databaseAllows for roll upsAllows for backfilling data

Слайд 17Web AppDjango based app for rendering graphs

Слайд 18Putting it all togetherCarbon cache listening on port 2003Write to diskListen

Слайд 19Getting more complicatedCarbon relay using consistent hashing to multiple cachesIndividual caches

Слайд 20More RelaysUse HAProxy to load balance between relaysUse more relays to

Слайд 21Even more relaysUseful for sending metrics to other locations

Слайд 22Replicate the metricsDuplicate your metrics for backup, and redundancy

Слайд 23More caches insteadConsistent hash across multiple nodes

Слайд 24Where does the aggregator fit?Aggregator uses a lot of CPU. Put

Слайд 25Scaling furtherUse nodes for particular functions:Use forwarding relay nodes solely to

Слайд 26

Слайд 27

Слайд 28Getting your data back outGraphite DashboardThird Party DashboardWe use Grafana http://grafana.org/Graphite-api

Слайд 29

Слайд 30TipsAggregate before ingestionControl the metrics that can be sentMetrics are a

Слайд 31Optimize your dashboard queriesservices.biz_app.*.*.timers.pyramid_uwsgi_metrics_tweens_*.p992154 results35 seconds to just find these files

Слайд 32

Слайд 33What’s the Future?InfluxDBCassandraThird party

Слайд 34We’re hiring!http://www.yelp.com/careersHiring SREs in Dublin, London, New York, San Francisco

Похожие презентации

Обратная связь

Что такое ThePresentation.ru?

Слайд 1…And Metrics For All
Paul O’Connor
github.com/pauloconnor
2015-05-19

Слайд 2About Yelp
Founded: 2004
Monthly Active Users: ~142 Million
Non-US Monthly Users: ~31 Million
Review:

Слайд 3What are metrics?
Name Value

Слайд 4What are metrics?
Name Value Timestamp

Слайд 5What are metrics?
Name Value Timestamp
server1.load.1m 28.826667 1431950640

Слайд 6What are metrics?
Name Value Timestamp
server1.load.1m 28.826667 1431950640
server1.load.1m 29.188333 1431950700
server1.load.1m 29.231667 1431950760
server1.load.1m

Слайд 7What are metrics?
Name Value Timestamp
server1.load.1m 28.826667 1431950640
server1.load.1m 29.188333 1431950700
server1.load.1m 29.231667 1431950760
server1.load.1m

Слайд 8Graphite Components
Carbon:
relay
cache
aggregator
Whisper
Web app

Слайд 9Carbon Relay
Deals with 2 things
Replication
Sharding

Слайд 10Relay Methods
Rules
[replicate]
pattern = ^services\.ads\..+
servers = 10.1.2.3, 10.2.2.3
continue = true
Consistent Hashing
Defines a

Слайд 11Carbon Cache
Receives metrics and persists them to disk
Writes based on storage

Слайд 12Storage Schemas
Details retention rates for storing metrics

[databases_10sec_1year]
pattern = ^servers\.db.*$
retentions = 10s:7d,1m:30d,5m:90d,30m:365d

Слайд 13Storage Aggregation
Rules for aggregating data to lower-precision retentions

[all_min]
pattern = \.min$
xFilesFactor =

Слайд 14Carbon Aggregator
Buffers metrics before forwarding to carbon cache
Roll up metrics based

Слайд 15Aggregation Rules
Not to be confused with storage aggregation
Tells the carbon aggregator

Слайд 16Whisper
Fixed size database
Allows for roll ups
Allows for backfilling data

Слайд 17Web App
Django based app for rendering graphs

Слайд 18Putting it all together
Carbon cache listening on port 2003
Write to disk
Listen

Слайд 19Getting more complicated
Carbon relay using consistent hashing to multiple caches
Individual caches

Слайд 20More Relays
Use HAProxy to load balance between relays
Use more relays to

Слайд 21Even more relays
Useful for sending metrics to other locations

Слайд 22Replicate the metrics
Duplicate your metrics for backup, and redundancy

Слайд 23More caches instead
Consistent hash across multiple nodes

Слайд 24Where does the aggregator fit?
Aggregator uses a lot of CPU. Put

Слайд 25Scaling further
Use nodes for particular functions:
Use forwarding relay nodes solely to

Слайд 28Getting your data back out
Graphite Dashboard
Third Party Dashboard
We use Grafana http://grafana.org/
Graphite-api

Слайд 30Tips
Aggregate before ingestion
Control the metrics that can be sent
Metrics are a

Слайд 31Optimize your dashboard queries
services.biz_app...timers.pyramid_uwsgi_metrics_tweens_*.p99
2154 results
35 seconds to just find these files

Слайд 33What’s the Future?
InfluxDB
Cassandra
Third party

Слайд 34We’re hiring!
http://www.yelp.com/careers
Hiring SREs in Dublin, London, New York, San Francisco