Developing Frameworks for Apache Mesos презентация

Содержание

1. Developing Frameworks for Apache Mesos
2. Joe Stein CEO of Elodina http://www.elodina.net/ a
3. What goes on Mesos? Framework =
4. What goes on Mesos?
6. Many, many things
7. Anything can be run on Mesos with
8. Framework = (Scheduler + Executor)
10. Scheduler
11. Executors
12. mesos/kafka https://github.com/mesos/kafka
13. Scheduler Provides the operational automation for a
14. CLI & REST API scheduler - starts
15. Launch 20 brokers in seconds ./kafka-mesos.sh add
16. What does it look like without a scheduler?
17. without a scheduler
18. without a scheduler
19. without a scheduler
20. without a scheduler
21. without a scheduler
22. We can do better using a scheduler
23. with a scheduler
24. with a scheduler
25. with a scheduler
26. with a scheduler
27. with a scheduler
28. Schedulers working together
32. Framework API & Examples
33. Mesos Protos https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto Everything is good
34. Mesos Framework Development Guide http://mesos.apache.org/documentation/latest/app-framework-development-guide/
35. Task Reconciliation http://mesos.apache.org/documentation/latest/reconciliation/ It is the
36. Language Bindings c++ → https://github.com/apache/mesos/tree/master/src/examples python
37. Rendler
38. Questions? Joe Stein http://www.elodina.net

Joe Stein CEO of Elodina http://www.elodina.net/ a big data as a service platform built on top open source software. The Elodina platform enables customers to analyze data streams and programmatically react

Слайд 1Developing Frameworks for Apache Mesos

Слайд 2Joe Stein
CEO of Elodina http://www.elodina.net/ a big data as a service

platform built on top open source software. The Elodina platform enables customers to analyze data streams and programmatically react to the results in real-time. We solve today’s data analytics needs by providing the tools and support necessary to utilize open source technologies. As users, contributors and committers, Elodina also provides support for frameworks that run on Mesos including Apache Kafka, Exhibitor (Zookeeper), Apache Storm, Apache Cassandra and a whole lot more!
LinkedIn: http://linkedin.com/in/charmalloc Twitter : @allthingshadoop

Слайд 3
What goes on Mesos?
Framework = (Scheduler + Executor)
What does it look

like without a scheduler?
We can do better using a scheduler!
Schedulers working together.
Framework API & Examples.

Overview

Слайд 4What goes on Mesos?

Слайд 5

Слайд 6Many, many things

Слайд 7Anything can be run on Mesos with Marathon or Aurora

https://mesosphere.github.io/marathon/
http://aurora.apache.org/

Слайд 8Framework = (Scheduler + Executor)

Слайд 9

Слайд 10Scheduler

Слайд 11Executors

Слайд 12mesos/kafka
https://github.com/mesos/kafka

Слайд 13Scheduler
Provides the operational automation for a Kafka Cluster.
Manages the changes to

the broker's configuration.
Exposes a REST API for the CLI to use or any other client.
Runs on Marathon for high availability.
Broker Failure Management “stickiness”

Executor
The executor interacts with the kafka broker as an intermediary to the scheduler

Scheduler & Executor

Слайд 14CLI & REST API
scheduler - starts the scheduler.
add - adds one

more more brokers to the cluster.
update - changes resources, constraints or broker properties one or more brokers.
remove - take a broker out of the cluster.
start - starts a broker up.
stop - this can either a graceful shutdown or will force kill it (./kafka-mesos.sh help stop)
rebalance - allows you to rebalance a cluster either by selecting the brokers or topics to rebalance. Manual assignment is still possible using the Apache Kafka project tools. Rebalance can also change the replication factor on a topic.
help - ./kafka-mesos.sh help || ./kafka-mesos.sh help {command}

Слайд 15Launch 20 brokers in seconds
./kafka-mesos.sh add 1000..1019 --cpus 0.01 --heap 128

--mem 256 --options num.io.threads=1
./kafka-mesos.sh start 1000..1019

Слайд 16What does it look like without a scheduler?

Слайд 17without a scheduler

Слайд 18without a scheduler

Слайд 19without a scheduler

Слайд 20without a scheduler

Слайд 21without a scheduler

Слайд 22We can do better using a scheduler

Слайд 23with a scheduler

Слайд 24with a scheduler

Слайд 25with a scheduler

Слайд 26with a scheduler

Слайд 27with a scheduler

Слайд 28Schedulers working together

Слайд 29

Слайд 30

Слайд 31

Слайд 32Framework API & Examples

Слайд 33Mesos Protos
https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto

Everything is good to understand but here is a good

place to start
FrameworkInfo
TaskInfo
TaskState
MasterInfo
SlaveInfo

Слайд 34Mesos Framework Development Guide
http://mesos.apache.org/documentation/latest/app-framework-development-guide/

Scheduler API
registered, reregistered, disconnected
resourceOffers, offerRescinded, statusUpdate, frameworkMessage
slaveLost,

executorLost, error
Executor API
registered, reregistered, disconnected
launchTask, killTask, frameworkMessage

Слайд 35Task Reconciliation
http://mesos.apache.org/documentation/latest/reconciliation/

It is the responsibility of Mesos (scheduler driver / Master)

to ensure that the framework is notified when a disconnection, and subsequent (re-)registration occurs. At this point, the scheduler should perform task state reconciliation.

Слайд 36Language Bindings
c++ →
https://github.com/apache/mesos/tree/master/src/examples
python →
https://github.com/apache/mesos/tree/master/src/examples/python
https://github.com/wickman/pesos
java →
https://github.com/apache/mesos/tree/master/src/examples/java
http://mesos.apache.org/api/latest/java/
https://github.com/groupon/jesos
go →
https://github.com/mesos/mesos-go

clojure →
https://github.com/dgrnbrg/clj-mesos
https://github.com/pyr/mesomatic
scala →
https://github.com/mesosphere/scala-sbt-mesos-framework.g8

Слайд 37Rendler

A rendering web crawler for Apache Mesos.
https://github.com/mesosphere/RENDLER

Слайд 38Questions?
Joe Stein
http://www.elodina.net

Скачать презентацию