How to reserve resources for certain frameworks in Mesos cluster ?

Note: This is an outdated article written for versions of Mesos prior to 0.23. A few things have changed since writing of this article:

  1. In Mesos 0.23 Dynamic reservations (MESOS-2018) were introduced. The Mesos project recommends specifying the total resources available on an agent as unreserved using --resources flag and manage reservations dynamically via master HTTP endpoints. ref: http://mesos.apache.org/documentation/latest/reservation/
  2. In Mesos 1.0 (MESOS-3945) /weights operator endpoint was introduced that allows for weights to be changed at runtime, while the --weights command line flag has been deprecated. ref: http://mesos.apache.org/documentation/latest/weights/

To reserve a certain amount of resources for a particular framework the concept of Roles in Mesos can be used
There are two ways to use roles to reserve resources for frameworks in your cluster:

  1. Dynamic Weighted reservations
  2. Static reservations

In either case,we’ll need to set up Mesos Master as well as the framework to use the new role.

Set up the cluster to use roles

Configure Mesos Master to define valid roles

Before we can use roles we need to define the roles that are going to be allowed in the cluster. Lets assume we want to have three roles in our cluster, namely : marathon, hdfs , chronos. To configure the mesos master:

  • Start the master with --roles="marathon,hdfs, chronos" command line flag

Associate the roles with the frameworks

Now we need to tell the framework which role to use, this will determine where the framework gets its resources from in the cluster. The way to configure role for a framework would vary among various frameworks, for example:

1) Marathon and Chronos let us configure the mesos role using commandline flag --mesos_role. To register Marathon with a role marathon

  • Start Marathon with --mesos_role="marathon"

2) HDFS lets us configure mesos role in the mesos-site.xml file by setting the value of mesos.hdfs.role property :

  <property>
    <name>mesos.hdfs.role</name>
    <value>hdfs</value>
  </property>

3) We can specify a custom role for your own framework(s) by setting FrameworkInfo.role to our desired role. See https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto

If we to set authorization on framework/roles we can set the register_frameworks ACL (receive_offers for versions older than Mesos 0.20) to specify which framework principals can register as which roles. This may not help us if both frameworks use the same principal, but we could at least prevent other framework principals from using those roles. See the --acls flag in mesos-master --help and docs/authorization.md for more info.

Note: If we don’t set a custom role for our own framework, and continue registering as the default “*” role, then there is no way to do role-based reservations to guarantee that certain resources do not get offered to Marathon.

At this point, we have frameworks registering as particular, non-default roles. Now let’s set some policies.

Setting resource policies for Framework roles

Dynamic weighted roles

If we want to dynamically partition resources in our cluster e.g. assign 1/3 resources to marathon and 2/3 resources to HDFS framework. This does not set hard limits or restrict the other frameworks/roles from using any particular resources, but adjusts Mesos’ allocator algorithm to consider the new fair share for each framework when choosing which framework to offer resoures to.

So if marathon and hdfs are the only two roles registered, and hdfs has double the weight of marathon, then marathon should get 1/3 of the cluster and hdfs 2/3 (rather than the default 50/50 split). Mesos uses this information to choose which framework gets the next resource offer, based on which role/framework is furthest below its fair share. Since hdfs will initially be further below its fair share, it will get first chance at offers (declined offers will still go to marathon), until it crosses the 1/3 threshold, at which point offers will alternate between marathon and hdfs, according to current fair share.

Configure Mesos Master for dynamic reservation

Lets assume we want to offer HDFS twice as much of the cluster as Marathon. We can configure the cluster using:

  • Start the master with --weights="marathon=1,hdfs=2” command line flag

Points to note :

  • Each role defaults to a value of 1. See mesos-master --help or https://mesos.apache.org/documentation/latest/configuration/
  • As hdfs declines offers, marathon can claim them with long-running services, potentially using more than its 1/3 fair share.
  • This share adjusts as new frameworks/roles join the cluster. If chronos registers with weight=1, the new breakdown is 50% hdfs, and 25% each for marathon and chronos.

Static Reservation

Instead (or in addition), we can use role-based reservations (aka “static reservations”, as opposed to MESOS-2018‘s Dynamic Reservations). This feature allows you to reserve specific sets of resources on a node for one or more specific roles at agent startup time. These resources will never be offered to any other role/framework, so you are guaranteed that capacity. In a future release, we may use MESOS-1607‘s Optimistic Offers to offer those resources to other frameworks with the caveat that they can be preempted when the reserving role/framework needs them.

Configure Mesos agent for static reservation

Mesos agent need to be configured to partition the resources among the various roles for static reservations. Lets assume we want to configure Marathon and HDFS to use 10 and 20 CPU shares and 2048MB and 4096MB for memory respectively. This can be done by:

  • Start the Mesos agent with command line flag --resources="cpus(marathon):10;cpus(hdfs):20;cpus(*):10;mem(marathon):2048;mem(hdfs):4096;mem(*):2048".

The ordering does not matter, but make sure your keys are cpus, mem, disk, or ports; not cpu or memory. If a resource type is not specified, its resources will be auto-detected and go to the default role. If at least one role is specified for a resource, but no unreserved (“*”) resources are specified, then only the role-reserved resources will be registered with Mesos, so make sure you do the math correctly.

Points to note :

  • Rather than splitting each node, you could devote entire nodes to a single role like --resources="cpus(hdfs):40;mem(cook):8192", or just set --default_role=hdfs and all autodetected resources will be reserved for that role. This may be especially valuable if there is resource contention of non-isolated resources, like disk cache.
  • Note that Mesos usually reserves 1GB RAM for the agent itself and other system processes. So if your agent has 16GB RAM, only 15GB will show up in Mesos.

An important note about roles in Mesos : we do not yet support a single framework registering with multiple roles. The current workaround is to start multiple instances of the framework, each with a different role. For Marathon, make sure you point each instance to a different —zk path, to ensure they get registered as different frameworkIds.

Have more questions? Submit a request

Comments

Powered by Zendesk