Goal: Introduce a metric that represents the amount of data exchanged over the HTTP channel.
For various reasons (some which we wish to get addressed, soon), there are many API consumers that query Marathon state in a very inefficient way. As a Marathon instance gets larger (more apps and tasks), the number of concurrent API consumers also tends to get larger, also. The total API load becomes a product of these two facts, to the point where Marathon will be generating terabytes of JSON responses per minute. (Generally speaking, we see, with compression disabled, and upper limit of about 2 TB per minute before functionality is significantly impaired).
The overall serialization overhead for an API request depends on the query string parameters and the resource route. A good surrogate for this that works across all API routes is the total amount of bytes returned by the API. Adding this metric will provide a general purpose monitor to see if any part of the API is being queried excessively to the point that other functionality will start to degrade.
- Metric: total API bytes generated per period (minute?)
- Metric: total API bytes consumed per period
- Proxy contributes to these metrics? ¯_(ツ)_/¯