Telegraf's statsd plugin on a leading DC/OS master dropped incoming UDP messages during a scale test:
E! Error: statsd message queue full. We have dropped 70000 messages so far. You may want to increase allowed_pending_messages in the config
allowed_pending_messages is currently set to 10000. We should increase this value to something that's less likely to cause dropped metrics at scale. Increasing the value to 100000 resulted in no more dropped metrics, and did not appear to significantly increase Telegraf's memory consumption.
We should also make this change for the dcos_statsd input plugin. It currently hardcodes allowed_pending_messages to 10000.