We need to be able to migrate data from one version to another in an HA mode.
The issue is we migrate data after leadership election. If M1V1 is killed, M2V1 will take leadership (if M1 had it). M1V2 is started up but is not the leader. We are now running with old data format with a V2 ready to take over leadership. M2 and M3 are killed and replace in the same way... when M1V2 takes over it looks for a framework ID in the "new" layout. not finding it, it will create a new framework ID and "register" (when it should be re-registering).
Now we have orphaned tasks etc.
We should either:
- consider migrating the frameworkID on the fly (this will likely need a version tag that marathon will verify as current and if not go into migration mode)
- force the migrations to run fully before trying to read the frameworkID