Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-4912

New Persistence API with higher scalability limits

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: Marathon 1.4.0
    • Component/s: None
    • Labels:

      Description

      Major changes

      • New Persistence API with support for ZK, Memory, and Two caching modes.
      • Move all storage related "stuff" into marathon/core/storage
      • Default the storage backend to the new ZK API.
      • Lazy storage of Groups (new API only) where Apps are stored separately.
      • Only GroupRepository is allowed write access to AppRepository
      • Completely unify the persistence API between new and old, keeping a single, simplified chain of Repositories.
      • Make versioned repositories explicit.
      • Prepare for a future where typesafe config is used as opposed to a million command line options (storage supports it now).
      • Refactor migration a bit to differentiate new API vs old API (needs a bit more work)
      • Run repository tests against all of the backing stores.
      • (in some cases) remove mocks and prefer the in memory implementation where it otherwise makes sense (not to simulate errors, etc).
      • Add ZookeeperServerTest that will spin up/down a zk server and provide a client on a ephemeral port. This has the major advantage of bringing us closer to being able to run tests in parallel.
      • Add a Garbage Collector to the new persistence store that will try to enforce max versions, but in the background.
        The garbage collector is triggered by deployment plan deletion.

      Minor changes

      • run scalastyle for test in compile, not test.
      • add plugin to auto-resolve documentation links
      • remove lots of Guice Injections for Storage.
      • Use immutable.Seq across the persistence layer.
      • Use OffsetDateTime (java8 time) for versioning in the new API as it is strictly better than timestamp.
      • Make some traits have companion objects so they can be imported instead of mixed-in.
      • Add a generic Sink stage that can drain into any collection, then wrap the Akka Sink API in a companion.
      • Add some tooling for enriched Locks.
      • Add scala friendly API around typesafe config.
      • Add RichFuture that adds "asTry" which makes a future always complete, but with a try instead (very nice for async/await)
      • Fix some oversight issues with CapConcurrentExecutions.
        I would like to remove the actor from this and make it a functional transform instead.
      • Add a SettableClock (the java8 time Clock) that lets you control time a bit easier for testing.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jgilanfarr Jason Gilanfarr (Inactive)
                Team:
                Orchestration Team
                Watchers:
                Marco Monaco
              • Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: