OTF: On-the-fly python

The OTF framework is a framework to write, run and debug complex machine distributed workflows. Unlike the traditional DAG based frameworks (e.g.: Luigi, Airflow, Kubeflow…) OTF allows you to use normal python control flow statements in your workflow (e.g.: if and while).

Features

Define by run control flow

Because we rewrite the workflows we can provide async/await semmantics without requiring a python interpreter to keep on running on a machine while we’re awaiting for results.

See Compiling workflows for an in-depth dive.

Explicit serialisation

The function() and environement() decorators make it easy to write code that can be shipped over the network. Serialisation framework cannot always capture the intention of the author of the data/code they are serializing. Whereas cloudpickle and dill implicitly extend the pickle serialisation mechanism, we require code authors to tell us what will actually get sent.

API Reference

The full API documentation is here:

• API
  • Decorators
  • Runtime Classes
  • Serialisation
  • Modules