OTF: On-the-fly python¶
The OTF
framework is a framework to write, run and debug complex machine
distributed workflows. Unlike the traditional DAG based frameworks (e.g.: Luigi, Airflow,
Kubeflow…) OTF allows you to use normal python
control flow statements in your workflow (e.g.: if
and while
).
Features¶
Define by run control flow
Because we rewrite the workflows we can provide
async/await
semmantics without requiring a python interpreter to keep on running on a machine while we’re awaiting for results.See Compiling workflows for an in-depth dive.
Predictable serialisation
The
function()
andenvironement()
decorators make it easy to write code that can be shipped over the network. Serialisation framework cannot always capture the intention of the author of the data/code they are serializing. Whereas cloudpickle and dill implicitly extend the pickle serialisation mechanism, we require code authors to tell us what will actually get sent.Note
OTF is still in early alpha. We currently only offer
pickle
as a serialisation format.
API Reference¶
The full API documentation is here: