Iconoclastic idea - Apache airflow

#1

Dear all,
Thank you for developing / maintaining Panoptes, it really is an awesome project !
I learnt so much while trying to read the code / implement my own stuff based on it, and the project goal is really awesome.

Recently, I have been discussing with a few people regarding the workflow / job scheduling from a software point of view, and I met a person that has been deeply involved in developing such a thing, from what I understand quite similar to the awesome job that has been achieved with Panoptes / pytransition.

That person adviced me to use this: https://airflow.apache.org/, along with a messaging system, like the one used in Panoptes (he used rabbitMQ but from what I can see, it looks very similar and as robust and error tolerant as zeroMQ ?)

Of course I know that I would not make sens to rewrite everything fronm the start, but how difficult you think it would be to rewrite panoptes with apache airflow, would it even make sense ?

Thank you for your help and long life to this project

#2

Hi @gnthibault, thanks for the comments and for the ideas! I know @james.synge has had some similar thoughts with regard to the workflow so I’d be curious for him to pipe in.

Part of our long-term (i.e. ~1 year) plan is to move to POCS over to a system where each of the components are less tightly coupled and could more less operate independently of each other. This system necessitates a build-up of the messaging system, which we currently only use for some basic communication.

The pytransitions module that we used is a simple for of the DAGs and accomplished what we needed for getting POCS up and running. I still think it could be expanded on for whatever designs we have as at the core it is a very simple and well put together library (disclaimer: I was an initial contributor and might still be listed as such but haven’t contributed in a few years).

That said, something like airflow is an interesting idea. We will most likely be laying out our roadmap for the software here on the forum or will at least link it here for open discussion so we would love to have your input then! Thanks!

1 Like
#3

I will move to airflow soon for two services:
-meteorological data coming from AAG polling, (likely every minute)
-writing down meteorological report with the nice matplotlib figure (likely every 5 minutes)

I am paranoid about loosing my configuration, so I will most likely setup a docker image with those builtin.