Celery

2026-09-27 05:51:06 +00:00 · 2026-04-10 15:50:19 +02:00
parent 8326ee5ea6
commit b47ab39d5a
17 changed files with 89429 additions and 1 deletions
@@ -0,0 +1,230 @@
+# Celery
+<!-- .slide: data-background-image="./assets/celery.jpg" -->
+
+---
+
+# Plan
+
+- **Qu'est ce que Celery**
+    - Tâche
+    - Worker
+    - Broker et Queue
+- Lancer une tâche
+- Débugger Celery
+- La parallélisation et les sous-tâches
+- Les barres de progression dans Celery
+
+
+
+## Qu'est ce que Celery
+
+<img src="./assets/celery_overview.svg">
+
+
+
+### Tâche
+
+```python
+from celery import shared_task
+
+@shared_task(name="ptf.tasks.test_task")
+def test_task():
+    print("toto")
+```
+Une tâche instanciée contient des métadonnées
+
+> /!\ Faire la distinction entre tâche "instanciée" et tâche "déclarée"
+
+
+
+
+
+### Worker
+
+```bash
+$ celery -A ptf_tools worker
+```
+
+
+
+### Broker et Queue
+
+```bash
+$ systemctl status rabbitmq-server.service 
+● rabbitmq-server.service - RabbitMQ Messaging Server
+     Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; preset: enabled)
+     Active: active (running) since Sun 2025-09-07 06:32:54 CEST; 6 months 21 days ago
+   Main PID: 2939178 (beam.smp)
+      Tasks: 36 (limit: 11885)
+     Memory: 149.3M
+        CPU: 1d 13h 28min 48.791s
+     CGroup: /system.slice/rabbitmq-server.service
+             ├─2939178 /usr/lib/erlang/erts-13.1.5/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -sbwt none -sbwtdcpu none -sbwtdio none -- -root /usr/lib/erlang -bindir /usr/lib/erlang/erts-13.1.5/bin -progname erl -- -home /var/lib/>
+             ├─2939188 erl_child_setup 65536
+             ├─2939260 /usr/lib/erlang/erts-13.1.5/bin/inet_gethost 4
+             ├─2939261 /usr/lib/erlang/erts-13.1.5/bin/inet_gethost 4
+             └─2939264 /bin/sh -s rabbit_disk_monitor
+
+Sep 07 06:32:49 mathdoc-new-trammel.u-ga.fr systemd[1]: Starting rabbitmq-server.service - RabbitMQ Messaging Server...
+Sep 07 06:32:54 mathdoc-new-trammel.u-ga.fr systemd[1]: Started rabbitmq-server.service - RabbitMQ Messaging Server.
+```
+
+---
+
+# Plan
+
+- Qu'est ce que Celery
+- **Lancer une tâche**
+    - Exemple simple
+    - Derrière les rideaux...
+- Débugger Celery
+- La parallélisation et les sous-tâches
+- Les barres de progression dans Celery
+
+
+
+### Exemple simple
+Prérequis : un worker et un broker lancés
+```python
+from celery import shared_task
+
+@shared_task(name="crawler.tasks.test_task")
+def test_task(self):
+    print("TEST TASK")
+
+# synchrone dans le thread actuel
+test_task()
+
+# asynchrone dans un worker
+test_task.delay()
+```
+> /!\ Pour le worker et django, les fichiers de code sont les mêmes...
+
+
+
+### Derrière les rideaux...
+<div data-mermaid>
+    <script type="text/mermaid">
+        %%{init: {'theme': 'light', 'themeVariables': { 'darkMode': false }}}%%
+        sequenceDiagram
+            actor User
+            participant Django
+            participant Broker@{ "type" : "database" }
+            participant Worker
+            participant Postgres@{ "type" : "database" }
+            User->>Django: Click button
+            Django->>Broker: Send message : schedule task
+            Broker->>Worker: Acquire task
+            activate Worker
+            Worker->>Postgres: Save results
+            deactivate Worker
+            User ->>Django: Query task result
+            Django->>Postgres: Query task result
+            Postgres-->>Django: task result
+            Django-->>User: task result
+    </script>
+</div>
+
+
+
+<div data-mermaid>
+    <script type="text/mermaid">
+        %%{init: {'theme': 'light', 'themeVariables': { 'darkMode': false }}}%%
+        sequenceDiagram
+            actor User
+            participant Django
+            participant Worker
+            participant Postgres@{ "type" : "database" }
+            User->>Django: Click button
+            Django->>()Worker: Schedule task
+            activate Worker
+            Worker->>Postgres: Save results
+            deactivate Worker
+            User ->>Django: Query task result
+            Django->>Postgres: Query task result
+            Postgres-->>Django: task result
+            Django-->>User: task result
+    </script>
+</div>
+---
+
+# Plan
+
+- Qu'est ce que Celery
+- Lancer une tâche
+- **Débugger Celery**
+    - Lister les worker disponibles
+    - Lister les tâches réalisables
+    - Flower
+    - Concepts avancés
+- La parallélisation et les sous-tâches
+- Les barres de progression dans Celery
+
+
+
+### Lister les workers disponibles
+
+`celery -A ptf_tools status`
+
+```bash
+env PYTHONPATH=/var/www/ptf_tools/current/src/ /var/www/ptf_tools/shared/venv/bin/celery -A ptf_tools status
+```
+
+```bash
+$ celery -A ptf_tools status
+->  executor@mathdoc-new-trammel.u-ga.fr: OK
+->  default@mathdoc-new-trammel.u-ga.fr: OK
+->  coordinator@mathdoc-new-trammel.u-ga.fr: OK
+3 nodes online.
+```
+
+
+
+### Lister les tâches réalisables
+
+`celery -A ptf_tools inspect registered`
+
+```bash
+env PYTHONPATH=/var/www/ptf_tools/current/src/ /var/www/ptf_tools/shared/venv/bin/celery -A ptf_tools inspect registered
+```
+
+```bash
+celery -A ptf_tools inspect registered
+->  executor@mathdoc-new-trammel.u-ga.fr: OK
+   * ptf_tools.tasks.archiving_tasks.archive_numdam_collection
+   * ptf_tools.tasks.archiving_tasks.archive_numdam_collections
+   * ptf_tools.tasks.archiving_tasks.archive_numdam_resource
+   * ptf_tools.tasks.archiving_tasks.archive_trammel_collection
+   * ptf_tools.tasks.archiving_tasks.archive_trammel_collections
+   * task.tasks.archiving_tasks.archive_collection
+   * task.tasks.archiving_tasks.archive_collections
+   * task.tasks.archiving_tasks.archive_resource
+   * task.tasks.archiving_tasks.check_archive
+   * task.tasks.increment_progress
+   * task.tasks.tex_tasks.convert_article_tex
+   * task.tasks.tex_tasks.remote_tex_to_xml
+   * task.tasks.tex_tasks.update_article_body_from_xml
+```
+
+
+
+## Flower
+
+```bash
+env FLOWER_UNAUTHENTICATED_API=true celery -A gdml flower --debug  --broker_api="http://guest:guest@localhost:45253/api/ptf_host"
+```
+
+http://localhost:5555/
+
+<img src="./assets/flower.png">
+
+
+
+## Concepts avancés
+- AsyncResult
+- Signature
+- Chain
+- Group
+- Chord
+
+https://docs.celeryq.dev/en/main/reference/celery.html