presentations/2026/celery/slides/00-intro.md

# Celery

https://gricad-gitlab.univ-grenoble-alpes.fr/tienyoun/presentations

<img src="./assets/celery.jpg">


---

# Plan

- **Qu'est ce que Celery**
    - Tâche
    - Worker
    - Broker et Queue
- Lancer une tâche
- Débugger Celery
- La parallélisation et les sous-tâches
- Les barres de progression dans Celery



## Qu'est ce que Celery

<img src="./assets/celery_overview.svg">



### Tâche

```python
from celery import shared_task

@shared_task(name="ptf.tasks.test_task")
def test_task():
    print("toto")
```
Une tâche instanciée contient des métadonnées

> /!\ Faire la distinction entre tâche "instanciée" et tâche "déclarée"





### Worker

```bash
$ celery -A ptf_tools worker
```



### Broker et Queue

```bash
$ systemctl status rabbitmq-server.service 
● rabbitmq-server.service - RabbitMQ Messaging Server
     Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; preset: enabled)
     Active: active (running) since Sun 2025-09-07 06:32:54 CEST; 6 months 21 days ago
   Main PID: 2939178 (beam.smp)
      Tasks: 36 (limit: 11885)
     Memory: 149.3M
        CPU: 1d 13h 28min 48.791s
     CGroup: /system.slice/rabbitmq-server.service
             ├─2939178 /usr/lib/erlang/erts-13.1.5/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -sbwt none -sbwtdcpu none -sbwtdio none -- -root /usr/lib/erlang -bindir /usr/lib/erlang/erts-13.1.5/bin -progname erl -- -home /var/lib/>
             ├─2939188 erl_child_setup 65536
             ├─2939260 /usr/lib/erlang/erts-13.1.5/bin/inet_gethost 4
             ├─2939261 /usr/lib/erlang/erts-13.1.5/bin/inet_gethost 4
             └─2939264 /bin/sh -s rabbit_disk_monitor

Sep 07 06:32:49 mathdoc-new-trammel.u-ga.fr systemd[1]: Starting rabbitmq-server.service - RabbitMQ Messaging Server...
Sep 07 06:32:54 mathdoc-new-trammel.u-ga.fr systemd[1]: Started rabbitmq-server.service - RabbitMQ Messaging Server.
```

---

# Plan

- Qu'est ce que Celery
- **Lancer une tâche**
    - Exemple simple
    - Derrière les rideaux...
- Débugger Celery
- La parallélisation et les sous-tâches
- Les barres de progression dans Celery



### Exemple simple
Prérequis : un worker et un broker lancés
```python
from celery import shared_task

@shared_task(name="crawler.tasks.test_task")
def test_task(self):
    print("TEST TASK")

# synchrone dans le thread actuel
test_task()

# asynchrone dans un worker
test_task.delay()
```
> /!\ Pour le worker et django, les fichiers de code sont les mêmes...



### Derrière les rideaux...
<div data-mermaid>
    <script type="text/mermaid">
        %%{init: {'theme': 'light', 'themeVariables': { 'darkMode': false }}}%%
        sequenceDiagram
            actor User
            participant Django
            participant Broker@{ "type" : "database" }
            participant Worker
            participant Postgres@{ "type" : "database" }
            User->>Django: Click button
            Django->>Broker: Send message : schedule task
            Broker->>Worker: Acquire task
            activate Worker
            Worker->>Postgres: Save results
            deactivate Worker
            User ->>Django: Query task result
            Django->>Postgres: Query task result
            Postgres-->>Django: task result
            Django-->>User: task result
    </script>
</div>



<div data-mermaid>
    <script type="text/mermaid">
        %%{init: {'theme': 'light', 'themeVariables': { 'darkMode': false }}}%%
        sequenceDiagram
            actor User
            participant Django
            participant Worker
            participant Postgres@{ "type" : "database" }
            User->>Django: Click button
            Django->>()Worker: Schedule task
            activate Worker
            Worker->>Postgres: Save results
            deactivate Worker
            User ->>Django: Query task result
            Django->>Postgres: Query task result
            Postgres-->>Django: task result
            Django-->>User: task result
    </script>
</div>
---

# Plan

- Qu'est ce que Celery
- Lancer une tâche
- **Débugger Celery**
    - Lister les worker disponibles
    - Lister les tâches réalisables
    - Flower
    - Concepts avancés
- La parallélisation et les sous-tâches
- Les barres de progression dans Celery



### Lister les workers disponibles

`celery -A ptf_tools status`

```bash
env PYTHONPATH=/var/www/ptf_tools/current/src/ /var/www/ptf_tools/shared/venv/bin/celery -A ptf_tools status
```

```bash
$ celery -A ptf_tools status
->  executor@mathdoc-new-trammel.u-ga.fr: OK
->  default@mathdoc-new-trammel.u-ga.fr: OK
->  coordinator@mathdoc-new-trammel.u-ga.fr: OK
3 nodes online.
```



### Lister les tâches réalisables

`celery -A ptf_tools inspect registered`

```bash
env PYTHONPATH=/var/www/ptf_tools/current/src/ /var/www/ptf_tools/shared/venv/bin/celery -A ptf_tools inspect registered
```

```bash
celery -A ptf_tools inspect registered
->  executor@mathdoc-new-trammel.u-ga.fr: OK
   * ptf_tools.tasks.archiving_tasks.archive_numdam_collection
   * ptf_tools.tasks.archiving_tasks.archive_numdam_collections
   * ptf_tools.tasks.archiving_tasks.archive_numdam_resource
   * ptf_tools.tasks.archiving_tasks.archive_trammel_collection
   * ptf_tools.tasks.archiving_tasks.archive_trammel_collections
   * task.tasks.archiving_tasks.archive_collection
   * task.tasks.archiving_tasks.archive_collections
   * task.tasks.archiving_tasks.archive_resource
   * task.tasks.archiving_tasks.check_archive
   * task.tasks.increment_progress
   * task.tasks.tex_tasks.convert_article_tex
   * task.tasks.tex_tasks.remote_tex_to_xml
   * task.tasks.tex_tasks.update_article_body_from_xml
```



## Flower

```bash
env FLOWER_UNAUTHENTICATED_API=true celery -A gdml flower --debug  --broker_api="http://guest:guest@localhost:45253/api/ptf_host"
```

http://localhost:5555/

<img src="./assets/flower.png">



## Concepts avancés
- AsyncResult
- Signature
- Chain
- Group
- Chord

https://docs.celeryq.dev/en/main/reference/celery.html