apache/airflow

reattach_on_restart doesn't work properly as exepected

Closed

#39,791 opened on May 23, 2024

View on GitHubContributor guide
 (13 comments) (0 reactions) (0 assignees)Python (16,781 forks)batch import
area:providersgood first issuekind:bugpending-responseprovider:cncf-kubernetesstale

Repository metrics

Stars
 (44,809 stars)
PR merge metrics
 (Avg merge 7d 18h) (834 merged PRs in 30d)

Description

Apache Airflow version

main (development)

If "Other Airflow 2 version" selected, which one?

2.8.3

What happened?

Issue : When a worker pod is killed , it is expected not to kill the task pods when reattach_on_restart is configured as True.

Case: Our current configuration of airflow includes usage of aws EC2 spot instances so we can say that workers are expected to be killed occasionally when the EC2 instance has be interrupted. When the EC2 instance has been removed from kubernetes nodes , it sends the SIGTERM to the worker pod which invokes the below _execute_task_with_callbacks method and on_kill method is called which will kill the task pod. So , reattach_on_restart won't work as expected since the task pod is deleted when a worker pod is killed. https://github.com/apache/airflow/blob/2d53c1089f78d8d1416f51af60e1e0354781c661/airflow/models/taskinstance.py#L2592-L2613

What you think should happen instead?

The task pod should remain active when a worker pod is terminated if reattach_on_restart is set to True.

How to reproduce

  • Create a task that uses kuberenetes pod operator.
  • Set reattach_on_restart to True in the task.
  • Delete the worker pod which will result in deletion of worker pod and the task pod.

Operating System

Amazon Linux 2

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==8.2.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

Suggestion : The on_kill method should send another parameter which states who called this method (Is it called by airflow UI clear task button or due to SIGTERM signal ) so based on it we can keep the pod or delete the pod. https://github.com/apache/airflow/blob/fe4605a10e26f1b8a180979ba5765d1cb7fb0111/airflow/providers/cncf/kubernetes/operators/pod.py#L989-L1003

If someone helps me on the taskinstance.py code to add the above recommendation , I'll be able to refactor the on_kill method.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Contributor guide

Research direction
Examine the ` execute task with callbacks` method in `airflow/models/taskinstance.py` and the `on kill` method in `airflow/providers/cncf/kubernetes/operators/pod.py`. Understand the current flow when a worker pod receives SIGTERM and how it leads to task pod deletion. Consider the suggested approach of adding a parameter to `on kill` to distinguish between user triggered kill and SIGTERM. Review related comments in the issue for additional context.
Tech stack
pythonkubernetes
Domain
devops
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Understanding of Airflow task lifecycleFamiliarity with KubernetesPodOperatorPython debugging skills
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70