kedro-org/kedro

Improve `%load_node` for node that doesn't have persisted dataset

Open

#4,169 opened on Sep 16, 2024

View on GitHubContributor guide
 (2 comments) (0 reactions) (1 assignee)Python (864 forks)batch import
HacktoberfestIssue: Feature Request

Repository metrics

Stars
 (9,146 stars)
PR merge metrics
 (Avg merge 8d 11h) (30 merged PRs in 30d)

Description

Description

Originated from https://github.com/kedro-org/vscode-kedro/issues/140

%load_node works great but it doesn't work on MemoryDataset, I don't save every node and it's not easy to figure out which nodes I need to run again to produce the data

Context

  • This is much more powerful because not every node has persisted data and this limited the usage of the feature.
  • This enable much more powerful slicing feature in kedro-viz, which is currently limited because we do not want the slicing generate a command that Kedro does not know how to run.
  • Same as above, if we are able to support more flexible slicing, we may end up expanding the API of kedro run as well since we can support different combinations of slicing

Possible Implementation

Check out the suggest_resume_scanerio in SequentialRunner: https://github.com/kedro-org/kedro/blob/91765e3e70f8a4ba7ef3931a7855aed0ec97eb82/kedro/runner/runner.py#L200-L210

This roughly has the logic to resume pipeline but it's hidden in a private API, we need to surface this for more generic usage.

Contributor guide

Research direction
Examine the `suggest resume scenario` method in `SequentialRunner` and its usage in the pipeline resuming logic. Then implement a generic function that can be used by `%load node` to handle nodes without persisted datasets.
Tech stack
python
Domain
backenddata engineering
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonGit
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60