huggingface/accelerate

Barebones dataloader to allow for any type of iterable dataloader-like object to be used. Should just handle device placement

Closed

#2,975 opened on Jul 29, 2024

View on GitHubContributor guide
 (5 comments) (0 reactions) (0 assignees)Python (626 forks)batch import
good first issue

Repository metrics

Stars
 (5,805 stars)
PR merge metrics
 (Avg merge 16d 17h) (23 merged PRs in 30d)

Description

Make a simplistic version of the DispatchDataLoader which allows for users to easily pass in any Iterable type of object and call it's __iter__. Likely situation to make sure Accelerator knows what's up during prepare is to add an option to the DataLoaderConfiguration for custom_classes which accepts types for us to do isinstance() on

Contributor guide

Research direction
Read the source code of DispatchDataLoader to understand its structure. Identify where DataLoaderConfiguration is defined and modify it to accept custom classes. Create a new class that inherits from Iterable and implements iter , moving data to the device as needed. Finally, update Accelerator.prepare to include an isinstance check for the new class.
Tech stack
pythonpytorch
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GitPythonPyTorch
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
75