huggingface/ratchet

Operation testing suite

Open

#235 opened on Jul 5, 2024

View on GitHubContributor guide
 (4 comments) (1 reaction) (0 assignees)Rust (44 forks)auto 404
help wanted

Repository metrics

Stars
 (768 stars)
PR merge metrics
 (PR metrics pending)

Description

As more and more browsers ship WebGPU, there may be minor discrepancies between implementations. This may cause us significant delays and issues if not addressed.

So, what we need is a test suite like no other. It must fuzz all functionality in all possible deployment settings.

Browser: Chrome, Safari, Firefox, OS: Windows, Macos, Linux

This gives us 7 combinations we need to fuzz all functionality on.

We do not currently do operation tests in the browser because they rely on pytorch for ground truth - this must be resolved by using pre-generated ground truth data (or some other great idea).

This will be done in conjunction with our property based testing, which runs locally and is ground truthed against pytorch.

Contributor guide

Research direction
Research existing WebGPU testing approaches in Rust, plan ground truth data generation to replace PyTorch dependency, and set up a CI matrix for browser/OS combinations for fuzz testing.
Tech stack
rust
Domain
testing qaweb dev
Issue type
Test
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Rust
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40