NVIDIA/numbast

Benchmarks of Existing Operations

Open

#12 opened on Mar 22, 2024

View on GitHubContributor guide
 (1 comment) (0 reactions) (0 assignees)C (22 forks)github user discovery
P0good first issue

Repository metrics

Stars
 (62 stars)
PR merge metrics
 (Avg merge 14d 17h) (4 merged PRs in 30d)

Description

Currently we haven't built a benchmark framework to compare the operations in Numba Kernels with RAW CUDA C++. One pending reason why we haven't done so is that we know there will be a performance gap awaiting LTO support. However, this shouldn't be a blocker for us to build the benchmark framework.

There are two aspects of performances that we want to capture by the benchmarks:

  1. Infrastructure to benchmark the performance gap between releases, this indicates the performance gain we get from optimizing Numbast, Numba, CUDA over time
  2. Infrastructure to benchmark the gap between native CUDA C++ and Numba kernels. This measures the overhead of the additional wrappers we built with Numbast (extra shim, IRs).

Contributor guide

Research direction
Examine the existing Numbast codebase to understand the current structure of kernel operations. Look for any existing test or example files that demonstrate how kernels are defined and executed. Research how CUDA C++ benchmarks are typically structured (e.g., using nsight or custom timing). Then, outline a plan to create a benchmark directory with separate subfolders for Numba and CUDA C++ versions of the same operations, using a common input generator and timing harness.
Tech stack
cpython
Domain
backendperformance
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GitPythonCUDA basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60