kubeflow/sdk

Refactor Spark options API to clearly distinguish SparkApplication and SparkConnect options

Open

#720 opened on Aug 4, 2026

View on GitHubContributor guide
 (7 comments) (0 reactions) (1 assignee)Python (196 forks)auto 404
area/sparkgood first issuehelp wantedkind/feature

Repository metrics

Stars
 (124 stars)
PR merge metrics
 (PR metrics pending)

Description

Summary

The current Spark options API exposes options through a common entry point even though some options are only applicable to SparkConnect (for example PodTemplateOverride). This can make it difficult for users to determine which options are supported by submit_job() versus connect().

Refactor the options API and documentation to provide a clearer, more consistent experience while preserving existing functionality.

Ref - https://github.com/kubeflow/sdk/pull/685#pullrequestreview-4859037596

Contributor guide

Research direction
Examine the current options API in the kubeflow/sdk repository to see how SparkApplication and SparkConnect options are exposed. Identify which options belong to each, review the existing tests and documentation, and plan a refactor that separates them while keeping backward compatibility.
Tech stack
kubernetespythonspark
Domain
backendapi
Issue type
Refactor
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
GitPython
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65