apache/arrow-java

Performance concern: fillHoles() method and read buffer expansion efficiency.

Open

#599 opened on Feb 10, 2025

View on GitHubContributor guide
 (9 comments) (3 reactions) (0 assignees)Java (132 forks)auto 404
Type: enhancementhelp wanted

Repository metrics

Stars
 (89 stars)
PR merge metrics
 (PR metrics pending)

Description

Hi everyone,

I have the following use case: I’m benchmarking the read throughput performance when dealing with a large number of non-dictionary string columns (300 columns). Based on the profiler output (see the attached picture), I’ve noticed that a significant amount of time is spent in the fillHoles() method, which is part of the read buffer expansion process.

My question is: why is the buffer filled one element at a time instead of using a bulk operation? Wouldn’t a batch approach be more efficient?

Looking forward to your insights. Thanks!

Image

Contributor guide

Research direction
Investigate the fillHoles() method in the context of read buffer expansion. Analyze why it operates element by element and evaluate the feasibility of batch operations to improve throughput. Examine the relevant source code and consider alternative approaches.
Tech stack
java
Domain
data
Issue type
Performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
JavaApache ArrowPerformance Profiling
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60