mozilla/overscripted

Can we use array extensions to speed up string processing with dask

Open

#36 opened on Mar 11, 2019

View on GitHub
 (13 comments) (0 reactions) (0 assignees)Jupyter Notebook (48 forks)auto 404
good first issueresearch question

Repository metrics

Stars
 (73 stars)
PR merge metrics
 (PR metrics pending)

Description

Much of the analysis done on this dataset uses dask. Dask is excellent for distributed numerical computing but seems to struggle with strings.

Array extensions unfortunately can't be serialized https://github.com/pandas-dev/pandas/issues/20612

https://github.com/xhochy/fletcher is an array extension that adds string processing functionality.

Doing some standard tasks for this dataset like: collecting all domains, or number of script domains per location domain compare the performance of spark, dask, and dask with extension arrays.

Contributor guide

Research direction
Set up a benchmark to compare string processing performance between Spark, Dask, and Dask with fletcher array extensions using the Overscripted dataset.
Tech stack
python
Domain
data
Issue type
Research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonDask
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60