cmu-delphi/delphi-epidata

Improve handling of queries beyond the row limit

Open

#1,020 opened on Nov 1, 2022

View on GitHub
 (1 comment) (0 reactions) (0 assignees)Python (67 forks)auto 404
code healthhelp wantedrefactor

Repository metrics

Stars
 (104 stars)
PR merge metrics
 (PR metrics pending)

Description

Previously, we would process a full query until it hit the limit and then just return nothing to the user

https://github.com/cmu-delphi/delphi-epidata/blob/9844600b0de41f7d8e93bfb7d7dff7e0740b0792/src/server/_printer.py#L90-L99

Then I added a LIMIT clause, but in an unintuitive place

https://github.com/cmu-delphi/delphi-epidata/blob/9844600b0de41f7d8e93bfb7d7dff7e0740b0792/src/server/_query.py#L238-L248

  • The LIMIT addition should probably be in the QueryBuilder object.
  • The row counting might be obsolete since the LIMIT query should handle that (cc @melange396 to confirm). Update: the row counting in the printer is needed when dealing with multiple SQL queries; interestingly, these are only used in non-covidcast endpoints
  • run_query probably doesn't need the p: APrinter then.

Contributor guide

Research direction
Examine the current LIMIT clause placement in query.py and the row counting in printer.py. Understand the QueryBuilder object and how queries are constructed. Consider moving the LIMIT addition to QueryBuilder and assess if run query can be simplified without the printer parameter. Test with non covidcast endpoints that use multiple SQL queries.
Tech stack
python
Domain
backend
Issue type
Refactor
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GitPython
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
75