Support client-side bloom filter blobs for bloom_match

Background

bloom_match(field, {blob}) is a new filter expression that tests approximate set membership against a client pre-built Split-Block Bloom Filter. It shipped server-side in milvus-io/milvus#51140 (merged 2026-07-28) and is on master.

The point is request size. A membership set of 10M int64 values is roughly 90 MB as a raw in [...] list, which blows past the proxy's gRPC receive limit. The same set as a bloom filter is about 14 MB at the default false-positive rate — the client builds the filter, ships the compact blob as an expression template parameter, and the proxy embeds it verbatim after validating the envelope. The server never rebuilds the filter, so every SDK must produce byte-identical blobs.

Design doc: docs/design-docs/design_docs/20260707-bloom-filter-expression.md

What to implement

1. A public helper that turns a membership set (int64 or UTF-8 string) plus a false-positive rate into a blob.
2. A bytes branch in the SDK's existing expression-template-value conversion, so the blob travels as protobuf TemplateValue.bytes_val rather than being base64-inflated through a string field (the body is not valid UTF-8, so a string round trip corrupts it besides).
3. Unit tests that check the produced blob against the shared golden vectors.

Wire format

The blob is a 32-byte MBF1 envelope followed by a Parquet Split-Block Bloom Filter body. All integers little-endian.

offset	size	field
0	4	magic MBF1
4	2	version = 1
6	2	algo = 1 (parquet_sbbf_xxh64)
8	8	n_declared (uint64, informational)
16	8	fpr_declared (float64, informational)
24	4	num_blocks (uint32; body length must equal num_blocks × 32)
28	1	domains bitmask: 0x01 = int64, 0x02 = utf8
29	3	reserved, must be 0
32	…	body

The body is bit-identical to Arrow C++'s parquet::BlockSplitBloomFilter and thus to the parquet-format BloomFilter.md spec:

• The body is a power-of-two number of 32-byte blocks; each block is eight little-endian uint32 words.

• Values are hashed with XXH64, seed 0. An integer hashes its 8-byte little-endian two's-complement encoding; a string hashes its raw UTF-8 bytes. (This matches Parquet plain encoding for INT64 / BYTE_ARRAY.)

• Block index is the multiply-shift reduction ((hash >> 32) * num_blocks) >> 32.

• Within the block, for each word i in 0..7, set bit 1 << ((uint32(hash) * SALT[i]) >> 27), where

  SALT = [0x47b6137b, 0x44974d91, 0x8824ad5b, 0xa2b7289d,
          0x705495c7, 0x2df1424b, 0x9efc4947, 0x5c6bfb31]
  

  Note uint32(hash) is the low 32 bits, the multiply wraps mod 2³², and the shift is logical.

Sizing mirrors Arrow's BlockSplitBloomFilter::OptimalNumOfBytes:

bits = -8 * n / ln(1 - fpr^(1/8))
bits = clamp(bits, 32*8, 128*1024*1024*8)   # MIN_FILTER_BYTES=32, MAX_FILTER_BYTES=128 MiB
bits = round_up_to_power_of_two(bits)
num_bytes = bits / 8

Bounds: fpr ∈ [0.0001, 0.05], default 0.005. An empty member set is legal — it records domains = 0 and matches nothing.

The domains bitmask matters. The two hash domains share one XXH64 output space, so an 8-byte string and the int64 with the same byte image hash identically. Recording which domains were actually inserted is what keeps a probe from aliasing across domains, and it lets the proxy reject a blob built for the wrong field type instead of silently returning fewer rows. Build the blob from the same value domain as the target field: integer fields hash int64, VARCHAR fields hash UTF-8.

Reference implementations

Both are already merged and produce identical bytes:

• Go — client/sbbf/sbbf.go (builder + parser) and client/milvusclient/bloom_filter.go (public API + template wiring). This is the cleanest one to port from.
• Python — pymilvus/client/bloom_filter.py (milvus-io/pymilvus#3690).

Acceptance criteria

• Blob is byte-identical to the shared golden vectors: client/sbbf/testdata/golden_vectors.json (3 cases: int64-only, string-only, mixed-domain) and client/sbbf/testdata/cpp_generated_100_int64.json (100 int64 values generated by Arrow C++ itself). Copy both files into the SDK's test fixtures and compare full hex — this is the real interop check, and it is worth more than any self-consistent round-trip test.
• fpr outside [0.0001, 0.05], NaN, and infinity are rejected.
• Members of mixed type are rejected by the convenience API (one call, one domain).
• The blob reaches the server as TemplateValue.bytes_val, verified by a test on the template-conversion function.
• A size-estimate helper (n, fpr → exact blob size) so callers can check against proxy.maxBloomFilterSize (64 MiB default) before building. Bodies are powers of two, so a count just past a tier boundary doubles the blob and a slightly higher fpr usually brings it back under.

Protobuf dependency

TemplateValue.bytes_val = 6 was added in milvus-io/milvus-proto#632 (commit 7efdf25). SDKs pinned before that commit must bump milvus-proto and regenerate.

message TemplateValue {
  oneof val {
    bool bool_val = 1;
    int64 int64_val = 2;
    double float_val = 3;
    string string_val = 4;
    TemplateArrayValue array_val = 5;
    bytes bytes_val = 6;   // <-- needed
  };
}

Status across SDKs

SDK	status
Go	✅ merged with the server PR (milvus-io/milvus#51140)
Python	🔄 milvus-io/pymilvus#3690 open, awaiting review
Java	📋 milvus-io/milvus-sdk-java#1981
Node	📋 milvus-io/milvus-sdk-node#597
C++	📋 milvus-io/milvus-sdk-cpp#562
Rust	this issue

Rust specifics

Landing spots. There is no generic value→TemplateValue dispatch to extend here; the SDK exposes crate::proto::schema::TemplateValue directly plus one hand-written builder per type:

• QueryOptions (src/v1/query.rs:379, field :385): add_template_value :641, add_template_bool :660, add_template_int64 :678, template_float :696, template_string :714
• SearchOptions (:837, field :846): add_template_value :1050, add_template_bool :1069, add_template_int64 :1087, add_template_float :1105, add_template_string :1123
• src/v1/iterator.rs: add_template_value :174, :312

So the change is an add_template_bytes on each of QueryOptions / SearchOptions (and ideally From<Value<'_>> for TemplateValue, which does not exist today — src/v1/value.rs:12's Value is only wired to FieldData at src/v1/data.rs:526). There is no ArrayVal helper either, if someone wants to fix that in passing.

Minor inconsistency worth cleaning up while in there: QueryOptions has template_float / template_string without the add_ prefix that every other method uses.

Proto needs a bump, and this one has a wrinkle. The milvus-proto submodule is pinned at fd141a09 (2026-06-12), which sits on the 2.6 line and has no bytes_val. Advancing straight to a commit containing 7efdf25 drags in ~73 other commits. Two options: bump the submodule to milvus-proto master (or the 3.0 branch, where the field was backported as ace1491), or cherry-pick just the bytes_val addition onto the 2.6 line. Either way cargo build regenerates template_value::Val::BytesVal automatically via build.rs / tonic-build — nothing is checked in (src/proto/*.rs is gitignored except mod.rs), and protoc is vendored (protoc-bin-vendored), so no system toolchain change.

No XXH64 in the dependency tree — no xxhash-rust, twox-hash, xxh3, seahash, parquet or arrow; in fact no hashing crate at all. Recommend xxhash-rust = { version = "0.8", features = ["xxh64"] }: pure Rust, no C toolchain, leaves the build.rs/protoc story untouched. twox-hash is the alternative. (Rust is the one language here where a hand-written port buys nothing — the crate is small and the arithmetic is exactly what u64's wrapping ops already give you.)

Build & test

git submodule update --init --recursive   # build.rs reads milvus-proto/proto
cargo build
cargo test --lib                          # pure unit tests, no server needed

Almost all tests here are integration targets under tests/v1/* that need a live Milvus. Inline #[cfg(test)] mod tests exists in only five files (src/v1/data.rs, index/mod.rs, mutate.rs, schema.rs, value.rs) and runs under cargo test --lib with no server — that is exactly where the golden-vector conformance test belongs.

DCO is enforced (git commit -s), and mergify carries a do-not-merge/missing-related-issue label, so the PR must reference this issue.