QQL 0.3.0: Bound Queries, Tenant IDF, Matching BM25

QQL 0.3.0 is the package line after 0.1.5. Install 0.3.0 from PyPI, npm, or crates.io. The grammar inside that pin is 1.5. Same five runtimes: Python, Node, Rust, WASM, edge.

0.1.5 was one parser across those runtimes. 0.3.0 is three changes on that parser:

1. Bound queries. QUERY TEXT :q FROM docs WHERE tenant_id = :tenant plus params={"q": "...", "tenant": "acme"}. Stop concatenating user text into QQL.
2. Tenant IDF as QQL WHERE. Qdrant 1.19 can score BM25 rarity on one tenant. We measured that on 457k docs: top-1 moved on about 20% of tenant-filtered queries. Language 1.5 writes it as PARAMS (idf = WHERE tenant_id = 'acme'). The JSON corpus object is gone.
3. Local BM25 uses the same token IDs as Qdrant’s server qdrant/bm25. You can upsert from QQL on the laptop and from Qdrant inference on the cluster, same collection. Before 0.3.0, QQL hashed with FNV-1a. Hits looked empty.

Package 0.3.0 / language 1.5 are different numbers, same as 0.1.5 / 1.2.

pip install pyqql==0.3.0
# npm install @veristamp/[email protected]
# cargo add [email protected]

The query that uses all three:

from pyqql import Client

client = Client("http://localhost:6333")
client.execute(
    """
    QUERY TEXT :q FROM recipes USING bm25
    WHERE tenant_id = :tenant
    PARAMS (idf = WHERE tenant_id = :tenant)
    LIMIT :lim
    """,
    params={"q": "How to bake cookies?", "tenant": "bakery", "lim": 5},
)

:q and :tenant are bound before parse. WHERE hides other tenants’ points. idf = WHERE hides other tenants’ vocabularies from BM25 rarity. Those are different jobs. The rest of this post is each one, then the hasher.

Bind and ? instead of concatenating strings

Placeholders are substituted before parse. Named :name, positional ?. Strings and -- comments are never rewritten. Compact dicts ({a:b}) are not placeholders. Write {key: :val} to bind a dict value.

$ is a legal identifier ($score, $1). Using $1 as a placeholder would rewrite payload fields. That is why the placeholders are :name and ?.

Host SDKs (Python, Node, WASM, edge) expose one bind(query, params):

• dict / object → named
• list / array → positional
• mix of :name and ? in one template → QQL-BIND-MIXED-STYLE

Client.execute takes the same params. Rust keeps bind_named / bind_positional and execute_with_params / execute_with_positional_params.

from pyqql import bind

print(bind(
    "QUERY TEXT :q FROM recipes LIMIT :lim",
    {"q": "How to bake cookies?", "lim": 5},
))
# QUERY TEXT 'How to bake cookies?' FROM recipes LIMIT 5

const { Client, bind } = require("@veristamp/nqql");
const client = new Client({ url: "http://localhost:6333" });
await client.execute(
  "QUERY TEXT :q FROM recipes USING bm25 WHERE tenant_id = :tenant LIMIT :lim",
  { params: { q: "How to bake cookies?", tenant: "bakery", lim: 5 } },
);

WASM bind takes a JS object or array, not a JSON string.

Binding stops quote-breaking. It does not replace inject_filter. Untrusted QQL still needs the AST merge in Prevent RAG data leaks. The Go gateway post is archived with qql-go.

Write tenant IDF as WHERE, not a JSON corpus

Matching token IDs gets terms into the inverted index. IDF still decides how rare those terms are.

A payload filter on tenant_id keeps tenant B’s documents out of the result list. It does not keep tenant B’s vocabulary out of tenant A’s BM25 scores. Qdrant 1.19 added params.idf.corpus for that second job. On CQADupStack (457,199 docs, 12 forums as tenants, 13,145 queries) per-tenant IDF changed top-1 on 19.9% of tenant-filtered queries. Labeled nDCG@10 moved +0.19 percentage points. Report the ranking-change rate. Do not sell a leaderboard delta. Full harness, tables, and plots: Qdrant 1.19 per-tenant IDF.¹

QQL 1.4 (package 0.2.1) accepted the Qdrant JSON form:

-- 1.4, removed in 1.5
PARAMS (idf = {corpus: {must: [{key: "tenant_id", match: {value: "acme"}}]}})

Language 1.5 drops that. IDF uses the same filter grammar as retrieval:

QUERY TEXT :q FROM recipes USING bm25
WHERE tenant_id = :tenant
SHARD :tenant
PARAMS (idf = WHERE tenant_id = :tenant)
LIMIT 10

JSON corpora fail at parse with QQL-VALIDATION-IDF. Isolation is not routing is not IDF. The planner lowers idf = WHERE … with the same top_level_filter path as the statement WHERE. Hosts do not build the corpus dict by hand.

If you inject a tenant predicate for retrieval and forget the IDF corpus, BM25 still prices terms against the mixed shard. That is the footgun in the benchmark post. Retrieval filter and IDF corpus are still allowed to differ: score rarity over a whole tenant, restrict hits further by year or status.

Mix local BM25 with Qdrant’s server BM25

Qdrant can embed qdrant/bm25 on the server.² You can also embed on the client (QQL, FastEmbed, a custom hasher). Sparse search is an inverted index over integer term IDs. The original token is not stored. If "cookies" hashes to different integers on the two paths, a query from one never hits a document from the other. Nothing in the API explains the empty list.

Until 0.3.0, QQL’s client BM25 used FNV-1a with a length prefix. Internally consistent. Not the server’s ID space.

0.3.0 uses the server defaults: word tokenizer (split on non-alphanumeric), Unicode lowercase, English stopwords (179 words), English snowball (recipe → recip, baking → bake, cookies → cooki), murmur3-32 seed 0. Queries get unit weights. Documents get BM25 tf saturation (k1=1.2, b=0.75, avg_len=256). IDF is still applied at search via modifier = 'idf'. It is not stored in the vector.

A golden test pins Qdrant’s own inference example. "Recipe for baking chocolate chip cookies" stores indices [112174620, 177304315, 662344706, 771857363, 1617337648] with weight 1.6697302 on every term (dl=5 after dropping for).³⁴

The pipeline is English-only, like the server defaults. Non-English corpora should use server-side inference with explicit language / stemmer / stopwords.

use qql_embed::SparseEmbedder;

let doc = SparseEmbedder::embed_document(
    "Recipe for baking chocolate chip cookies",
);
let mut ids = doc.indices.clone();
ids.sort_unstable();
assert_eq!(
    ids,
    vec![112174620, 177304315, 662344706, 771857363, 1617337648]
);

from pyqql import Client

client = Client("http://localhost:6333")
client.execute("""
CREATE COLLECTION recipes (
  bm25 SPARSE WITH SPARSE (modifier = 'idf')
);
CREATE INDEX ON COLLECTION recipes FOR tenant_id
  TYPE keyword WITH (is_tenant = true);
""")
client.execute("""
UPSERT INTO recipes VALUES {
  id: 1,
  text: 'Recipe for baking chocolate chip cookies',
  tenant_id: 'bakery'
} USING SPARSE ON FIELD text INTO bm25
""")

USING SPARSE without a MODEL name is this local hasher. MODEL 'qdrant/bm25' is the same token space when the host embedder is this pipeline, or when the server infers it. SPLADE / BGE-M3 sparse is a different space.

On QQL Edge, sparse defaults to the same local BM25. No ONNX download. Set sparse_model only for SPLADE / BGE-M3. See local edge search.⁵

After that, a Qdrant Document(text, model="qdrant/bm25") upsert into the same named vector is legal. Point 1 from QQL and point 2 from server inference both match QUERY TEXT 'cookies' FROM recipes USING bm25.

Microbenchmark on an i5-10400F, 100,000 iterations: 256,174 docs/s, 865,066 queries/s. Slow versus the old bare FNV hasher. Irrelevant next to a dense ONNX pass.

Custom Embedder implementations that overrode embed_sparse move to embed_sparse_query (search, unit weights) and embed_sparse_document / _batch (ingest, tf saturation). Defaults need no changes.

Re-embed existing sparse collections

No compatibility mode. FNV IDs and murmur3 IDs in one named vector is a corrupted index.

1. Pin pyqql==0.3.0 (or [email protected] / [email protected]) everywhere that writes sparse vectors.
2. Recreate the sparse vector, or the collection, with modifier = 'idf'.
3. Replay upserts.
4. Confirm the golden string before backfilling production. If that cookie sentence does not produce those five IDs, you are not on 0.3.0’s hasher.
5. Only then enable server-side qdrant/bm25 inference on that vector.

VS Code extension 0.3.0 ships the rebuilt nodejs-target WASM bundle, so diagnostics, format, and bind match the crate. 0.2.4 already fixed the 0.2.3 load failure. 0.3.0 realigns the language surface (:name, idf = WHERE …).

What else is in the 0.3.0 tag

Parser throughput on the same i5: simple QUERY at 1,228,994 ops/s in Rust, 1,438,309 through pyqql. Compile-path numbers, not Qdrant search latency.

0.2.2 was bumped on dev and never tagged. Its notes shipped in 0.3.0. Install 0.3.0.

Frequently asked questions

What is QQL 0.3.0?

The current package line of QQL, a SQL-like language for Qdrant. Language version inside it is 1.5. After 0.1.5: parameter binding, idf = WHERE …, and client BM25 token IDs that match Qdrant server inference.

How do I write per-tenant IDF in QQL?

PARAMS (idf = WHERE tenant_id = 'acme') on a sparse query, plus a retrieval WHERE (or inject_filter) for isolation. Do not send a Qdrant JSON {corpus: {must: […]}} object. That form is QQL-VALIDATION-IDF. Why the extra filter exists, and the 20% top-1 number: per-tenant IDF benchmark.

Can I mix QQL client BM25 with Qdrant server qdrant/bm25?

Yes, on 0.3.0, if the sparse vector has modifier = 'idf' and the collection was written or re-embedded by this hasher. FastEmbed Qdrant/bm25 is the same family. SPLADE and BGE-M3 sparse are not.

Do I have to re-embed my QQL sparse collection?

Yes, if it was written before 0.3.0. FNV-1a to murmur3-32. No remap.

Why not $1 for parameters?

$ is an identifier character. Fields like $score would be rewritten. Placeholders are :name and ? only.

Pin 0.3.0. Bind instead of concat. Pair WHERE with idf = WHERE. Re-embed sparse collections. Previous pin: QQL 0.1.5. Scoring numbers: per-tenant IDF. Docs: qql.veristamp.in(opens in a new tab). PRs: #34(opens in a new tab) (IDF WHERE), #36(opens in a new tab) (bind), #38(opens in a new tab) (BM25), #44(opens in a new tab) (release).⁶