Qdrant SQL with QQL 0.4.0: Bulk Upsert, Batch & Migrate

QQL 0.4.0 is the package line after 0.3.0. QQL is a typed Qdrant SQL query layer. It compiles declarative queries into validated Qdrant HTTP and gRPC requests across four environments: Python, Node, Rust, and Edge. You can install 0.4.0 from PyPI, npm, or crates.io. The grammar inside that pin is 1.7.

0.3.0 bound parameters on raw text. 0.4.0 revamps the entire query pipeline into a zero-walk, 32-byte AST and typed intermediate representation.

Five pillars define the 0.4.0 release:

1. AST parameter binding and bulk ingestion. UPSERT INTO docs VALUES :rows pairs with upsert_many across all drivers. Whole point dictionaries splice in place, and packed float buffers bind with a single copy.

2. First-class BATCH blocks. BATCH { QUERY ...; QUERY ... } lowers multiple queries or mutations into a single network roundtrip, sharing one collection and batch family.

3. Contract-driven convert and live traffic capture. qql convert transforms 25 Qdrant OpenAPI endpoints into canonical Qdrant SQL that re-parses by construction. qql record acts as a transparent proxy capturing live REST calls without application changes.

4. Zero-snapshot cluster migration and sharded dumps. qql migrate streams schema and points between clusters with automated shard-key discovery, suppressed indexing thresholds, and crash-safe resume checkpoints. qql dump exports custom-sharded collections into reproducible replay scripts.

5. Closed typed response model and developer tooling. Every backend normalizes into a closed ExecData enum. Python and Node expose native PyO3 and N-API result objects, memory-bounded scroll cursors, a Python DB-API 2.0 interface, and offline static analysis with qql lint --fix.

pip install pyqql==0.4.0
# npm install @veristamp/[email protected]
# cargo add [email protected]

The offline quickstart pipeline runs end-to-end without a server:

import pyqql

q = """
WITH
  dense  AS (QUERY TEXT 'vector databases' FROM docs USING dense  LIMIT 100),
  sparse AS (QUERY TEXT 'vector databases' FROM docs USING sparse LIMIT 100)
QUERY FUSION RRF FROM docs PREFETCH (dense, sparse) LIMIT 10
"""
assert pyqql.is_valid(q)

stmt = pyqql.parse(q)[0]
stmt.inject_filter("tenant_id", "=", "acme")
stmt.shard_key = "acme"

route = pyqql.compile_query(
    "QUERY TEXT :q FROM docs WHERE tenant_id = :t LIMIT :lim",
    {"q": "vector databases", "t": "acme", "lim": 10},
)
assert route["path"] == "/collections/docs/points/query"

tpl = pyqql.parse("UPSERT INTO docs VALUES :rows")[0]
bound = tpl.bind({"rows": [
    {"id": 1, "vector": {"dense": [0.1, 0.2, 0.3]}, "tag": "a"},
]})
assert "id: 1" in str(bound)

That script is examples/quickstart.py in the repository. The Node twin is examples/quickstart.mjs. Live execute and upsert_many run in integration test suites.

Package 0.4.0 and language 1.7 represent distinct version tracks, matching the relationship between package 0.3.0 and language 1.5.

1. AST Parameter Binding and High-Throughput Bulk Ingestion

In QQL 0.4.0, parameter binding moves from string interpolation to the syntax tree. 0.3.0 substituted placeholders in raw query text before lexical analysis. QQL 0.4.0 binds directly on pre-parsed statement structures. You parse the Qdrant SQL statement once, then bind parameters across thousands of executions without re-parsing.

1.1 Pre-Parsed Statement Binding and Dot Notation Expansion

from pyqql import Client

client = Client("http://localhost:6333")
stmt = client.prepare("QUERY TEXT :q FROM docs WHERE tenant_id = :t LIMIT :lim")
rows = client.execute(stmt, params={"q": "refund policy", "t": "acme", "lim": 10})

Nested parameter dictionaries expand using dot notation. Dotted expressions like :loc.lat resolve against {"loc": {"lat": 1}}. When flattened keys collide, QQL rejects the payload fail-closed with QQL-BIND-DUPLICATE-PARAM. Providing {"loc.lat": 1, "loc": {"lat": 2}} returns an explicit error instead of guessing precedence.

Batch parameter binding operates per statement. You pass a list containing one parameter dictionary per statement. A length mismatch raises QQL-BIND-BATCH-LENGTH. Re-binding a previously bound statement raises QQL-BIND-ALREADY-BOUND. Attempting to bind against DDL statements raises QQL-BIND-UNSUPPORTED-STATEMENT.

Placeholders reach every location where literal values appear. Valid positions include LIMIT :lim, OFFSET :off, SCROLL AFTER :cursor, FACET LIMIT :lim, HYBRID TEXT :q, CROSS RERANK :k, and formula expressions like TARGET = :when. Bound LIMIT parameters enforce values greater than zero with QQL-BIND-INVALID-INTEGER.

Literal colons inside query strings like QUERY TEXT ':heart:' remain protected and never trigger placeholder substitution. Triple-quoted strings and raw string literals also bypass replacement.

1.2 Zero-Walk Packed Float Vectors via MemoryViews

Vectors bind directly as raw byte data. Float32Array and Float64Array in Node bind as packed f32 buffers with a single memory copy. NumPy buffers, array.array, and memoryviews follow the same single-copy path in Python via PyMemoryView.

Providing an unadorned ArrayBuffer without a typed float view fails closed with concrete guidance. Python lists containing 32 or more floating-point numbers pack into native float buffers automatically, while smaller or nested lists preserve standard list semantics.

1.3 High-Throughput Bulk Ingestion with upsert_many

Whole points bind as dictionaries in Qdrant SQL:

UPSERT INTO docs VALUES :rows

client.upsert_many("docs", rows, batch_size=100)

await client.upsertMany("docs", rows, { batchSize: 100 });

exec.upsert_many("docs", rows, 100, OnError::Stop).await?;

upsert_many caches collection schema information once during prepare(). Each chunk moves through an ephemeral internal reference slot instead of cloning point dictionaries. This zero-clone path saves roughly 8 milliseconds per 10,000 points, yielding a 7% throughput gain on 128-dimensional vectors¹.

Passing a batch size below one triggers QQL-VALIDATION-UPSERT-BATCH. An empty row list returns an empty success report without dispatching network calls. Setting OnError::Continue collects per-chunk failures while allowing remaining chunks to proceed. Point shapes missing an identifier fail with QQL-VALIDATION-UPSERT-ID, while mismatched field types trigger QQL-BIND-TYPE-MISMATCH.

AST binding stops injection bugs on the write path just as 0.3.0 stopped them on retrieval. It does not replace inject_filter. When accepting untrusted user queries, production systems must continue applying AST-level filter merging as described in Prevent RAG data leaks.

2. First-Class BATCH Blocks in One Roundtrip

Sequential network requests penalize application latency. When a dashboard needs dense hits, sparse hits, and facet aggregations for a single tenant, making separate HTTP requests wastes milliseconds on TCP handshakes and wire overhead.

2.1 Single Wire RPC Execution over Qdrant Batch Endpoints

QQL 0.4.0 introduces the BATCH block. Multiple Qdrant SQL queries or mutations execute inside a single wire RPC call:

BATCH {
  QUERY TEXT 'refund policy' FROM docs WHERE tenant_id = 'acme' LIMIT 10;
  QUERY [0.1, 0.2, 0.3] FROM docs WHERE tenant_id = 'acme' LIMIT 10;
}

Statements inside a BATCH block must target the same collection and share the same operational family. Queries batch with queries, while mutations batch with mutations. Combining a QUERY and an UPSERT in the same block triggers QQL-VALIDATION-BATCH-FAMILY.

2.2 Envelope Header Propagation and Batch Vector Updates

Execution headers apply at the batch envelope level. Setting WAIT true or specifying search parameters like PARAMS (consistency = 'majority') on the BATCH header propagates those settings to every enclosed statement across three transports: REST, gRPC, and Edge. Specifying WAIT or PARAMS inside individual member statements fails closed with QQL-VALIDATION-BATCH-WAIT or QQL-VALIDATION-BATCH-PARAMS.

Ambient multi-statement scripts continue executing statement by statement. The BATCH keyword signifies an intentional, single-roundtrip execution barrier.

Vector updates support batch execution syntax:

UPDATE docs SET VECTOR VALUES {id: 1, vector: {dense: [0.1, 0.2]}}, {id: 2, vector: {dense: [0.3, 0.4]}};

This statement lowers to PUT /points/vectors on REST and UpdatePointVectors on gRPC. Single-point updates remain available via SET VECTOR [name] = ... WHERE id = .... Named vector maps like {dense: ..., sparse: ...} function identically on individual records.

2.3 Language 1.7 Conformance and Extended Filter Expressions

Language 1.7 introduces additional filter expressions. New conditions include MIN SHOULD n (...) threshold logic, MATCH TOKENS, and MATCH EXCEPT. The SCROLL command gains ORDER BY clauses with payload selectors and START FROM pagination cursors.

Mutations add UPDATE FILTER with explicit UPDATE MODE strategies. The formal conformance corpus now spans 41 valid test files containing 300 statements, alongside 73 invalid rejection cases².

3. Bidirectional REST Conversion and Live Traffic Capture

Migrating an established codebase from raw REST calls to typed SQL often stalls on manual query rewriting. QQL 0.4.0 eliminates this friction through automated bidirectional translation and transparent traffic recording.

3.1 Contract-Driven Conversion Across 25 OpenAPI Routes

The qql convert command decodes Qdrant OpenAPI request payloads into a typed AST and renders the result through qql_core::fmt. Because the converter uses the identical code path behind qql fmt, emitted statements adhere to canonical formatting rules and re-parse cleanly across all 25 supported routes.

Geographic query predicates lower directly to GEO_BBOX, GEO_RADIUS, and GEO_POLYGON syntax. Request fields that QQL cannot represent fail closed with a structured ConvertError detailing the exact path, preventing placeholder text from corrupting emitted scripts.

qql convert --collection docs request.json
qql convert capture.jsonl

A single entry point accepts wrapped {method, path, query, body} envelopes, bare JSON request bodies, and streaming JSONL captures. Unrecognized JSON keys trigger immediate validation errors. Query parameters including wait, timeout, and consistency translate into equivalent WAIT and PARAMS clauses. Bare request bodies require an explicit collection argument because QQL rejects statements lacking a target collection.

Durability parameters survive full roundtrip translation. When lowering mutations, the query planner emits ?wait=false explicitly rather than omitting the parameter. Consequently, running plan -> route -> convert -> replan preserves durability guarantees without silently flipping operations to synchronous execution.

3.2 Zero-Code Traffic Capture Proxy with qql record

For applications running in production without code modification, qql record provides a zero-downtime capture path. It runs an in-process proxy using Axum and Reqwest, forwarding inbound REST traffic to target clusters while preserving HTTP headers, status codes, query strings, and authentication credentials.

qql record --target http://localhost:6333 --capture capture.jsonl --qql-out out.qql

The recorder writes raw requests to JSONL and appends converted Qdrant SQL statements to --qql-out. Bodyless routes like collection inspection and index deletion capture cleanly. When a request contains unmapped options, the proxy appends -- ERROR annotations to the output script and continues forwarding production traffic without interruption.

4. Zero-Snapshot Cluster Migration and Lossless Sharded Dumps

Transferring collections between clusters typically relies on native snapshots. While binary snapshots restore quickly on identical hardware, they impose severe operational constraints. Native snapshots require matching minor versions, cannot alter shard topologies, and carry over index fragmentation. Moving across multiple minor versions, such as jumping from Qdrant 1.15 to 1.19.1, forces teams through sequential rolling upgrades.

4.1 Logical Streaming Migration versus Binary Disk Snapshots

qql migrate provides a logical streaming alternative for Qdrant SQL workloads. It reads schema definitions from the source cluster, configures the destination, streams points through checkpointed cursors, and verifies final counts.

Logical migration enables in-flight transformations impossible with binary snapshots. You can increase shard counts on collections with immutable shard topologies, apply turbo4 1.5-bit or 2-bit quantization to shrink RAM footprints by up to 90%, and export filtered subsets with --where "status = 'active'".

qql migrate --to http://target:6333 \
  --batch-size 500 --workers 4 \
  --checkpoint ./ckpt --resume \
  --shard-key-field tenant_id

4.2 Automated Shard-Key Discovery with Fast-Bulk Suppression

The migration pipeline follows five coordinated phases:

1. Schema extraction and replication. The tool reads schema details: vector configurations, distances, and payload indices. It creates the destination collection with optional overrides like --shard-number 12 or --quantize turbo.

2. Fast-bulk indexing suppression. Before streaming points, the tool sets indexing_threshold to 2 GB on the destination. This prevents Qdrant’s optimizer from repeatedly constructing small HNSW graphs mid-stream, eliminating CPU starvation and write stalls.

3. Automated shard-key discovery. Supplying --shard-key-field <field> executes FACET <field> LIMIT 10000 EXACT true on the source. If results reach 10,000 hits or the field lacks an index, it falls back to payload-only scrolling. It registers discovered keys on the destination with is_tenant = true indexes, routing points with typed SHARD keys.

4. Resilient point streaming. Worker pools read source points using cursor-based scrolling and write batches via :rows templates. The --checkpoint flag records progress in atomic JSON state files. If network connections fail, rerunning with --resume continues from the last acknowledged point.

5. Optimizer restoration and verification. A signal handler ensures the optimizer threshold returns to its original setting upon completion, error, or manual interrupt via Ctrl+C. The tool queries exact point counts on both clusters. Passing --cutover <alias> atomically switches a collection alias to the new cluster.

4.3 Lossless Sharded Dumps with qql dump Replay

For file-based workflows, qql dump exports custom-sharded collections into scripts containing CREATE SHARD KEY definitions and shard-routed UPSERT batches. Replaying the output file with qql execute reconstructs the collection with identical counts³.

qql dump my_collection --output ./backup.qql --batch-size 1000
qql execute ./backup.qql --target http://production:6333

qql dump streams records to an atomic temporary file (.tmp) and swaps only on successful completion. Every batch retains its partition context, ensuring custom-sharded multi-tenant setups restore cleanly without manual shard mapping.

5. Developer Tooling: Lint Autofix, Setup Wizard, Fmt, and Doctor

A database query language requires dependable tooling. QQL 0.4.0 expands the developer toolchain to include offline linting, interactive onboarding, canonical script formatting, and automated diagnostics.

5.1 Offline Static Analysis and Autofix with qql lint

qql lint performs static analysis on individual files, directories, or standard input without requiring a live Qdrant cluster:

qql lint ./queries/ --fix
qql lint query.qql --json

The lint engine executes a four-stage inspection pass:

1. Syntax error recovery via Parser::parse_all_recovering, gathering diagnostics across all statements simultaneously.
2. Anti-pattern detection, flagging redundant clauses such as explicit WITH PAYLOAD true on queries and scrolls.
3. Offline plannability verification through qql_plan::compile_statement.
4. Canonical format checks.

When invoked with --fix, the linter excises offending tokens based on exact lexer spans while preserving comments and string literals. The --json flag emits machine-readable reports for automated CI/CD quality gates.

5.2 Canonical Script Formatting with qql fmt

The qql fmt command enforces uniform styling across Qdrant SQL scripts:

qql fmt --check ./queries/
qql fmt --write ./queries/

Formatting rules enforce lowercase keywords, standardized clause line-breaks, aligned CTE blocks, and consistent trailing newlines. Canonical formatting guarantees that formatted statements remain re-parseable without syntactic drift.

5.3 Interactive Onboarding and Diagnostics with qql setup and qql doctor

First-time developers can configure environment settings using the interactive setup wizard:

qql setup

The wizard guides operators through cluster endpoints, API keys, local Edge directories, and default embedding models. Non-interactive environments can supply options directly using --url, --api-key, and --embed-url.

For system diagnostics, qql doctor and qql check run multi-stage triage pipelines:

qql check
qql doctor

qql doctor probes active embedding endpoints (EMBED_URL) to determine actual output dimensions, comparing measurements against collection vector topologies. When mismatches occur, the doctor reports QQL-EMBEDDING-DIM or QQL-BACKEND-DIMENSION-MISMATCH with actionable remediation steps.

6. Engine Revamp: 32-Byte AST Layout and Closed Typed Models

Beneath the user-facing CLI and client drivers, QQL 0.4.0 incorporates major internal rewrites across memory layout, response envelopes, and runtime safety.

6.1 Memory Optimization: Boxed Parameter Spans in Rust Edition 2024

The entire workspace upgrades to Rust Edition 2024. Codebases across qql-core, qql-plan, and qql-runtime use let-chains in pattern matching to simplify AST lowerers and routing logic. Large modules were decomposed into cohesive units under 400 lines, separating parser grammar, statement transforms, execution routines, and gRPC routing.

AST memory footprint received targeted optimization. In previous releases, Span in crates/qql-core/src/error.rs contained two 8-byte integers. Storing Option<Span> alongside parameter strings expanded size_of::<Value>() to 56 bytes due to alignment padding. QQL 0.4.0 boxes parameter spans as Option<Box<Span>>. Rust’s null-pointer niche optimization reduces the span wrapper to an 8-byte pointer, holding size_of::<Value>() at exactly 32 bytes. This compact layout restores CPU cache locality, yielding a 3% to 11% speedup during AST traversal and validation passes.

pub enum Value {
    F32Array(Vec<f32>),
    Param(String, Option<Box<Span>>),
    PositionalParam(usize, Option<Box<Span>>),
}

6.2 Closed ExecData Response Model and Native SDK Classes

The execution layer closes the response model. Previous versions allowed raw JSON passthrough. QQL 0.4.0 normalizes all backend responses into a closed ExecData enum:

pub enum ExecData {
    Hits(Vec<SearchHit>),
    Groups(Vec<GroupedSearchResult>),
    Count(u64),
    Facet(Vec<FacetHit>),
    Mutation { affected: Option<u64> },
    Collections(Vec<String>),
    Collection(CollectionInfo),
    ShardKeys(Vec<PlanShardKey>),
    Quotas(QuotaConfig),
}

REST responses parse once at the transport boundary against OpenAPI schemas. Malformed server payloads fail closed with QQL-BACKEND-ENVELOPE. gRPC and in-process Edge engines populate typed variants directly from protobuf structures and engine structs.

Python and Node drivers return native PyO3 and N-API ExecutionReport and ScoredPoint classes instead of dictionary wrappers:

report = client.execute("QUERY TEXT 'refund' FROM docs LIMIT 5")
hits = report.hits()
ids = report.ids()
print(report.ok, report.succeeded, report.failed)

Typed accessors include .hits(), .points(), .ids(), .facet(), .count(), and .groups(). The .groups() method returns native hit objects. Facet bucket aggregations return {value, count} records without instantiating empty point objects. Floating-point scores format using shortest round-trip representations, printing 0.95 instead of 0.949999988079071.

6.3 Python DB-API 2.0 and Memory-Bounded Scroll Cursors

For Python applications following relational conventions, pyqql.connect() provides a DB-API 2.0 interface:

from pyqql import connect

db = connect("http://localhost:6333")
cur = db.cursor()
cur.execute("QUERY TEXT :q FROM docs LIMIT :lim", {"q": "policy", "lim": 5})
for row in cur.fetchall():
    print(row)

The connection provides standard cursor methods, row iteration, multi-statement isolation via cursor.nextset(), and unified exception handling under QqlError. Calling cur.executemany on VALUES :rows routes directly to bulk upsert_many.

Large-scale retrieval benefits from memory-bounded cursors. Python provides client.scroll_cursor(...), while Node offers client.scrollCursor and client.scrollStream. Cursors buffer at most one page in memory, automatically handle identifier escaping, and detect unadvancing pagination tokens.

Replace removed legacy helpers with their modern equivalents:

Backend error handling maps transport issues to specific error codes. Unauthorized requests raise QQL-BACKEND-AUTH. Missing collections raise QQL-BACKEND-COLLECTION-NOT-FOUND. Vector dimension mismatches raise QQL-BACKEND-DIMENSION-MISMATCH. Outgoing gRPC requests attach unique x-request-id metadata headers, recording request identifiers inside .fields["request_id"] on failure.

Local edge deployments achieve engine parity, as documented in our guide to local vector search on Edge. The qql edge optimize <collection> command triggers merge and vacuum loops, warning when indexing lags behind ingestion. Running qql edge bootstrap streams remote shard snapshots directly into local storage directories with atomic directory swaps. Unsupported storage options fail closed with QQL-EDGE-UNSUPPORTED-WAL and QQL-EDGE-UNSUPPORTED-METADATA.

Local BM25 scoring introduces configurable document parameters. You can adjust k1, b, and avg_len via Bm25Params across Rust, Python, Node, CLI configuration files, and WebAssembly drivers. This tuning applies exclusively to ingestion passes, building on the multi-tenant sparse retrieval benchmarks in per-tenant IDF on Qdrant. Query term weights remain unit-scaled to match Qdrant’s server scoring semantics.

Production deployments managing multi-tenant retrieval can combine these runtime controls with the architecture in QQL retrieval operations.

6.4 Verification Benchmarks Across Runtimes

Developer tooling gains automated verification commands and editor upgrades:

Benchmarks measured on an Intel Core i5-10400F under Rust 1.98.0 with Python 3.14 and Node 24 demonstrate high throughput across parser and planning gates.

Core engine throughput:

Host binding throughput:

Compilation figures reflect local query validation and route generation speed. Actual search latency depends on cluster networking and index size.

Frequently asked questions

Can you query Qdrant using SQL?

QQL provides a typed SQL query layer for Qdrant across Python, Node, Rust, WASM, and Edge runtimes. Rather than writing nested JSON filter objects, developers write declarative SQL statements like QUERY TEXT :q FROM docs WHERE tenant_id = :t. QQL compiles these queries into validated Qdrant HTTP and gRPC requests.

How do you bulk upsert points into Qdrant with QQL?

Call client.upsert_many("docs", rows, batch_size=100) in Python or Node. QQL prepares UPSERT INTO docs VALUES :rows once at the AST level and streams chunks through a zero-clone splice path. Typed arrays like NumPy buffers and Float32Array bind as packed floats with a single memory copy.

How do you execute batch queries in Qdrant in one roundtrip?

Wrap statements in a BATCH { ... } block. QQL lowers all enclosed queries or mutations into a single network RPC wire call. Statements within a batch must share one collection and family, while query headers like WAIT and PARAMS propagate automatically across both REST and gRPC transports.

How do you migrate a Qdrant collection to another cluster without snapshots?

Run qql migrate --to <target> --batch-size 500 --checkpoint ./ckpt --resume. The command copies schema first, suppresses indexing_threshold during bulk ingestion to prevent CPU starvation, verifies exact COUNT parity upon completion, and supports automatic shard-key discovery with --shard-key-field alongside atomic alias cutovers.

How do you convert existing Qdrant JSON requests to SQL?

Run qql convert request.json or qql convert capture.jsonl. It maps 25 Qdrant OpenAPI endpoints into canonical, re-parseable QQL. To record live traffic without code changes, run qql record --target <url> --capture capture.jsonl --qql-out out.qql to capture and convert incoming requests through a transparent proxy.

How do qql lint, fmt, and dump work in CI/CD pipelines?

Run qql lint with --fix or --json to detect and repair query errors and plannability issues offline. Use qql fmt --check to enforce canonical script formatting in git hooks. Export collections with qql dump to generate reproducible SQL replay scripts that preserve sharding topologies.

Pin QQL 0.4.0 for production deployments. Bind parameters directly against statement trees instead of building strings. Execute multi-query batches in single roundtrips. Convert legacy REST payloads before rewriting client code. Migrate collections with durable, resumeable checkpoints. QQL 0.4.0 keeps the offline quickstart suite green without server dependencies. Prior release: QQL 0.3.0. Documentation: qql.veristamp.in(opens in a new tab). Source code: qql-rs repository(opens in a new tab).