Dune API Gets a Major Overhaul – DuckDB Powers New Pagination, Filtering and Larger Result Sets
By [Your Name] – March 4 2026
San Francisco, CA – Dune Analytics, the data‑driven platform that supplies charts, dashboards and an API for blockchain‑related queries, announced a comprehensive upgrade to its Dune API. The changes were sparked by a recurring request from developers for result‑set pagination and have led the engineering team to adopt DuckDB as the engine behind API query execution. The move lifts the previous 1‑GB result ceiling, introduces a suite of data‑manipulation features and promises sub‑200 ms response times for interactive use cases.
From a Single Feature Request to a System‑wide Redesign
Late in 2023, Dune’s product team received consistent feedback that the existing API could not handle queries returning more than a gigabyte of data and offered no way to page through large result sets. While the original architecture – built around static visualisations for on‑platform dashboards – seldom needed such capabilities, a growing segment of users began employing Dune API for data‑science pipelines, mobile‑app back‑ends and custom analytics tools that require streaming or slicing of massive transaction datasets.
The engineering group initially set out to add a simple pagination layer. In the process they identified a series of constraints tied to the platform’s core components:
- Result size cap – A hard limit of 1 GB was enforced to protect memory usage.
- No native pagination – Visualisations consumed an entire result set at once, making paging unnecessary.
- Heavy reliance on compressed JSON – The format was convenient for small payloads but inefficient for large‑scale processing.
- Cache‑centric execution – Queries were cached for repeated dashboard use, but this model hindered ad‑hoc data extraction.
By stepping back and assessing the broader product roadmap, Dune’s team recognised an opportunity to replace the underlying query engine with a technology better suited for high‑throughput, low‑latency data access.
Why DuckDB?
DuckDB is an embeddable analytical database that runs entirely in‑process, supports both Parquet and JSON data natively, and offers a modern SQL engine optimized for columnar workloads. The platform’s developers highlighted several advantages that aligned with Dune’s needs:
- Scalable result handling – DuckDB can process multi‑gigabyte result sets without the memory pressure that previously forced the 1 GB cap.
- Native Parquet support – Switching from compressed JSON to Parquet cuts storage size and speeds up I/O.
- Fast, deterministic query latency – Benchmarks showed typical query latencies of 100–200 ms, meeting Dune’s interactive performance targets.
- Low operational cost – Running DuckDB inside existing services eliminates the need for a separate, heavyweight analytical engine.
A prototype that loaded a Dune query result into DuckDB and executed a filtered view was built in a single day, confirming the feasibility of the migration.
New API Capabilities
With DuckDB now powering the backend, Dune API offers a richer set of operations:
| Feature | Description |
|---|---|
| Pagination | Clients can request data in configurable page sizes, making it possible to stream results that exceed the former 1 GB limit. |
| Filtering | Column‑level filters can be applied server‑side, reducing the amount of data transferred. |
| Sorting | Results can be ordered by any column before being paginated. |
| Sampling | Uniform random samples of a dataset are available for quick exploratory analysis. |
| Aggregation Functions | Users can request column‑level aggregations (sum, avg, count, etc.) directly via the API. |
| Custom Endpoints | Developers can expose a saved Dune query as a dedicated endpoint, with the new execution engine handling response formatting. |
| Preset Endpoints | Frequently requested datasets are now available through pre‑built, low‑latency endpoints. |
| Parquet Output | The API can return results in Parquet, greatly improving download speed for large datasets. |
The documentation also emphasizes that these capabilities work with both the legacy compressed‑JSON format and the newer Parquet files, ensuring backward compatibility for existing integrations.
Impact on the DeFi Ecosystem
The enhancements are likely to ripple across the decentralized finance (DeFi) community:
- Data‑intensive applications – Analytics platforms, on‑chain risk engines and portfolio trackers can now fetch granular transaction histories without custom ETL pipelines.
- Mobile and edge use‑cases – Smaller, paginated payloads reduce bandwidth consumption, a boon for apps operating on limited connections.
- Research and academic work – Scholars can retrieve full‑size blockchain datasets for machine‑learning models, opening doors to more sophisticated on‑chain behaviour studies.
Industry observers note that the update positions Dune as one of the few blockchain analytics providers that combine an expressive, SQL‑like query language with a lightweight, high‑performance execution engine accessible via a public API.
Key Takeaways
- User‑driven development – A persistent request for pagination catalysed a broader re‑architecture, demonstrating the value of responsive product roadmaps.
- DuckDB adoption – By integrating an embeddable analytical engine, Dune eliminated a hard result‑size ceiling and lowered query latency.
- Expanded functionality – New filtering, sorting, sampling and aggregation endpoints broaden the API’s utility beyond simple data retrieval.
- Future‑proofing – The shift to Parquet and the removal of the 1 GB cap lay the groundwork for handling datasets in the tens of gigabytes, aligning with the growing scale of blockchain data.
- Cost efficiency – Running DuckDB in‑process reduces infrastructure overhead, allowing Dune to maintain low per‑query costs while supporting higher request volumes.
Looking Ahead
Dune’s team acknowledges that the overhaul was a classic case of “feature creep,” where a modest enhancement evolved into a system‑wide upgrade. However, they argue that this approach yielded a more robust platform that can better serve the diverse needs of the DeFi community. The company has pledged to continue gathering feedback, with upcoming releases expected to further refine performance and add new data‑processing primitives.
For developers eager to experiment with the new API, detailed guides and reference documentation are now available on the Dune website. The rollout is already live, and early adopters report markedly faster data retrieval and smoother integration into their pipelines.
For more information, see Dune’s API documentation and the recent blog post outlining the technical details of the DuckDB integration.
Source: https://dune.com/blog/how-weve-improved-dune-api-using-duckdb
