Skip to content

Sources

ODBC

Load tables and query results from any database with an ODBC driver: connections, choosing tables, query tables, value sizes and types.

From sqlload 0.27.0, an odbc source loads tables, views and query results from any database that has an ODBC driver. SQLLoad reads the rows as Arrow through the driver and loads them into the source’s target.

From 0.38.0 (not released yet), the sqlload and sqlload-cli wheels include ODBC on every platform. Before that, and in a build from source (cargo install, or pip building an sdist where no wheel fits), ODBC is built only with the odbc feature (cargo install sqlload --features odbc); without it, plan refuses an odbc source.

[connections.erp]
kind = "odbc"
connection_string = "${ENV:ERP_ODBC}"
[sources.erp]
kind = "odbc"
connection = "erp"
description = "Tables of the ERP database."
include = ["sales.*"]
exclude = ["sales.tmp_*"]
max_text_size = 8000
[sources.erp.tables."sales.items"]
description = "Items, one row per item."
cursor_column = "updated_at"
unique_key = ["id"]
write_strategy = "merge"
[sources.erp.tables.recent]
description = "Items added recently."
sql = "SELECT id, name FROM sales.items WHERE id > 2400"
$ sqlload plan
Plan ready 2 selected, 2 objects to create, 2 to load
Objects (2) sqlload apply
├── raw.erp__items create table
│ ├── id int -> BIGINT
│ ├── name text -> VARCHAR
│ ├── price decimal(10,2) -> DECIMAL(10,2)
│ ├── created date -> DATE
│ ├── updated_at timestamp_ntz -> TIMESTAMP
│ └── + _sqlload_load_id, _loaded_at
└── raw.erp__recent create table
├── id int -> BIGINT
├── name text -> VARCHAR
└── + _sqlload_load_id, _loaded_at
Loads (2) sqlload load
├── erp__items merge (unique_key: id)
│ ├── source odbc erp sales.items
│ ├── route read as Arrow -> DuckDB takes Arrow
│ └── cursor updated_at first run, everything
└── erp__recent replace
├── source odbc erp query recent
└── route read as Arrow -> DuckDB takes Arrow
Key Meaning
connection_string The driver’s connection string, such as Driver={PostgreSQL Unicode};Server=db.example.com;Database=erp;Uid=loader;Pwd=.... Use ${ENV:NAME}, since it usually holds a password.
dsn A data source name from odbc.ini, instead of connection_string. It and user can’t contain ;, { or }.
user, password With dsn, when the DSN doesn’t hold them.

Set connection_string or dsn, not both. Install the database’s ODBC driver on the machine SQLLoad runs on; SQLLoad reaches it through a driver manager:

  • Windows: the driver manager built into Windows; register drivers and DSNs as usual (ODBC Data Sources).
  • Linux and macOS wheels (from 0.38.0, not released yet): the unixODBC 2.3.14 driver manager, bundled as a separate shared library, so no system unixODBC is needed. It reads drivers and system DSNs from odbcinst.ini and odbc.ini in /etc on Linux, where distribution driver packages register, and in Homebrew’s etc on macOS (/opt/homebrew/etc on Apple silicon, /usr/local/etc on Intel), or in the folder ODBCSYSINI names; user DSNs from ~/.odbc.ini. A driver registered by file name alone, as Debian and Ubuntu packages register theirs (Driver=psqlodbcw.so), is looked for in /usr/lib/<arch>-linux-gnu/odbc and /usr/lib/<arch>-linux-gnu (then /usr/lib64/odbc and /usr/lib64) on Linux, and Homebrew’s lib on macOS. unixODBC is LGPL-2.1: its notice and licence are in each wheel’s .dist-info/licenses, and its source is in the package’s sdist on PyPI.
  • cargo install --features odbc: the system’s unixODBC, which it links against: unixodbc-dev on Debian and Ubuntu, unixodbc with Homebrew.

SQLLoad writes names in ANSI double quotes ("sales"."items"). A database that doesn’t read them as names is refused when the source is read; switch it to ANSI quoting. For MySQL, add INITSTMT=SET SESSION sql_mode=CONCAT(@@SESSION.sql_mode,',ANSI_QUOTES'); to the connection string.

Driver Tested
psqlODBC (PostgreSQL Unicode) Yes, against Postgres 17 in containers on Linux and on Windows. Add BoolsAsChar=0 to the connection string so boolean columns load as bool rather than text.
Microsoft ODBC Driver 18 for SQL Server Yes, against SQL Server 2022 in containers on Linux and against SQL Server LocalDB on Windows, datetime2 and legacy datetime cursors included.
MySQL Connector/ODBC (MySQL ODBC Unicode Driver) Yes, against MySQL 8.4 in containers on Linux and MySQL 8.4 on Windows, in an ANSI_QUOTES session (above). On Linux, install MySQL’s package for your distribution rather than the generic glibc tarball: the tarball’s own OpenSSL fails to load once another driver, such as SQL Server’s or psqlODBC, has loaded the system one in the same run.
OpenEdge (Progress) Tested for timestamp handling (parsing {ts} values and binding timestamp bounds), not against a live server.
Others Not yet. A driver needs to describe prepared statements and list tables; run sqlload plan to see the types it reports.

From 0.38.0 (not released yet), a cursor filter compares the cursor column with bound parameters, such as "updated_at" > ?, each passed as the cursor’s own type: a timestamp without a time zone as an ODBC TIMESTAMP, with every fractional digit; a date as a DATE; text as text. No database’s literal rules can change a bound, and a millisecond maximum such as 23:59:59.997 resumes exactly. SQL Server’s legacy datetime is the one exception: it stores 1/300 s ticks, which its driver reads rounded to milliseconds, so its bounds go as ISO 8601 text (2026-01-31T23:59:59.997) in the column’s own precision, which SQL Server converts to the column’s type alike under every login language. A cursor column pinned to a type (columns = { ... }) compares with a quoted literal, as the pinned type may not be the column’s own, such as a varchar read as a timestamp. A numeric cursor compares with a bare number, such as "id" > 41. Any other cursor, a timestamp with a time zone for one, still compares with a quoted literal, which the database converts to the column’s type; check that yours does before relying on it. For a timestamp or date cursor, --start-cursor, --end-cursor and cursor_start also take ODBC escape values, such as {ts '2026-01-31 23:59:59.997'} or {d '2026-01-31'}.

From 0.38.0 (not released yet), a timestamp column is read in at least microseconds whatever precision the driver describes: MySQL Connector/ODBC describes every DATETIME(n) with none, and its fractional seconds were dropped before.

include and exclude are schema.table patterns, matched without regard to case (* matches any text). SQLLoad lists the tables and views the driver reports. For a database without schemas, the patterns match the table name alone. A [sources.erp.tables."schema.table"] entry sets options for one table; on its own, it doesn’t add the table.

A source needs at least one include pattern or one query table. The primary key the driver reports becomes the table’s default unique_key.

A table entry with sql is a query table: its rows are the query’s result, and the entry’s key is its name. SQLLoad prepares the query at plan to read its columns, without running it.

cursor_column names a column of the query’s result. SQLLoad wraps the query and filters on the cursor outside it, so the query needs no placeholder. Write the query in the database’s own SQL.

Drivers read values into buffers of a fixed size per column, from the length the driver reports. max_text_size (bytes) caps the buffer of every text column, and max_binary_size that of every binary column, so set them when the driver reports a huge length for unbounded columns such as TEXT or VARCHAR(MAX), or none at all, which the read then refuses, naming the setting. A value longer than its buffer, even in a VARCHAR(10000) column capped at 8000 bytes, fails the read, naming the setting to raise; it’s never cut short.

SQLLoad reads each column’s type as the driver describes it:

ODBC type Loads as
TINYINT, SMALLINT smallint
INTEGER, BIGINT int
NUMERIC(p,s), DECIMAL(p,s), p up to 38 decimal(p,s), read as text so no digit is lost
REAL, FLOAT, DOUBLE float
BIT bool
DATE, TIME date, time
TIMESTAMP timestamp_ntz; a timestamp with a time zone, in the session’s time zone
CHAR, VARCHAR and their wide and long forms text
BINARY, VARBINARY, LONGVARBINARY binary
Anything else, and NUMERIC above 38 digits text, as the driver writes it

A decimal value with more places than its column’s scale fails the read rather than being rounded. psqlODBC describes an unbounded numeric as NUMERIC(28,6), so pin such a column, for example columns = { amount = "decimal(38,10)" } or "text".

Pin a type to load a column another way, such as columns = { total = "decimal(18,2)" }. A value that doesn’t convert to the pinned type fails the read, naming the column. See Types.

The database user needs to read the catalog (to list tables and their keys) and SELECT on the tables and views.