Installation and customization

Linux/BSD provided

Most Linux & BSD distributions have packaged APSW which may trail the SQLite and APSW releases by a year, or more. The distribution provided APSW uses the system wide SQLite library.

Debian

Install python3-apsw

Fedora

Install python3-apsw

Ubuntu

Install python3-apsw

Gentoo

Install dev-python/apsw

Arch

Install python-apsw

There is a full list (150+) of distributions, the package name for APSW, and what APSW version they are currently on.

Building and customization

There are currently several different ways (that all work) from the Python packaging solutions.

Build process

No matter what tool is used, ultimately it invokes setup.py. This is a standard way of building C extensions originally provided by distutils, and now by setuptools.

A series of commands and options are given to setup.py in this pattern:

python setup.py cmdone --option --option value cmdtwo --option \
   cmdthree --option --option value

The only necessary command is build. You can get help by --help:

python setup.py build --help

Each command takes options which can be specified on the command line, or in a configuration file named setup.cfg or setup.apsw. The leading double dash on options is omitted, and dashes should become underscores.

# This is used with pypi source and binary builds

[build]
# download corresponding sqlite release
fetch = True
# all extensions included
enable_all_extensions = True
# ... except icu (not abi stable)
omit = icu
# for Cursor.description_full
enable = COLUMN_METADATA

SQLite options

It is important to understand SQLite's compile time options. They provide control over functionality and APIs included or excluded from SQLite.

APSW needs to know the options chosen so it can adapt. For example if extension loading is omitted from SQLite then APSW also needs to omit the same functionality, otherwise compilation or linking will fail.

Finding SQLite

APSW can fetch SQLite as detailed below, and places it in a sqlite3/ subdirectory. You can place your own SQLite in that directory. If there is a sqlite3.c (ie the amalgamation) then it will be statically included inside APSW. A compiled SQLite will be picked up if present. If none of that is present, then the standard compiler locations are used (eg /usr/include on Unix).

If sqlite3/sqlite3config.h is present it is included before sqlite3/sqlite3.c. It is a good location to put platform configuration which APSW's fetch does automatically by running configure.

Source

It is recommended you get the source from Github releases. If you get the source from PyPi then ensure you edit the setup.apsw file inside.

Verifying your download

Source releases are digitally signed so you can verify they have not been tampered with. Download and extract the corresponding zip file of signatures. These instructions are for GNU Privacy Guard. (GPG is installed as standard on most Unix/Linux platforms and can be downloaded for Windows.)

Verify

To verify a file use --verify specifying the corresponding .asc filename. This example verifies the source:

$ gpg --verify apsw-3.43.1.0.zip.asc
gpg: Signature made ... date ... using DSA key ID 0DFBD904
gpg: Good signature from "Roger Binns <rogerb@rogerbinns.com>"

If you get a "good signature" then the file has not been tampered with and you are good to go.

Getting the signing key

You may not have the signing key available in which case the last line will be something like this:

gpg: Can't check signature: public key not found

You can get a copy of the key using this command:

$ gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys 0DFBD904
gpg: requesting key 0DFBD904 from hkp server keyserver.ubuntu.com
gpg: /home/username/.gnupg/trustdb.gpg: trustdb created
gpg: key 0DFBD904: public key "Roger Binns <rogerb@rogerbinns.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1

Repeat the verify step.

Commands and their options

These are the relevant setup.py commands and their relevant options.

build

Does the complete build. This will invoke build_ext - use only one of build or build_ext.

--fetch

Fetches the corresponding SQLite version

--enable-all-extensions

Enables all the standard extensions

--enable

A comma separated list of options to enable that are normally off omitting the SQLITE_ENABLE prefix. They will be uppercased. eg --enable column_metadata,fts5

--omit

A comma separated list of options to omit that are normally enabled omitting the SQLITE_OMIT prefix. They will be uppercased. eg --omit automatic_index

fetch

This provides more fine grained control over what is fetched.

--version

Specify an explicit version of SQLite to fetch

--fetch-sqlite

Downloads the SQLite amalgamation

--all

Downloads all SQLite components other than the amalgamation. Over time this has included additional extensions and SQLite functions, but currently is nothing.

--missing-checksum-ok

APSW includes checksums of SQLite releases and will fail a fetch if you specify a version for which no checksum is known. This allows proceeding.

build_ext

This performs the compilation of the C code.

--use-system-sqlite-config

Uses ctypes to determine the system wide SQLite library compilation options

--definevalues

Additional #defines separated by commas. eg --definevalues SQLITE_MAX_ATTACHED=37,SQLITE_EXTRA_INIT=mycore_init

--enable-all-extensions

Enables all the standard extensions

--enable

A comma separated list of options to enable that are normally off omitting the SQLITE_ENABLE prefix. They will be uppercased. eg --enable column_metadata,fts5

--omit

A comma separated list of options to omit that are normally enabled omitting the SQLITE_OMIT prefix. They will be uppercased. eg --omit automatic_index

Testing

SQLite itself is extensively tested. It has considerably more code dedicated to testing than makes up the actual database functionality.

APSW includes tests which use the standard Python testing modules to verify correct operation. New code is developed alongside the tests. Reported issues also have test cases to ensure the issue doesn't happen or doesn't happen again.:

$ python3 -m apsw.tests
                Python  /usr/bin/python3 sys.version_info(major=3, minor=10, micro=4, releaselevel='final', serial=0)
Testing with APSW file  /space/apsw/apsw/__init__.cpython-310-x86_64-linux-gnu.so
          APSW version  3.39.2.0
    SQLite lib version  3.39.2
SQLite headers version  3039002
    Using amalgamation  True
...............................................................................................
----------------------------------------------------------------------
Ran 95 tests in 25.990s

OK

The tests also ensure that as much APSW code as possible is executed including alternate paths through the code. 95.5% of the APSW code is executed by the tests. If you checkout the APSW source then there is a script tools/coverage.sh that enables extra code that deliberately induces extra conditions such as memory allocation failures, SQLite returning undocumented error codes etc. That brings coverage up to 99.6% of the code.

A memory checker Valgrind is used while running the test suite. The test suite is run multiple times to make any memory leaks or similar issues stand out. A checking version of Python is also used. See tools/valgrind.sh in the source. The same testing is also done with the compiler's sanitizer option.

To ensure compatibility with the various Python versions, a script downloads and compiles all supported Python versions in both debug and release configurations (and 32 and 64 bit) against the APSW and SQLite supported versions running the tests. See tools/megatest.py in the source.

In short both SQLite and APSW have a lot of testing!