.. Automatically generated by code2rst.py Edit src/vtable.c not this file! .. currentmodule:: apsw .. _virtualtables: Virtual Tables ************** `Virtual Tables `__ let a developer provide an underlying table implementations, while still presenting a normal SQL interface to the user. The person writing SQL doesn't need to know or care that some of the tables come from elsewhere. Some examples of how you might use this: * Translating to/from information stored in other formats * Accessing the data remotely (eg you could make a table that backends into the cloud) * Dynamic information (eg currently running processes, files and directories, objects in your program) * There are other examples on the `SQLite page `__ .. tip:: You'll find initial development a lot quicker by using :meth:`apsw.ext.make_virtual_module` which lets you export a Python function as a virtual table, being able to provide positional and keyword arguments as part of your query. See :ref:`the example `. To write a virtual table, you need to have 3 types of object. A :class:`module ` providing the module, a :class:`virtual table `, and a :class:`cursor ` that moves through a table. IndexInfo class =============== .. class:: IndexInfo IndexInfo represents the `sqlite3_index_info `__ and associated methods used in the :meth:`VTTable.BestIndexObject` method. Naming is identical to the C structure rather than Pythonic. You can access members directly while needing to use get/set methods for array members. You will get :exc:`ValueError` if you use the object outside of an BestIndex method. :meth:`apsw.ext.index_info_to_dict` provides a convenient representation of this object as a :class:`dict`. .. attribute:: IndexInfo.colUsed :type: set[int] (Read-only) Columns used by the statement. Note that a set is returned, not the underlying integer. .. index:: sqlite3_vtab_distinct .. attribute:: IndexInfo.distinct :type: int (Read-only) How the query planner would like output ordered if the query is using group by or distinct. Calls: `sqlite3_vtab_distinct `__ .. attribute:: IndexInfo.estimatedCost :type: float Estimated cost of using this index .. attribute:: IndexInfo.estimatedRows :type: int Estimated number of rows returned .. method:: IndexInfo.get_aConstraintUsage_argvIndex(which: int) -> int Returns *argvIndex* for *aConstraintUsage[which]* .. index:: sqlite3_vtab_in .. method:: IndexInfo.get_aConstraintUsage_in(which: int) -> bool Returns True if the constraint is *in* - eg column in (3, 7, 9) Calls: `sqlite3_vtab_in `__ .. method:: IndexInfo.get_aConstraintUsage_omit(which: int) -> bool Returns *omit* for *aConstraintUsage[which]* .. index:: sqlite3_vtab_collation .. method:: IndexInfo.get_aConstraint_collation(which: int) -> str Returns collation name for *aConstraint[which]* Calls: `sqlite3_vtab_collation `__ .. method:: IndexInfo.get_aConstraint_iColumn(which: int) -> int Returns *iColumn* for *aConstraint[which]* .. method:: IndexInfo.get_aConstraint_op(which: int) -> int Returns *op* for *aConstraint[which]* .. index:: sqlite3_vtab_rhs_value .. method:: IndexInfo.get_aConstraint_rhs(which: int) -> SQLiteValue Returns right hand side value if known, else None. Calls: `sqlite3_vtab_rhs_value `__ .. method:: IndexInfo.get_aConstraint_usable(which: int) -> bool Returns *usable* for *aConstraint[which]* .. method:: IndexInfo.get_aOrderBy_desc(which: int) -> bool Returns *desc* for *aOrderBy[which]* .. method:: IndexInfo.get_aOrderBy_iColumn(which: int) -> int Returns *iColumn* for *aOrderBy[which]* .. attribute:: IndexInfo.idxFlags :type: int Mask of :attr:`SQLITE_INDEX_SCAN flags ` .. attribute:: IndexInfo.idxNum :type: int Number used to identify the index .. attribute:: IndexInfo.idxStr :type: Optional[str] Name used to identify the index .. attribute:: IndexInfo.nConstraint :type: int (Read-only) Number of constraint entries .. attribute:: IndexInfo.nOrderBy :type: int (Read-only) Number of order by entries .. attribute:: IndexInfo.orderByConsumed :type: bool True if index output is already ordered .. method:: IndexInfo.set_aConstraintUsage_argvIndex(which: int, argvIndex: int) -> None Sets *argvIndex* for *aConstraintUsage[which]* .. index:: sqlite3_vtab_in .. method:: IndexInfo.set_aConstraintUsage_in(which: int, filter_all: bool) -> None If *which* is an *in* constraint, and *filter_all* is True then your :meth:`VTCursor.Filter` method will have all of the values at once. Calls: `sqlite3_vtab_in `__ .. method:: IndexInfo.set_aConstraintUsage_omit(which: int, omit: bool) -> None Sets *omit* for *aConstraintUsage[which]* VTModule class ============== .. class:: VTModule .. note:: There is no actual *VTModule* class - it is shown this way for documentation convenience and is present as a `typing protocol `__. A module instance is used to create the virtual tables. Once you have a module object, you register it with a connection by calling :meth:`Connection.create_module`:: # make an instance mymod=MyModuleClass() # register the vtable on connection con con.create_module("modulename", mymod) # tell SQLite about the table con.execute("create VIRTUAL table tablename USING modulename('arg1', 2)") The create step is to tell SQLite about the existence of the table. Any number of tables referring to the same module can be made this way. .. method:: VTModule.Connect(connection: Connection, modulename: str, databasename: str, tablename: str, *args: tuple[SQLiteValue, ...]) -> tuple[str, VTTable] The parameters and return are identical to :meth:`~VTModule.Create`. This method is called when there are additional references to the table. :meth:`~VTModule.Create` will be called the first time and :meth:`~VTModule.Connect` after that. The advise is to create caches, generated data and other heavyweight processing on :meth:`~VTModule.Create` calls and then find and reuse that on the subsequent :meth:`~VTModule.Connect` calls. The corresponding call is :meth:`VTTable.Disconnect`. If you have a simple virtual table implementation, then just set :meth:`~VTModule.Connect` to be the same as :meth:`~VTModule.Create`:: class MyModule: def Create(self, connection, modulename, databasename, tablename, *args): # do lots of hard work Connect=Create `SQLite xConnect reference `__ .. method:: VTModule.Create(connection: Connection, modulename: str, databasename: str, tablename: str, *args: tuple[SQLiteValue, ...]) -> tuple[str, VTTable] Called when a table is first created on a :class:`connection `. :param connection: An instance of :class:`Connection` :param modulename: The string name under which the module was :meth:`registered ` :param databasename: The name of the database. `main`, `temp`, the name in `ATTACH `__ :param tablename: Name of the table the user wants to create. :param args: Any arguments that were specified in the `create virtual table `_ statement. :returns: A list of two items. The first is a SQL `create table `_ statement. The columns are parsed so that SQLite knows what columns and declared types exist for the table. The second item is an object that implements the :class:`table ` methods. The corresponding call is :meth:`VTTable.Destroy`. `SQLite xCreate reference `__ .. method:: VTModule.ShadowName(table_suffix: str) -> bool This method is called to check if *table_suffix* is a `shadow name `__ The default implementation always returns *False*. If a virtual table is created using this module named :code:`example` and then a real table is created named :code:`example_content`, this would be called with a *table_suffix* of :code:`content` `SQLite xShadowName reference `__ VTTable class ============= .. class:: VTTable .. note:: There is no actual *VTTable* class - it is shown this way for documentation convenience and is present as a `typing protocol `__. The :class:`VTTable` object contains knowledge of the indices, makes cursors and can perform transactions. A virtual table is structured as a series of rows, each of which has the same number of columns. The value in a column must be one of the `5 supported types `_, but the type can be different between rows for the same column. The virtual table routines identify the columns by number, starting at zero. Each row has a **unique** 64 bit integer `rowid `_ with the :class:`Cursor ` routines operating on this number, as well as some of the :class:`Table ` routines such as :meth:`UpdateChangeRow `. It is possible to `not have a rowid `__ .. method:: VTTable.Begin() -> None This function is used as part of transactions. You do not have to provide the method. `SQLite xBegin reference `__ .. method:: VTTable.BestIndex(constraints: Sequence[tuple[int, int]], orderbys: Sequence[tuple[int, int]]) -> Any This is a complex method. To get going initially, just return *None* and you will be fine. You should also consider using :meth:`BestIndexObject` instead. Implementing this method reduces the number of rows scanned in your table to satisfy queries, but only if you have an index or index like mechanism available. .. note:: The implementation of this method differs slightly from the `SQLite documentation `__ for the C API. You are not passed "unusable" constraints. The argv/constraintarg positions are not off by one. In the C api, you have to return position 1 to get something passed to :meth:`VTCursor.Filter` in position 0. With the APSW implementation, you return position 0 to get Filter arg 0, position 1 to get Filter arg 1 etc. The purpose of this method is to ask if you have the ability to determine if a row meets certain constraints that doesn't involve visiting every row. An example constraint is ``price > 74.99``. In a traditional SQL database, queries with constraints can be speeded up `with indices `_. If you return None, then SQLite will visit every row in your table and evaluate the constraints itself. Your index choice returned from BestIndex will also be passed to the :meth:`~VTCursor.Filter` method on your cursor object. Note that SQLite may call this method multiple times trying to find the most efficient way of answering a complex query. **constraints** You will be passed the constraints as a sequence of tuples containing two items. The first item is the column number and the second item is the operation. Example query: ``select * from foo where price > 74.99 and quantity<=10 and customer='Acme Widgets'`` If customer is column 0, price column 2 and quantity column 5 then the constraints will be:: (2, apsw.SQLITE_INDEX_CONSTRAINT_GT), (5, apsw.SQLITE_INDEX_CONSTRAINT_LE), (0, apsw.SQLITE_INDEX_CONSTRAINT_EQ) Note that you do not get the value of the constraint (ie "Acme Widgets", 74.99 and 10 in this example). If you do have any suitable indices then you return a sequence the same length as constraints with the members mapping to the constraints in order. Each can be one of None, an integer or a tuple of an integer and a boolean. Conceptually SQLite is giving you a list of constraints and you are returning a list of the same length describing how you could satisfy each one. Each list item returned corresponding to a constraint is one of: None This means you have no index for that constraint. SQLite will have to iterate over every row for it. integer This is the argument number for the constraintargs being passed into the :meth:`~VTCursor.Filter` function of your :class:`cursor ` (the values "Acme Widgets", 74.99 and 10 in the example). (integer, boolean) By default SQLite will check what you return. For example if you said that you had an index on price and so would only return rows greater than 74.99, then SQLite will still check that each row you returned is greater than 74.99. If the boolean is True then SQLite will not double check, while False retains the default double checking. Example query: ``select * from foo where price > 74.99 and quantity<=10 and customer=='Acme Widgets'``. customer is column 0, price column 2 and quantity column 5. You can index on customer equality and price. +----------------------------------------+--------------------------------+ | Constraints (in) | Constraints used (out) | +========================================+================================+ | :: | :: | | | | | (2, apsw.SQLITE_INDEX_CONSTRAINT_GT), | 1, | | (5, apsw.SQLITE_INDEX_CONSTRAINT_LE), | None, | | (0, apsw.SQLITE_INDEX_CONSTRAINT_EQ) | 0 | | | | +----------------------------------------+--------------------------------+ When your :class:`~VTCursor.Filter` method in the cursor is called, constraintarg[0] will be "Acme Widgets" (customer constraint value) and constraintarg[1] will be 74.99 (price constraint value). You can also return an index number (integer) and index string to use (SQLite attaches no significance to these values - they are passed as is to your :meth:`VTCursor.Filter` method as a way for the BestIndex method to let the :meth:`~VTCursor.Filter` method know which of your indices or similar mechanism to use. **orderbys** The second argument to BestIndex is a sequence of orderbys because the query requested the results in a certain order. If your data is already in that order then SQLite can give the results back as is. If not, then SQLite will have to sort the results first. Example query: ``select * from foo order by price desc, quantity asc`` Price is column 2, quantity column 5 so orderbys will be:: (2, True), # True means descending, False is ascending (5, False) **Return** You should return up to 5 items. Items not present in the return have a default value. 0: constraints used (default None) This must either be None or a sequence the same length as constraints passed in. Each item should be as specified above saying if that constraint is used, and if so which constraintarg to make the value be in your :meth:`VTCursor.Filter` function. 1: index number (default zero) This value is passed as is to :meth:`VTCursor.Filter` 2: index string (default None) This value is passed as is to :meth:`VTCursor.Filter` 3: orderby consumed (default False) Return True if your output will be in exactly the same order as the orderbys passed in 4: estimated cost (default a huge number) Approximately how many disk operations are needed to provide the results. SQLite uses the cost to optimise queries. For example if the query includes *A or B* and A has 2,000 operations and B has 100 then it is best to evaluate B before A. **A complete example** Query is ``select * from foo where price>74.99 and quantity<=10 and customer=="Acme Widgets" order by price desc, quantity asc``. Customer is column 0, price column 2 and quantity column 5. You can index on customer equality and price. :: BestIndex(constraints, orderbys) constraints= ( (2, apsw.SQLITE_INDEX_CONSTRAINT_GT), (5, apsw.SQLITE_INDEX_CONSTRAINT_LE), (0, apsw.SQLITE_INDEX_CONSTRAINT_EQ) ) orderbys= ( (2, True), (5, False) ) # You return ( (1, None, 0), # constraints used 27, # index number "idx_pr_cust", # index name False, # results are not in orderbys order 1000 # about 1000 disk operations to access index ) # Your Cursor.Filter method will be called with: 27, # index number you returned "idx_pr_cust", # index name you returned "Acme Widgets", # constraintarg[0] - customer 74.99 # constraintarg[1] - price `SQLite xBestIndex reference `__ .. method:: VTTable.BestIndexObject(index_info: IndexInfo) -> bool This method is called instead of :meth:`BestIndex` if *use_bestindex_object* was *True* in the call to :meth:`Connection.create_module`. Use the :class:`IndexInfo` to tell SQLite about your indexes, and extract other information. Return *True* to indicate all is well. If you return *False* or there is an error, then `SQLITE_CONSTRAINT `__ is returned to SQLite. `SQLite xBestIndex reference `__ .. method:: VTTable.Commit() -> None This function is used as part of transactions. You do not have to provide the method. `SQLite xCommit reference `__ .. method:: VTTable.Destroy() -> None The opposite of :meth:`VTModule.Create`. This method is called when the table is no longer used. Note that you must always release resources even if you intend to return an error, as it will not be called again on error. `SQLite xDestroy reference `__ .. method:: VTTable.Disconnect() -> None The opposite of :meth:`VTModule.Connect`. This method is called when a reference to a virtual table is no longer used, but :meth:`VTTable.Destroy` will be called when the table is no longer used. `SQLite xDisconnect reference `__ .. method:: VTTable.FindFunction(name: str, nargs: int) -> None | Callable | tuple[int, Callable] Called to find if the virtual table has its own implementation of a particular scalar function. You do not have to provide this method. :param name: The function name :param nargs: How many arguments the function takes Return *None* if you don't have the function. Zero is then returned to SQLite. Return a callable if you have one. One is then returned to SQLite with the function. Return a sequence of int, callable. The int is returned to SQLite with the function. This is useful for *SQLITE_INDEX_CONSTRAINT_FUNCTION* returns. It isn't possible to tell SQLite about exceptions in this function, so an :ref:`unraisable exception ` is used. .. seealso:: * :meth:`Connection.overload_function` `SQLite xFindFunction reference `__ .. method:: VTTable.Integrity(schema: str, name: str, is_quick: int) -> str | None If present, check the integrity of the virtual table. :param schema: Database name `main`, `temp`, the name in `ATTACH `__ :param name: Name of the table :param is_quick: 0 if `pragma integrity_check `__ was used, 1 if `pragma quick_check `__ was used, and may contain other values in the future. :returns: None if there are no problems, else a string to be used as an error message. The string is returned to the pragma as is, so it is recommended that you include the database and table name to clarify what database and table the message is referring to. `SQLite xIntegrity reference `__ .. method:: VTTable.Open() -> VTCursor Returns a :class:`cursor ` object. `SQLite xOpen reference `__ .. method:: VTTable.Release(level: int) -> None Release nested transactions back to *level*. If you do not provide this method then the call succeeds (matching SQLite behaviour when no callback is provided). `SQLite xRelease reference `__ .. method:: VTTable.Rename(newname: str) -> None Notification that the table will be given a new name. If you return without raising an exception, then SQLite renames the table (you don't have to do anything). If you raise an exception then the renaming is prevented. You do not have to provide this method. `SQLite xRename reference `__ .. method:: VTTable.Rollback() -> None This function is used as part of transactions. You do not have to provide the method. `SQLite xRollbackTo reference `__ .. method:: VTTable.Savepoint(level: int) -> None Set nested transaction to *level*. If you do not provide this method then the call succeeds (matching SQLite behaviour when no callback is provided). `SQLite xSavepoint reference `__ .. method:: VTTable.Sync() -> None This function is used as part of transactions. You do not have to provide the method. `SQLite xSync reference `__ .. method:: VTTable.UpdateChangeRow(row: int, newrowid: int, fields: tuple[SQLiteValue, ...]) -> None Change an existing row. You may also need to change the rowid - for example if the query was ``UPDATE table SET rowid=rowid+100 WHERE ...`` :param row: The existing 64 bit integer rowid :param newrowid: If not the same as *row* then also change the rowid to this. :param fields: A tuple of values the same length and order as columns in your table `SQLite xUpdate reference `__ .. method:: VTTable.UpdateDeleteRow(rowid: int) -> None Delete the row with the specified *rowid*. :param rowid: 64 bit integer `SQLite xUpdate reference `__ .. method:: VTTable.UpdateInsertRow(rowid: Optional[int], fields: tuple[SQLiteValue, ...]) -> Optional[int] Insert a row with the specified *rowid*. :param rowid: *None* if you should choose the rowid yourself, else a 64 bit integer :param fields: A tuple of values the same length and order as columns in your table :returns: If *rowid* was *None* then return the id you assigned to the row. If *rowid* was not *None* then the return value is ignored. `SQLite xUpdate reference `__ VTCursor class ============== .. class:: VTCursor .. note:: There is no actual *VTCursor* class - it is shown this way for documentation convenience and is present as a `typing protocol `__. The :class:`VTCursor` object is used for iterating over a table. There may be many cursors simultaneously so each one needs to keep track of where in the table it is. .. method:: VTCursor.Close() -> None This is the destructor for the cursor. Note that you must cleanup. The method will not be called again if you raise an exception. `SQLite xClose reference `__ .. method:: VTCursor.Column(number: int) -> SQLiteValue Requests the value of the specified column *number* of the current row. If *number* is -1 then return the rowid. :returns: Must be one one of the :ref:`5 supported types ` `SQLite xColumn reference `__ .. index:: sqlite3_vtab_nochange .. method:: VTCursor.ColumnNoChange(number: int) -> SQLiteValue :meth:`VTTable.UpdateChangeRow` is going to be called which includes values for all columns. However this column is not going to be changed in that update. If you return :attr:`apsw.no_change` then :meth:`VTTable.UpdateChangeRow` will have :attr:`apsw.no_change` for this column. If you return anything else then it will have that value - as though :meth:`VTCursor.Column` had been called. This method will only be called if *use_no_change* was *True* in the call to :meth:`Connection.create_module`. `SQLite xColumn reference `__ Calls: `sqlite3_vtab_nochange `__ .. method:: VTCursor.Eof() -> bool Called to ask if we are at the end of the table. It is called after each call to Filter and Next. :returns: False if the cursor is at a valid row of data, else True .. note:: This method can only return True or False to SQLite. If you have an exception in the method or provide a non-boolean return then True (no more data) will be returned to SQLite. `SQLite xEof reference `__ .. index:: sqlite3_vtab_in_first, sqlite3_vtab_in_next .. method:: VTCursor.Filter(indexnum: int, indexname: str, constraintargs: Optional[tuple]) -> None This method is always called first to initialize an iteration to the first row of the table. The arguments come from the :meth:`~VTTable.BestIndex` or :meth:`~VTTable.BestIndexObject` with constraintargs being a tuple of the constraints you requested. If you always return None in BestIndex then indexnum will be zero, indexstring will be None and constraintargs will be empty). If you had an *in* constraint and set :meth:`IndexInfo.set_aConstraintUsage_in` then that value will be a :class:`set`. `SQLite xFilter reference `__ Calls: * `sqlite3_vtab_in_first `__ * `sqlite3_vtab_in_next `__ .. method:: VTCursor.Next() -> None Move the cursor to the next row. Do not have an exception if there is no next row. Instead return False when :meth:`~VTCursor.Eof` is subsequently called. If you said you had indices in your :meth:`VTTable.BestIndex` return, and they were selected for use as provided in the parameters to :meth:`~VTCursor.Filter` then you should move to the next appropriate indexed and constrained row. `SQLite xNext reference `__ .. method:: VTCursor.Rowid() -> int Return the current rowid. `SQLite xRowid reference `__