12. Module Reference¶
This chapter contains all the documentation auto-generated from the source code. It is probably not terribly useful for reading through, but may be useful as a searchable reference.
12.1. piwheels.master¶
12.2. piwheels.master.db¶
12.3. piwheels.master.cloud_gazer¶
12.4. piwheels.master.the_oracle¶
12.5. piwheels.master.seraph¶
12.6. piwheels.master.the_architect¶
12.7. piwheels.master.slave_driver¶
12.8. piwheels.master.mr_chase¶
12.9. piwheels.master.file_juggler¶
12.10. piwheels.master.big_brother¶
12.11. piwheels.master.the_secretary¶
12.12. piwheels.master.the_scribe¶
12.13. piwheels.slave¶
12.14. piwheels.slave.builder¶
12.15. piwheels.initdb¶
Contains the functions that make up the piw-initdb script.
-
piwheels.initdb.
main
(args=None)[source]¶ This is the main function for the piw-initdb script. It creates the piwheels database required by the master or, if it already exists, upgrades it to the current version of the application.
-
piwheels.initdb.
detect_users
(conn, test_user)[source]¶ Test that the user for conn is a cluster superuser (so we can drop and create anything we want in the database), and that test_user (which will be granted limited rights to various objects for the purposes of the piw-master script) exists and is not a cluster superuser.
-
piwheels.initdb.
detect_version
(conn)[source]¶ Detect the version of the database. This is typically done by reading the contents of the
configuration
table, but before that was added we can guess a couple of versions based on what tables exist (or don’t). ReturnsNone
if the database appears uninitialized, and raisesRuntimeError
is the version is so ancient we can’t do anything with it.
-
piwheels.initdb.
get_connection
(dsn)[source]¶ Return an SQLAlchemy connection to the specified dsn or raise
RuntimeError
if the database doesn’t exist (the administrator is expected to create the database before running this script).
-
piwheels.initdb.
get_script
(version=None)[source]¶ Generate the script to get the database from version (the result of
detect_version()
) to the current version of the software. If version isNone
, this is simply the contents of thesql/create_piwheels.sql
script. Otherwise, it is a concatenation of various update scripts.
-
piwheels.initdb.
parse_statements
(script)[source]¶ This is an extremely crude statement splitter for PostgreSQL’s dialect of SQL. It understands
--comments
,"quoted identifiers"
,'string literals'
and$delim$ extended strings $delim$
, but notE'\escaped strings'
or/* C-style comments */
. If you start using such things in the update scripts, you’ll need to extend this function to accommodate them.It returns a generator which yields individiual statements from script, delimited by semi-colon terminators.
12.16. piwheels.importer¶
12.17. piwheels.remove¶
12.18. piwheels.transport¶
12.19. piwheels.protocols¶
12.20. piwheels.tasks¶
12.21. piwheels.states¶
This module defines several classes which permit interested tasks to track the
state of build slaves (SlaveState
), file transfers
(TransferState
), build attempts (BuildState
), build
artifacts (FileState
) and various loggers.
-
class
piwheels.states.
FileState
(filename, filesize, filehash, package_tag, package_version_tag, py_version_tag, abi_tag, platform_tag, dependencies, transferred=False)[source]¶ Represents the state of an individual build artifact (a package file, or wheel) including its
filename
,filesize
, the SHA256filehash
, and various tags extracted from the build. Also tracks whether or not the file has beentransferred
.Parameters: - filename (str) – The original filename of the build artifact.
- filesize (int) – The size of the file in bytes.
- filehash (str) – The SHA256 hash of the file contents.
- package_tag (str) – The package tag extracted from the filename (first “-” separated component).
- package_version_tag (str) – The package version tag extracted from the filename (second “-” separated component).
- py_version_tag (str) – The python version tag extracted from the filename (third from last “-” separated component).
- abi_tag (str) – The python ABI tag extracted from the filename (second from last “-” separated component).
- platform_tag (str) – The platform tag extracted from the filename (last “-” separated component).
- dependencies (set) – The set of dependencies that are required to use this particular wheel.
- transferred (bool) –
True
if the file has been transferred from the build slave that generated it to the file server.
-
as_message
()[source]¶ Convert the
FileState
object into a simpler list for serialization and transport.
-
classmethod
from_message
(value)[source]¶ Convert the output from
as_message()
back into aBuildState
.
-
class
piwheels.states.
BuildState
(slave_id, package, version, abi_tag, status, duration, output, files, build_id=None)[source]¶ Represents the state of a package build including the
package
,version
,status
, buildduration
, and all the lines ofoutput
. Thefiles
attribute is a mapping containing details of each successfully built package file.Parameters: - slave_id (int) – The master’s identifier for the build slave.
- package (str) – The name of the package to build.
- version (str) – The version number of the package to build.
- abi_tag (str) – The ABI for which the build was attempted (must not be
'none'
). - status (bool) –
True
if the build succeeded,False
if it failed. - duration (timedelta) – The amount of time (in seconds) it took to complete the build.
- output (str) – The log output of the build.
- files (dict) – A mapping of filenames to
FileState
objects for each artifact produced by the build. - build_id (int) – The integer identifier generated for the build by the database
(
None
until the build has been inserted into the database).
-
as_message
()[source]¶ Convert the
BuildState
, and its nestedFileState
objects into simpler lists for serialization and transport.
-
classmethod
from_message
(value)[source]¶ Convert the output from
as_message()
back into aBuildState
.
-
next_file
¶ Returns the filename of the next file that needs transferring or
None
if all files have been transferred.
-
transfers_done
¶ Returns
True
if all files have been transferred.
-
class
piwheels.states.
SlaveState
(address, timeout, native_py_version, native_abi, native_platform, label)[source]¶ Tracks the state of a build slave. The master updates this state which each request and reply sent to and received from the slave, and this class in turn manages the associated
BuildState
(accessible frombuild
) andTransferState
(accessible fromtransfer
). The class also tracks the time a request was last seen from the build slave, and includes akill()
method.
-
class
piwheels.states.
TransferState
(slave_id, file_state)[source]¶ Tracks the state of a file transfer. All file transfers are held in temporary locations until
verify()
indicates the transfer was successful, at which point they are atomically renamed into their final location.The state is intimately tied to the file transfer protocol and includes methods to write a recevied
chunk()
, and to determine the next chunk tofetch()
, as well as a property to determine when the transfer isdone
.
12.22. piwheels.ranges¶
A set of utility routines for efficiently tracking byte ranges within a stream. These are used to track which chunks of a file have been received during file transfers from build slaves.
See FileJuggler
for the usage of these functions.
-
piwheels.ranges.
consolidate
(ranges)[source]¶ Given a list of ranges in ascending order, this generator function returns the list with any overlapping ranges consolidated into individual ranges. For example:
>>> list(consolidate([range(0, 5), range(4, 10)])) [range(0, 10)] >>> list(consolidate([range(0, 5), range(5, 10)])) [range(0, 10)] >>> list(consolidate([range(0, 5), range(6, 10)])) [range(0, 5), range(6, 10)]
-
piwheels.ranges.
exclude
(ranges, ex)[source]¶ Given a list of non-overlapping ranges in ascending order, and a range ex to exclude, this generator function returns ranges with all values covered by ex removed from any contained ranges. For example:
>>> list(exclude([range(10)], range(2))) [range(2, 10)] >>> list(exclude([range(10)], range(2, 4))) [range(0, 2), range(4, 10)]
-
piwheels.ranges.
intersect
(range1, range2)[source]¶ Given two ranges range1 and range2 (which must both have a step of 1), returns the range formed by the intersection of the two ranges, or
None
if the ranges do not overlap. For example:>>> intersect(range(10), range(5)) range(0, 5) >>> intersect(range(10), range(10, 2)) >>> intersect(range(10), range(2, 5)) range(2, 5)
-
piwheels.ranges.
split
(ranges, i)[source]¶ Given a list of non-overlapping ranges in ascending order, this generator function returns the list with the range containing i split into two ranges, one ending at i and the other starting at i. If i is not contained in any of the ranges, then ranges is returned unchanged. For example:
>>> list(split([range(10)], 5)) [range(0, 5), range(5, 10)] >>> list(split([range(10)], 0)) [range(0, 10)] >>> list(split([range(10)], 20)) [range(0, 10)]