8. Module Reference

This chapter contains all the documentation auto-generated from the source code. It is probably not terribly useful for reading through, but may be useful as a searchable reference.

8.1. piwheels.master

Defines the PiWheelsMaster class. An instance of this is the entry-point for the piw-master script.

class piwheels.master.PiWheelsMaster[source]

This is the main class for the piw-master script. It spawns various worker threads, then spends its time communicating with any attached monitor applications (see piw-monitor) and build slaves (see piw-slave).

static configure_parser()[source]

Construct the command line parser for piw-master with its many options (this method only exists to simplify the main method).

do_hello()[source]

Handler for the HELLO message; this indicates a new monitor has been attached and would like all the build slave’s HELLO messages replayed to it.

do_kill(slave_id)[source]

Handler for the KILL message; this terminates the specified build slave by its master id.

do_pause()[source]

Handler for the PAUSE message; this requests all tasks pause their operations.

do_quit()[source]

Handler for the QUIT message; this terminates the master.

do_resume()[source]

Handler for the RESUME message; this requests all tasks resume their operations.

main_loop()[source]

This is the main loop of the piw-master script. It receives messages from the internal status queue and forwards them onto the external status queue (for any piw-monitor scripts that are attached). It also retrieves any messages sent to the control queue and dispatches them to a handler.

piwheels.master.sig_term(signo, stack_frame)[source]

Handler for the SIGTERM signal; raises SystemExit which will cause the PiWheelsMaster.main_loop() method to terminate.

8.2. piwheels.master.tasks

Implements the base classes (Task and its derivative PauseableTask) which form the basis of all the tasks in the piwheels master.

exception piwheels.master.tasks.TaskQuit[source]

Exception raised when the “QUIT” message is received by the internal control queue.

class piwheels.master.tasks.Task(config)[source]

The Task class is a Thread derivative which is the base for all tasks in the piwheels master. The run() method is overridden to perform a simple task loop which calls loop() once a cycle, and poll() to react to any messages arriving into queues. Queues are associated with handlers via the register() method.

handle_control(queue)[source]

Default handler for the internal control queue. In this base implementation it simply handles the “QUIT” message by raising TaskQuit (which the run() method will catch and use as a signal to end).

loop()[source]

This method is called once per loop of the task’s run() method. If the task needs to do some work periodically, this is the place to do it.

pause()[source]

Requests that the task pause itself. This is an idempotent method; it’s always safe to call repeatedly and even if the task isn’t pauseable it’ll simply be ignored.

poll(timeout=1000)[source]

This method is called once per loop of the task’s run() method. It polls all registered queues and calls their associated handlers if the poll is successful.

quit()[source]

Requests that the task terminate at its earliest convenience. To wait until the task has actually closed, call join() afterwards.

register(queue, handler, flags=<Mock object>)[source]

Register queue to be polled on each cycle of the task. Any messages with the relevant flags (defaults to POLLIN) will trigger the specified handler method which is expected to take a single argument which will be queue.

Parameters:
  • queue (zmq.Socket) – The queue to poll.
  • handler – The function or method to call when a message with matching flags arrives in queue.
  • flags (int) – The flags to match in the queue poller (defaults to POLLIN).
resume()[source]

Requests that the task resume itself. This is an idempotent method; it’s safe to call repeatedly and even if the task isn’t pauseable it’ll simply be ignored.

run()[source]

This method is the main task loop. Override this to perform one-off startup processing within the task’s background thread, and to perform any finalization required.

class piwheels.master.tasks.PauseableTask(config)[source]

Derivative of Task that implements a rudimentary pausing mechanism. When the “PAUSE” message is received on the internal control queue, the task will enter a loop which simply polls the control queue waiting for “RESUME” or “QUIT”. No other work will be done (Task.loop() and Task.poll() will not be called) until the task is resumed (or terminated).

8.3. piwheels.master.states

This module defines several classes which permit interested tasks to track the state of build slaves (SlaveState), file transfers (TransferState), build attempts (BuildState) and build artifacts (FileState).

class piwheels.master.states.FileState(filename, filesize, filehash, package_tag, package_version_tag, py_version_tag, abi_tag, platform_tag, transferred=False)[source]

Represents the state of an individual build artifact (a package file, or wheel) including its filename, filesize, the SHA256 filehash, and various tags extracted from the build. Also tracks whether or not the file has been transferred.

Parameters:
  • filename (str) – The original filename of the build artifact.
  • filesize (int) – The size of the file in bytes.
  • filehash (str) – The SHA256 hash of the file contents.
  • package_tag (str) – The package tag extracted from the filename (first “-” separated component).
  • package_version_tag (str) – The package version tag extracted from the filename (second “-” separated component).
  • py_version_tag (str) – The python version tag extracted from the filename (third from last “-” separated component).
  • abi_tag (str) – The python ABI tag extracted from the filename (second from last “-” separated component).
  • platform_tag (str) – The platform tag extracted from the filename (last “-” separated component).
  • transferred (bool) – True if the file has been transferred from the build slave that generated it to the file server.
verified()[source]

Called to set transferred to True after a file transfer has been successfully verified.

class piwheels.master.states.BuildState(slave_id, package, version, abi_tag, status, duration, output, files, build_id=None)[source]

Represents the state of a package build including the package, version, status, build duration, and all the lines of output. The files attribute is a mapping containing details of each successfully built package file.

Parameters:
  • slave_id (int) – The master’s identifier for the build slave.
  • package (str) – The name of the package to build.
  • version (str) – The version number of the package to build.
  • abi_tag (str) – The ABI for which the build was attempted (must not be 'none').
  • status (bool) – True if the build succeeded, False if it failed.
  • duration (datetime.timedelta) – The amount of time it took to complete the build.
  • output (str) – The log output of the build.
  • files (dict) – A mapping of filenames to FileState objects for each artifact produced by the build.
  • build_id (int) – The integer identifier generated for the build by the database (None until the build has been inserted into the database).
classmethod from_db(db, build_id)[source]

Construct an instance by querying the database for the specified build_id.

Parameters:
  • db (Database) – A Database instance to query.
  • build_id (int) – The integer identifier of an attempted build.
logged(build_id)[source]

Called to fill in the build’s ID in the backend database.

files

A mapping of filename to FileState instances.

next_file

Returns the filename of the next file that needs transferring or None if all files have been transferred.

transfers_done

Returns True if all files have been transferred.

class piwheels.master.states.SlaveState(address, timeout, native_py_version, native_abi, native_platform)[source]

Tracks the state of a build slave. The master updates this state which each request and reply sent to and received from the slave, and this class in turn manages the associated BuildState (accessible from build) and TransferState (accessible from transfer). The class also tracks the time a request was last seen from the build slave, and includes a kill() method.

class piwheels.master.states.TransferState(slave_id, file_state)[source]

Tracks the state of a file transfer. All file transfers are held in temporary locations until verify() indicates the transfer was successful, at which point they are atomically renamed into their final location.

The state is intimately tied to the file transfer protocol and includes methods to write a recevied chunk(), and to determine the next chunk to fetch(), as well as a property to determine when the transfer is done.

8.4. piwheels.master.ranges

A set of utility routines for efficiently tracking byte ranges within a stream. These are used to track which chunks of a file have been received during file transfers from build slaves.

See FileJuggler for the usage of these functions.

piwheels.master.ranges.consolidate(ranges)[source]

Given a list of ranges in ascending order, this generator function returns the list with any overlapping ranges consolidated into individual ranges. For example:

>>> list(consolidate([range(0, 5), range(4, 10)]))
[range(0, 10)]
>>> list(consolidate([range(0, 5), range(5, 10)]))
[range(0, 10)]
>>> list(consolidate([range(0, 5), range(6, 10)]))
[range(0, 5), range(6, 10)]
piwheels.master.ranges.exclude(ranges, ex)[source]

Given a list of non-overlapping ranges in ascending order, and a range ex to exclude, this generator function returns ranges with all values covered by ex removed from any contained ranges. For example:

>>> list(exclude([range(10)], range(2)))
[range(2, 10)]
>>> list(exclude([range(10)], range(2, 4)))
[range(0, 2), range(4, 10)]
piwheels.master.ranges.intersect(range1, range2)[source]

Returns two ranges range1 and range2 (which must both have a step of 1), returns the range formed by the intersection of the two ranges, or None if the ranges do not overlap. For example:

>>> intersect(range(10), range(5))
range(0, 5)
>>> intersect(range(10), range(10, 2))
>>> intersect(range(10), range(2, 5))
range(2, 5)
piwheels.master.ranges.split(ranges, i)[source]

Given a list of non-overlapping ranges in ascending order, this generator function returns the list with the range containing i split into two ranges, one ending at i and the other starting at i. If i is not contained in any of the ranges, then ranges is returned unchanged. For example:

>>> list(split([range(10)], 5))
[range(0, 5), range(5, 10)]
>>> list(split([range(10)], 0))
[range(0, 10)]
>>> list(split([range(10)], 20))
[range(0, 10)]

8.5. piwheels.master.db

This module defines the low level database API, Database. This is a simple core SQLAlchemy affair which runs trivial queries against the PostgreSQL database. All the serious logic is defined within views in the database itself.

class piwheels.master.db.Database(dsn)[source]

PiWheels database connection class

add_new_package(package)[source]

Insert a new package record into the database. Key violations are ignored as packages is effectively an append-only table.

add_new_package_version(package, version)[source]

Insert a new package version record into the database. Key violations are ignored as versions is effectively an append-only table.

delete_build(package, version)[source]

Remove all builds for the specified package and version, along with all files and download records

get_all_package_versions()[source]

Returns the set of all known (package, version) tuples

get_all_packages()[source]

Returns the set of all known package names

get_build(build_id)[source]

Return all details about a given build.

get_build_abis()[source]

Return the set of ABIs that the master should attempt to build

get_build_queue()[source]

Returns a generator covering the entire builds_pending view; streaming results are activated for this query as it’s more important to get the first result quickly than it is to retrieve the entire set.

get_files(build_id)[source]

Return all details about the files generated by a given build.

get_package_files(package)[source]

Returns all details required to build the index.html for the specified package.

get_pypi_serial()[source]

Return the serial number of the last PyPI event

get_statistics()[source]

Return various build related statistics from the database (see the definition of the statistics view in the database creation script for more information.

get_version_files(package, version)[source]

Returns the names of all files for version of package

log_build(build)[source]

Log a build attempt in the database, including build output and wheel info if successful

log_download(download)[source]

Log a download in the database, including data derived from JSON in pip’s user-agent.

log_file(build, file)[source]

Log a pending file transfer in the database, including file-size, hash, and various tags

set_pypi_serial(serial)[source]

Update the serial number of the last PyPI event

skip_package(package)[source]

Mark a package to prevent future builds of all versions (and all future versions).

skip_package_version(package, version)[source]

Mark a version of a package to prevent future build attempts.

test_package_version(package, version)[source]

Check whether version of package already exists in the database. Returns a boolean.

8.6. piwheels.master.cloud_gazer

Defines the CloudGazer task; see class for more details.

class piwheels.master.cloud_gazer.CloudGazer(config)[source]

This task scrapes PyPI for the list of available packages, and the versions of those packages. This information is written into the backend database for TheArchitect to use.

8.7. piwheels.master.the_oracle

Defines TheOracle task and the DbClient RPC class for talking to it.

class piwheels.master.the_oracle.TheOracle(config)[source]

This task provides an RPC-like interface to the database; it handles requests such as registering a new package, version, or build, and answering queries about the hashes of files. The primary clients of this class are SlaveDriver, IndexScribe, and CloudGazer.

Note that because database requests are notoriously variable in length the client RPC class (DbClient) doesn’t directly talk to TheOracle. Rather, multiple instances of TheOracle are spawned and Seraph sits in front of these acting as a simple load-sharing router for the RPC clients.

do_allpkgs()[source]

Handler for “ALLPKGS” message, sent by DbClient to request the set of all packages define known to the database.

do_allvers()[source]

Handler for “ALLVERS” message, sent by DbClient to request the set of all (package, version) tuples known to the database.

do_delbuild(package, version)[source]

Handler for “DELBUILD” message, sent by DbClient to remove all builds (and files and downloads by cascade) for version of package.

do_getabis()[source]

Handler for “GETABIS” message, sent by DbClient to request the list of all ABIs to build for.

do_getpypi()[source]

Handler for “GETPYPI” message, sent by DbClient to request the record of the last serial number from the PyPI changelog.

do_getstats()[source]

Handler for “GETSTATS” message, sent by DbClient to request the latest database statistics, returned as a list of (field, value) tuples.

do_logbuild(build)[source]

Handler for “LOGBUILD” message, sent by DbClient to register a new build result.

do_logdownload(download)[source]

Handler for “LOGDOWNLOAD” message, sent by DbClient to register a new download.

do_newpkg(package)[source]

Handler for “NEWPKG” message, sent by DbClient to register a new package.

do_newver(package, version)[source]

Handler for “NEWVER” message, sent by DbClient to register a new (package, version) tuple.

do_pkgexists(package, version)[source]

Handler for “PKGEXISTS” message, sent by DbClient to request whether or not the specified version of package exists.

do_pkgfiles(package)[source]

Handler for “PKGFILES” message, sent by DbClient to request details of all wheels assocated with package.

do_setpypi(serial)[source]

Handler for “SETPYPI” message, sent by DbClient to update the last seen serial number from the PyPI changelog.

do_skippkg(package)[source]

Handler for “SKIPPKG” message, sent by DbClient to skip building all versions of a package.

do_skipver(package, version)[source]

Handler for “SKIPVER” message, sent by DbClient to skip building a specific version of a package.

do_verfiles(package, version)[source]

Handler for “VERFILES” message, sent by DbClient to request the filenames of all wheels associated with version of package.

handle_db_request(queue)[source]

Handle incoming requests from DbClient instances.

class piwheels.master.the_oracle.DbClient(config)[source]

RPC client class for talking to TheOracle.

add_new_package(package)[source]

See db.Database.add_new_package().

add_new_package_version(package, version)[source]

See db.Database.add_new_package_version().

delete_build(package, version)[source]

See db.Database.delete_build().

get_all_package_versions()[source]

See db.Database.get_all_package_versions().

get_all_packages()[source]

See db.Database.get_all_packages().

get_build_abis()[source]

See db.Database.get_build_abis().

get_package_files(package)[source]

See db.Database.get_package_files().

get_pypi_serial()[source]

See db.Database.get_pypi_serial().

get_statistics()[source]

See db.Database.get_statistics().

get_version_files(package, version)[source]

See db.Database.get_version_files().

log_build(build)[source]

See db.Database.log_build().

log_download(download)[source]

See db.Database.log_download().

set_pypi_serial(serial)[source]

See db.Database.set_pypi_serial().

skip_package(package)[source]

See db.Database.skip_package().

skip_package_version(package, version)[source]

See db.Database.skip_package_version().

test_package_version(package, version)[source]

See db.Database.test_package_version().

8.8. piwheels.master.seraph

Defines the Seraph task; see class for more details.

class piwheels.master.seraph.Seraph(config)[source]

This task is a simple load-sharing router for TheOracle tasks.

handle_back(queue)[source]

Receive a response from an instance of TheOracle on the back queue. Strip off the worker’s address frame and add it back to the available queue then send the response back to the client that made the original request.

handle_front(queue)[source]

If any workers are currently available, receive DbClient requests from the front queue and send it on to the worker including the client’s address frame.

8.9. piwheels.master.the_architect

Defines TheArchitect task; see class for more details.

class piwheels.master.the_architect.TheArchitect(config)[source]

This task queries the backend database to determine which versions of packages have yet to be built (and aren’t marked to be skipped). It places a tuple of (package, version) for each such build into the internal “builds” queue for SlaveDriver to read.

handle_builds(queue)[source]

Handler for the task’s builds queue. Whenever a build slave asks SlaveDriver for a new task, SlaveDriver passes the slave’s ABI to TheArchitect via this queue. We simply pop the first entry (if any) off the relevant queue and send it back.

loop()[source]

The architect simply runs the build queue query repeatedly. On each loop iteration, an entry from the result set is added to the relevant ABI queue. The queues are limited in length to prevent silly memory usage on the initial run (which will involve millions of entries). This does mean that a single loop over the query will potentially miss entries, but that’s fine as it’ll just be repeated again.

8.10. piwheels.master.slave_driver

Defines the SlaveDriver task; see class for more details.

class piwheels.master.slave_driver.SlaveDriver(config)[source]

This task handles interaction with the build slaves using the slave protocol. Interaction is driven by the slaves (i.e. the master doesn’t push jobs, rather the slaves request a job and the master replies with the next (package, version) tuple from the internal “builds” queue).

The task also incidentally interacts with several other queues: the internal “status” queue is sent details of every reply sent to a build slave (the main_loop() method passes this information on to any listening monitors). Also, the internal “indexes” queue is informed of any packages that need web page indexes re-building (as a result of a successful build).

active_builds()[source]

Generator method which yields all (package, version) tuples currently being built by build slaves.

do_built(slave)[source]

Handler for the build slave’s “BUILT” message, which is sent after an attempted package build succeeds or fails. The handler logs the result in the database and, if files have been generated by the build, informs the FileJuggler task to expect a file transfer before sending “SEND” back to the build slave with the required filename.

If no files were generated (e.g. in the case of a failed build, or a degenerate success), “DONE” is returned indicating that the build slave is free to discard all resources generated during the build and return to its idle state.

do_bye(slave)[source]

Handler for the build slave’s final “BYE” message upon shutdown. This removes the associated state from the internal slaves dict.

Parameters:slave (SlaveState) – The object representing the current status of the build slave.
do_hello(slave)[source]

Handler for the build slave’s initial “HELLO” message. This associates the specified slave state with the slave’s address and returns “HELLO” with the master’s id for the slave (the id communicated back simply for consistency of logging; administrators can correlate master log messages with slave log messages when both have the same id number; we can’t use IP address for this as multiple slaves can run on one machine).

Parameters:slave (SlaveState) – The object representing the current status of the build slave.
do_idle(slave)[source]

Handler for the build slave’s “IDLE” message (which is effectively the slave requesting work). If the master wants to terminate the slave, it sends back “BYE”. If the build queue (for the slave’s ABI) is empty or the task is currently paused, “SLEEP” is returned indicating the slave should wait a while and then try again.

If a job can be retrieved from the (ABI specific) build queue, then a “BUILD” message is sent back with the required package and version.

Parameters:slave (SlaveState) – The object representing the current status of the build slave.
do_sent(slave)[source]

Handler for the build slave’s “SENT” message indicating that it’s finished sending the requested file to FileJuggler. The FsClient RPC mechanism is used to ask FileJuggler to verify the transfer against the stored hash and, if this is successful, a message is sent to IndexScribe to regenerate the package’s index.

If further files remain to be transferred, another “SEND” message is returned to the build slave. Otherwise, “DONE” is sent to free all build resources.

If a transfer fails to verify, another “SEND” message with the same filename is returned to the build slave.

handle_control(queue)[source]

Handle incoming requests to the internal control queue.

Whilst the SlaveDriver task is “pauseable”, it can’t simply stop responding to requests from build slaves. Instead, its pause is implemented as an internal flag. While paused it simply tells build slaves requesting a new job that none are currently available but otherwise continues servicing requests.

It also understands a couple of extra control messages unique to it, specifically “KILL” to tell a build slave to terminate, and “HELLO” to cause all “HELLO” messages from build slaves to be replayed (for the benefit of a newly attached monitor process).

handle_slave(queue)[source]

Handle requests from build slaves.

See the piw-slave chapter for an overview of the protocol for messages between build slaves and SlaveDriver. This method retrieves the message from the build slave, finds the associated SlaveState and updates it with the message, then calls the appropriate message handler. The handler will be expected to return a reply (in the usual form of a list of strings) or None if no reply should be sent (e.g. for a final “BYE” message).

kill_slave(slave_id)[source]

Additional task control method to trigger a “KILL” message to the internal control queue. See quit() for more information.

list_slaves()[source]

Additional task control method to trigger a “HELLO” message to the internal control queue. See quit() for more information.

8.11. piwheels.master.mr_chase

Defines the MrChase task; see class for more details.

class piwheels.master.mr_chase.MrChase(config)[source]

This task handles smuggling packages into the database manually. It is the task that the piw-import script talks to in order to import packages.

Internally, the task is essentially an abbreviated SlaveDriver (in as much as it has to perform similar database and file-system interactions) but without having to handle talking to lots of build slaves.

do_import(state)[source]

Handler for the importer’s initial “IMPORT” message. This method checks the information in the state passes some simple tests, then ensures that the requested package and version exist in the database (creating them if necessary).

do_remove(state)[source]

Handler for the importer’s “REMOVE” message, indicating a request to remove a specific version of a package from the system.

do_sent(state)[source]

Handler for the importer’s “SENT” message indicating that it’s finished sending the requested file to FileJuggler. The file is verified (as in SlaveDriver) and, if this is successful, a mesasge is sent to IndexScribe to regenerate the package’s index.

If further files remain to be transferred, another “SEND” message is returned to the build slave. Otherwise, “DONE” is sent to free all build resources.

If a transfer fails to verify, another “SEND” message with the same filename is returned to the build slave.

handle_import(queue)[source]

Handle requests from piw-import instances.

See the piw-import and piw-remove chapters for an overview of the protocol for messages between the importer and MrChase.

8.12. piwheels.master.file_juggler

Defines the FileJuggler task and the FsClient RPC class for interacting with it.

exception piwheels.master.file_juggler.TransferError[source]

Base class for errors raised during a file transfer.

exception piwheels.master.file_juggler.TransferIgnoreChunk[source]

Exception raised when a build slave sends CHUNK instead of HELLO as the first message (see FileJuggler.new_transfer()).

exception piwheels.master.file_juggler.TransferDone[source]

Exception raised when a transfer is complete. It may seem a little odd to use an exception for this, but it is “exceptional” behaviour to terminate the file transfer.

class piwheels.master.file_juggler.FileJuggler(config)[source]

This task handles file transfers from the build slaves. The specifics of the file transfer protocol are best understood from the implementation of the FileState class.

However, to detail how a file transfer begins: when a build slave has successfully completed a build it informs the master via the SlaveDriver task. That task replies with a “SEND” instruction to the slave (including a filename). The slave then initiates the transfer with a “HELLO” message to this task. Once transfers are complete the slave sends a “SENT” message to the SlaveDriver task which verifies the transfer and either retries it (when verification fails) or sends back “DONE” indicating the slave can wipe the source file.

current_transfer(transfer, msg, *args)[source]

Called for messages associated with an existing file transfer.

Usually this is “CHUNK” indicating another chunk of data. Rarely, it can be “HELLO” if the master has fallen silent and dropped tons of packets.

Parameters:
  • transfer (TransferState) – The object representing the state of the transfer.
  • msg (str) – The message sent during the transfer.
  • *args – All additional arguments; for “CHUNK” the first must be the file offset and the second the data to write to that offset.
do_expect(slave_id, file_state)[source]

Message sent by FsClient to inform file juggler that a build slave is about to start a file transfer. The message includes the full FileState. The state is stored in the pending map.

Parameters:
  • slave_id (int) – The identity of the build slave about to begin the transfer.
  • file_state (FileState) – The details of the file to be transferred including the expected hash.
do_remove(package, filename)[source]

Message sent by FsClient to request that filename in package should be removed.

do_statvfs()[source]

Message sent by FsClient to request that file juggler return stats on the output file-system.

do_verify(slave_id, package)[source]

Message sent by FsClient to request that juggler verify a file transfer against the expected hash and, if it matches, rename the file into its final location.

Parameters:
  • slave_id (int) – The identity of the build slave that sent the file.
  • package (str) – The name of the package that the file is to be committed to, if valid.
handle_file(queue)[source]

Handle incoming file-transfer messages from build slaves.

The file transfer protocol is in some ways very simple (see the chart in the piw-slave chapter for an overview of the message sequence) and in some ways rather complex (read the ZeroMQ guide chapter on file transfers for more detail on why multiple messages must be allowed in flight simultaneously).

The “normal” state for a file transfer is to be requesting and receiving chunks. Anything else, including redundant re-sends, and transfer completion is handled as an exceptional case.

handle_fs_request(queue)[source]

Handle incoming messages from FsClient instances.

new_transfer(msg, *args)[source]

Called for messages initiating a new file transfer.

The first message must be HELLO along with the id of the slave starting the transfer. The metadata for the transfer will be looked up in the pending list (which is written to by do_expect()).

Parameters:
  • msg (str) – The message sent to start the transfer (must be “HELLO”)
  • *args – All additional arguments (expected to be an integer slave id).
class piwheels.master.file_juggler.FsClient(config)[source]

RPC client class for talking to FileJuggler.

expect(slave_id, file_state)[source]

See FileJuggler.do_expect().

remove(package, filename)[source]

See FileJuggler.do_remove().

statvfs()[source]

See FileJuggler.do_statvfs().

verify(slave_id, package)[source]

See FileJuggler.do_verify().

8.13. piwheels.master.big_brother

Defines the BigBrother task; see class for more details.

class piwheels.master.big_brother.BigBrother(config)[source]

This task periodically queries the database and output file-system for various statistics like the number of packages known to the system, the number built, the number of packages built in the last hour, the remaining file-system space, etc. These statistics are written to the internal “status” queue which main_loop() uses to pass statistics to any listening monitors.

handle_index(queue)[source]

Handler for the index_queue. Whenever a slot becomes available, and an updated status_info1 package is available, send a message to update the home page.

handle_status(queue)[source]

Handler for the internal status queue. Whenever a slot becomes available, and an updated status_info2 package is available, send a message with the latest status (ultimately this winds up going to any attached monitors via the external status queue).

8.14. piwheels.master.index_scribe

Defines the IndexScribe task; see class for more details.

class piwheels.master.index_scribe.IndexScribe(config)[source]

This task is responsible for writing web-page index.html files. It reads the names of packages off the internal “indexes” queue and rebuilds the index.html for that package and, optionally, the overall index.html if the package is one that wasn’t previously present.

Note

It is important to note that package names are never pushed into the internal “indexes” queue until all file-transfers associated with the build are complete. Furthermore, while the entire index for a package is re-built, hashes are never re-calculated from the disk files (they are always read from the database).

handle_index(queue)[source]

Handle incoming requests to (re)build index files. These will be in the form of “HOME”, a request to write the homepage with some associated statistics, or “PKG”, a request to write the index for the specified package.

Note

In all handlers below, care is taken to ensure clients never see a partially written file and that temporary files are cleaned up in the event of any exceptions.

setup_output_path()[source]

Called on task startup to copy all static resources into the output path (and to make sure the output path exists as a directory).

write_homepage(status_info)[source]

Re-writes the site homepage using the provided statistics in the homepage template (which is effectively a simple Python format string).

Parameters:status_info (tuple) – A namedtuple containing statistics obtained by BigBrother.
write_package_index(package, files)[source]

(Re)writes the index of the specified package. The file meta-data (including the hash) is retrieved from the database, never from the file-system.

Parameters:
  • package (str) – The name of the package to write the index for
  • files (list) – A list of (filename, filehash) tuples.
write_root_index()[source]

(Re)writes the index of all packages. This is implicitly called when a request to write a package index is received for a package not present in the task’s cache.

8.15. piwheels.slave

Defines the PiWheelsSlave class. An instance of this is the entry-point for the piw-slave script.

class piwheels.slave.PiWheelsSlave[source]

This is the main class for the piw-slave script. It connects (over zmq sockets) to a master (see piw-master) then loops around the slave protocol (see the piw-slave chapter). It retrieves source packages directly from PyPI, attempts to build a wheel in a sandbox directory and, if successful, transmits the results to the master.

do_build(package, version)[source]

Alternatively, in response to “IDLE”, the master may send “BUILD” package version. We should then attempt to build the specified wheel and send back a “BUILT” message with a full report of the outcome.

do_bye()[source]

The master may respond with “BYE” at any time indicating we should immediately terminate (first cleaning up any extant build). We return None to tell the main loop to quit.

do_done()[source]

After all files have been sent (and successfully verified), the master will reply with “DONE” indicating we can remove all associated build artifacts. We respond with “IDLE”.

do_hello(new_id, pypi_url)[source]

In response to our initial “HELLO” (detailing our various PEP 425 tags), the master is expected to send “HELLO” back with an integer identifier and the URL of the PyPI repository to download from. We use the identifier in all future log messages for the ease of the administrator.

We reply with “IDLE” to indicate we’re ready to accept a build job.

do_send(filename)[source]

If a build succeeds and generates files (detailed in a “BUILT” message), the master will reply with “SEND” filename indicating we should transfer the specified file (this is done on a separate socket with a different protocol; see builder.PiWheelsPackage.transfer() for more details). Once the transfers concludes, reply to the master with “SENT”.

do_sleep()[source]

If, in response to an “IDLE” message we receive “SLEEP” this indicates the master has nothing for us to do currently. Sleep for a little while then try “IDLE” again.

handle_reply(reply, *args)[source]

Dispatch a message from the master to an appropriate handler method.

piwheels.slave.duration(s)[source]

Convert s, a string representing a duration, into a datetime.timedelta.

8.16. piwheels.slave.builder

Defines the classes which use pip to build wheels.

class piwheels.slave.builder.PiWheelsPackage(path)[source]

Records the state of a build artifact, i.e. a wheel package. The filename is deconstructed into the fields specified by PEP 425.

Parameters:path (pathlib.Path) – The path to the wheel on the local filesystem.
open(mode='rb')[source]

Open the wheel in binary mode and return the open file object.

transfer(queue, slave_id)[source]

Transfer the wheel via the specified queue. This is the client side implementation of the file_juggler.FileJuggler protocol.

abi_tag

Return the ABI part of the wheel’s filename (the penultimate “-” separated element).

build_tag

Return the optional build part of the wheel’s filename (the third “-” separated element when 6 elements exist in total).

filehash

Return an SHA256 digest of the wheel’s contents.

filename

Return the filename of the wheel as a simple string (with no path components).

filesize

Return the size of the wheel in bytes.

metadata

Return the contents of the metadata.json file inside the wheel.

package_tag

Return the package part of the wheel’s filename (the first “-” separated element).

package_version_tag

Return the version part of the wheel’s filename (the second “-” separated element).

platform_tag

Return the platform part of the wheel’s filename (the last “-” separated element).

py_version_tag

Return the python version part of the wheel’s filename (third from last “-” separated element).

class piwheels.slave.builder.PiWheelsBuilder(package, version)[source]

Class responsible for building wheels for a given version of a package.

Parameters:
  • package (str) – The name of the package to attempt to build wheels for.
  • version (str) – The version of the package to attempt to build.
build(timeout=None, pypi_index='https://pypi.python.org/simple')[source]

Attempt to build the package within the specified timeout.

Parameters:
  • timeout (float) – The number of seconds to wait for pip to finish before raising subprocess.TimeoutExpired.
  • pypi_index (str) – The URL of the PEP 503 compliant repository from which to fetch packages for building.
clean()[source]

Remove the temporary build directory and all its contents.

as_message

Return the state as a list suitable for use in several protocol messages (specifically those used by piw-slave and piw-import).

8.17. piwheels.initdb

Contains the functions that make up the piw-initdb script.

piwheels.initdb.main(args=None)[source]

This is the main function for the piw-initdb script. It creates the piwheels database required by the master or, if it already exists, upgrades it to the current version of the application.

piwheels.initdb.detect_users(conn, test_user)[source]

Test that the user for conn is a cluster superuser (so we can drop and create anything we want in the database), and that test_user (which will be granted limited rights to various objects for the purposes of the piw-master script) exists and is not a cluster superuser.

piwheels.initdb.detect_version(conn)[source]

Detect the version of the database. This is typically done by reading the contents of the configuration table, but before that was added we can guess a couple of versions based on what tables exist (or don’t). Returns None if the database appears uninitialized, and raises RuntimeError is the version is so ancient we can’t do anything with it.

piwheels.initdb.get_connection(dsn)[source]

Return an SQLAlchemy connection to the specified dsn or raise RuntimeError if the database doesn’t exist (the administrator is expected to create the database before running this script).

piwheels.initdb.get_script(version)[source]

Generate the script to get the database from version (the result of detect_version()) to the current version of the software. If version is None, this is simply the contents of the sql/create_piwheels.sql script. Otherwise, it is a concatenation of various update scripts.

piwheels.initdb.parse_statements(script)[source]

This is an extremely crude statement splitter for PostgreSQL’s dialect of SQL. It understands --comments, "quoted identifiers", 'string literals' and $delim$ extended strings $delim$, but not E'\escaped strings' or /* C-style comments */. If you start using such things in the update scripts, you’ll need to extend this function to accommodate them.

It returns a generator which yields individiual statements from script, delimited by semi-colon terminators.

8.18. piwheels.importer

Contains the functions that implement the piw-import script.

piwheels.importer.main(args=None)[source]

This is the main function for the piw-import script. It uses some bits of the piw-slave script to deconstruct the filenames passed to it in order to build all the requried information that MrChase needs.

piwheels.importer.print_builder(config, builder)[source]

Dumps a human-readable description of the builder to the log / console.

Parameters:
  • config – The configuration generated from the command line argument parser.
  • builder (PiWheelsBuilder) – The builder to print the description of.
piwheels.importer.abi(config, builder, default=None)[source]

Calculate the ABI from the given config and the first file contained by the builder state. If the configuration contains no ABI override, and the ABI of the first file is ‘none’, return default.

piwheels.importer.do_import(config, builder)[source]

Handles constructing and sending the initial “IMPORT” message to master.mr_chase.MrChase. If “SEND” is then received, uses do_send() to handle transmitting files.

Parameters:
  • config – The configuration obtained from parsing the command line.
  • builder (PiWheelsBuilder) – The object representing the state of the build.
piwheels.importer.do_send(builder, filename)[source]

Handles sending files when requested by do_import().

8.19. piwheels.remove

Contains the functions that implement the piw-remove script.

piwheels.remove.main(args=None)[source]

This is the main function for the piw-remove script. It uses MrChase to remove built packages from the system.

piwheels.remove.do_remove(config)[source]

Handles constructing and sending the “REMOVE” message to master.mr_chase.MrChase.

Parameters:config – The configuration obtained from parsing the command line.