Overview

Start here for all things distlib.

Distlib evolved out of packaging

Distlib is a library which implements low-level functions that relate to packaging and distribution of Python software. It consists in part of the functions in the packaging Python package, which was intended to be released as part of Python 3.3, but was removed shortly before Python 3.3 entered beta testing.

What was the problem with packaging?

The packaging software just wasn’t ready for inclusion in the Python standard library. The amount of work needed to get it into the desired state was too great, given the number of people able to work on the project, the time they could devote to it, and the Python 3.3 release schedule.

The approach taken by packaging was seen to be a good one: to ensure interoperability and consistency between different tools in the packaging space by defining standards for data formats through PEPs, and to do away with the ad hoc nature of installation encouraged by the distutils approach of using executable Python code in setup.py. Where custom code was needed, it could be provided in a standardised way using installation hooks.

While some very good work was done in defining PEPs to codify some of the best practices, packaging suffered from some drawbacks, too:

  • Not all the PEPs may have been functionally complete, because some important use cases were not considered – for example, built (binary) distributions for Windows.

  • It continued the command-based design of distutils, which had resulted in distutils being difficult to extend in a consistent, easily understood, and maintainable fashion.

  • Some important features required by distribution authors were not considered – for example:

    • Access to data files stored in Python packages.
    • Support for plug-in extension points.
    • Support for native script execution on Windows.

    These features are supported by third-party tools (like setuptools / Distribute) using pkg_resources, entry points and console scripts.

  • There were a lot of rough edges in the packaging implementation, both in terms of bugs and in terms of incompletely implemented features. This can be seen (with the benefit of hindsight) as due to the goals being set too ambitiously; the project developers bit off more than they could chew.

How Distlib can help

The idea behind Distlib is expressed in this python-dev mailing-list post, though a different name was suggested for the library. Basically, Distlib contains the implementations of the packaging PEPs and other low-level features which relate to packaging, distribution, and deployment of Python software. If Distlib can be made genuinely useful, then it is possible for third-party packaging tools to transition to using it. Their developers and users then benefit from standardised implementation of low-level functions, time saved by not having to reinvent wheels, and improved interoperability between tools.

How you can help

If you have some time and the inclination to improve the state of Python packaging, then you can help by trying out Distlib, raising issues where you find problems, contributing feedback and/or patches to the implementation, documentation, and underlying PEPs.

Main features

Distlib currently offers the following features:

  • The package distlib.database, which implements a database of installed distributions, as defined by PEP 376, and distribution dependency graph logic. Support is also provided for non-installed distributions (i.e. distributions registered with metadata on an index like PyPI), including the ability to scan for dependencies and building dependency graphs.
  • The package distlib.index, which implements an interface to perform operations on an index, such as registering a project, uploading a distribution or uploading documentation. Support is included for verifying SSL connections (with domain matching) and signing/verifying packages using GnuPG.
  • The package distlib.metadata, which implements distribution metadata as defined by PEP 426, PEP 345, PEP 314 and PEP 241.
  • The package distlib.markers, which implements environment markers as defined by PEP 426.
  • The package distlib.manifest, which implements lists of files used in packaging source distributions.
  • The package distlib.locators, which allows finding distributions, whether on PyPI (XML-RPC or via the “simple” interface), local directories or some other source.
  • The package distlib.resources, which allows access to data files stored in Python packages, both in the file system and in .zip files.
  • The package distlib.scripts, which allows installing of scripts with adjustment of shebang lines and support for native Windows executable launchers.
  • The package distlib.version, which implements version specifiers as defined by PEP 440 / PEP 426, but also support for working with “legacy” versions (setuptools/distribute) and semantic versions.
  • The package distlib.wheel, which provides support for building and installing from the Wheel format for binary distributions (see PEP 427).
  • The package distlib.util, which contains miscellaneous functions and classes which are useful in packaging, but which do not fit neatly into one of the other packages in distlib.* The package implements enhanced globbing functionality such as the ability to use ** in patterns to specify recursing into subdirectories.

Python version and platform compatibility

Distlib is intended to be used on any Python version >= 2.6 and is tested on Python versions 2.6, 2.7, 3.1, 3.2, and 3.3 on Linux, Windows, and Mac OS X (not all versions are tested on all platforms, but are expected to work correctly).

Project status

The project has reached alpha status in its development: there is a test suite and it has been exercised on Windows, Ubuntu and Mac OS X. To work with the project, you can download a release from PyPI, or clone the source repository or download a tarball from it.

Coverage results are available at:

http://www.red-dove.com/distlib/coverage/

Continuous integration test results are available at:

https://travis-ci.org/vsajip/distlib/

The source repository for the project is on BitBucket:

https://bitbucket.org/pypa/distlib/

You can leave feedback by raising a new issue on the issue tracker (BitBucket registration not necessary, but recommended).

Change log for distlib

0.2.5 (future)

Released: not yet.

0.2.4

Released: 2016-09-30

  • compat

    • Updated to not fail on import if SSL is unavailable.
  • index

    • Switch from using gpg in preference to gpg2 for signing. This is to avoid gpg2’s behaviour of prompting for passwords, which interferes with the tests on some machines.
  • locators

    • Changed project name comparisons to follow PEP 503. Thanks to Steven Arcangeli for the patch.
    • Added errors queue to Locator.
  • manifest

    • Changed match logic to work under Python 3.6, due to differences in how fnmatch.translate behaves.
  • resources

    • Updated finder registry logic to reflect changes in Python 3.6.
  • scripts

    • Fixed regular expression in generated script boilerplate.
  • util

    • Updated to not fail on import if SSL is unavailable.
    • Added normalize_name fpr project name comparisons using PEP 503.
  • tests

    • Updated to skip certain tests if SSL is unavailable.
    • Numerous other test refinements, not detailed further here.

0.2.3

Released: 2016-04-30

  • util

    • Changed get_executable to return Unicode rather than bytes.
    • Fixed #84: Allow + character in output script names.
    • Relaxed too-stringent test looking for application/json in headers.
  • wheel

    • sorted the entries in RECORD before writing to file.
  • tests

    • Numerous test refinements, not detailed further here.

0.2.2

Released: 2016-01-30

  • database

    • Issue #81: Added support for detecting distributions installed by wheel versions >= 0.23 (which use metadata.json rather than pydist.json). Thanks to Te-jé Rodgers for the patch.
  • locators

  • metadata

    • Updated to use different formatting for description field for V1.1 metadata.
    • Corrected “classifier” to “classifiers” in the mapping for V1.0 metadata.
  • scripts

    • Improved support for Jython when quoting executables in output scripts.
  • util

    • Issue #77: Made the internal URL used for extended metadata fetches configurable via a module attribute.
    • Issue #78: Improved entry point parsing to handle leading spaces in ini-format files.
  • docs

    • Numerous documentation updates, not detailed further here.
  • tests

    • renamed environment variable SKIP_SLOW to SKIP_ONLINE in tests and applied to some more tests.
    • Numerous other test refinements, not detailed further here.

0.2.1

Released: 2015-07-07

  • locators

    • Issue #58: Return a Distribution instance or None from locate().
    • Issue #59: Skipped special keys when looking for versions.
    • Improved behaviour of PyPIJSONLocator to be analogous to that of other locators.
  • resource

    • Added resource iterator functionality.
  • scripts

    • Issue #71: Updated launchers to decode shebangs using UTF-8. This allows non-ASCII pathnames to be correctly handled.
    • Ensured that the executable written to shebangs is normcased.
    • Changed ScriptMaker to work better under Jython.
  • util

    • Changed the mode setting method to work better under Jython.
    • Changed get_executable() to return a normcased value.
  • wheel

    • Handled multiple-architecture wheel filenames correctly.
  • docs

    • Numerous documentation updates, not detailed further here.
  • tests

    • Numerous test refinements, not detailed further here.

0.2.0

Released: 2014-12-17

  • compat

    • Updated match_hostname to use the latest Python implementation.
  • database

    • Added download_urls and digests attributes to Distribution.
  • locators

    • Issue #48: Fixed the problem of adding a tuple containing a set (unhashable) to a set, by wrapping with frozenset().
    • Issue #55: Return multiple download URLs for distributions, if available.
  • manifest

    • Issue #57: Remove unhelpful warnings about pattern matches.
  • metadata

    • Updated to reflect changes to PEP 426.
  • resources

    • Issue #50: The type of the path needs to be preserved on 2.x.
  • scripts

    • Updated (including launchers) to support providing arguments to interpreters in shebang lines.
    • The launcher sources are now included in the repository and the source distribution (they are to be found in the PC directory).
    • Added frames support in IronPython (patch by Pawel Jasinski).
    • Issue #51: encode shebang executable using utf-8 rather than fsencode.
  • util

    • Removed reference to __PYVENV_LAUNCHER__ when determining executable for scripts (relevant only on OS X).
    • Updated to support changes to PEP 426.
  • version

    • Updated to reflect changes to versioning proposed in PEP 440.
  • wheel

    • Updated build() code to respect interpreter arguments in prebuilt scripts.
    • Updated to support changes to PEP 426 / PEP 440.
  • docs

    • Numerous documentation updates, not detailed further here.
  • tests

    • Numerous test refinements, not detailed further here.

0.1.9

Released: 2014-05-19

  • index

    • Added keystore keyword argument to signing and verification APIs.
  • scripts

    • Issue #47: Updated binary launchers to fix double-quoting bug where script executable paths have spaces.
  • docs

    • Numerous documentation updates, not detailed further here.
  • tests

    • Numerous test refinements, not detailed further here.

0.1.8

Released: 2014-03-18

  • index

    • Improved thread-safety in SimpleScrapingLocator (issue #45).
    • Replaced absolute imports with relative ones.
    • Added search method to PackageIndex.
  • locators

    • Improved thread-safety in SimpleScrapingLocator (issue #45).
  • metadata

    • Fixed bug in add_requirements implementation.
  • resources

    • The Cache class was refactored into distlib.util.Cache and distlib.resources.ResourceCache classes.
  • scripts

    • Implement quoting for executables with spaces in them.
  • util

    • Gained the Cache class, which is also used in distlib.wheel.
  • version

    • Allowed versions with a single numeric component and a local version component.
    • Adjusted pre-release computation for legacy versions to be the same as the logic in the setuptools documentation.
  • wheel

    • Added verify, update, is_compatible and is_mountable methods to the Wheel class.
    • Converted local version separators from ‘-‘ to ‘_’ and back.
    • If SOABI not available, used Py_DEBUG, Py_UNICODE_SIZE and WITH_PYMALLOC to derive the ABI.
    • Added “exists” property to Wheel instances.
    • Factored out RECORD writing and zip building to separate methods.
    • Provided the ability to determine the location where extensions are extracted, by using the distlib.util.Cache class.
    • Avoided using pydist.json in 1.0 wheels (bdist_wheel writes a non-conforming pydist.json.)
    • Improved computation of compatible tags on OS X, and made COMPATIBLE_TAGS a set.
  • _backport/sysconfig

    • Replaced an absolute import with a relative one.
  • docs

    • Numerous documentation updates, not detailed further here.
  • tests

    • Numerous test refinements, not detailed further here.

0.1.7

Released: 2014-01-16

  • metadata

    • Added some more fields to the metadata for the index.
  • resources

    • Use native literal string in cache path.
    • Issue #40: Now does path adjustments differently for files and zips.
  • scripts

    • Improved checking for venvs when generating scripts.
  • util

    • Issue #39: Fall back to temporary directory for cache if home directory unavailable.
  • wheel

    • Use native literal string in cache path.

0.1.6

Released: 2013-12-31

  • scripts

    • Updated binary launchers because the wrong variant was shipped with the previous release.
  • version

    • Added support for local component in PEP 440 versions.
  • tests

    • Numerous test refinements, not detailed further here.

0.1.5

Released: 2013-12-15

  • compat

    • Changed source of import for unescape in Python >= 3.4.
  • index

    • Used dummy_threading when threading isn’t available.
    • Used https for default index.
  • locators

    • Used dummy_threading when threading isn’t available.
  • scripts

    • Defaulted to setting script mode bits on POSIX.
    • Use uncompressed executable launchers, since some anti-virus products raise false positive errors.
  • util

    • Used dummy_threading when threading isn’t available.
  • docs

    • Updated out-of-date links in overview.
  • tests

    • Used dummy_threading when threading isn’t available.

0.1.4

Released: 2013-10-31

  • scripts

    • Updated the logic for finding the distlib package using a relative, rather than absolute method. This fixes a problem for pip, where distlib is kept in the pip.vendor.distlib package.
  • _backport/sysconfig

    • The analogous change to that made for scripts, described above.

0.1.3

Released: 2013-10-18

  • database

    • Added support for PEP 426 JSON metadata (pydist.json).
    • Generalised digests to support e.g. SHA256.
    • Fixed a bug in parsing legacy metadata from .egg directories.
    • Removed duplicated code relating to parsing “provides” fields.
  • index

    • Changes relating to support for PEP 426 JSON metadata (pydist.json).
  • locators

    • Changes relating to support for PEP 426 JSON metadata (pydist.json).
    • Fixed a bug in scoring download URLs for preference when multiple URLs are available.
    • The legacy scheme is used for the default locator.
    • Made changes relating to parsing “provides” fields.
    • Generalised digests to support e.g. SHA256.
    • If no release version is found for a requirement, prereleases are now considered even if not explicitly requested.
  • markers

    • Added support for markers as specified in PEP 426.
  • metadata

    • Added support for PEP 426 JSON metadata (pydist.json). The old metadata class is renamed to LegacyMetadata, and the (new) Metadata class wraps the JSON format (and also the legacy format, through LegacyMetadata).
    • Removed code which was only used if docutils was installed. This code implemented validation of .rst descriptions, which is not done in distlib.
  • scripts

    • Updated the logic for writing executable files to deal as best we can with files which are already in use and hence cannot be deleted on Windows.
    • Changed the script generation when launchers are used to write a single executable which wraps a script (whether pre-built or generated) and includes a manifest to avoid UAC prompts on Windows.
    • Changed the interface for script generation options: the make and make_multiple methods of ScriptMaker now take an optional options dictionary.
  • util

    • Added extract_by_key() to copy selected keys from one dict to another.
    • Added parse_name_and_version() for use in parsing “provides” fields.
    • Made split_filename more flexible.
  • version

    • Added support for PEP 440 version matching.
    • Removed AdaptiveVersion, AdaptiveMatcher etc. as they don’t add sufficient value to justify keeping them in.
  • wheel

    • Added wheel_version kwarg to Wheel.build API.
    • Changed Wheel.install API (after consultation on distutils-sig).
    • Added support for PEP 426 JSON metadata (pydist.json).
    • Added lib_only flag to install() method.
  • docs

    • Numerous documentation updates, not detailed further here.
  • tests

    • Numerous test refinements, not detailed further here.

0.1.2

Released: 2013-04-30

  • compat

    • Added BaseConfigurator backport for 2.6.
  • database

    • Return RECORD path from write_installed_files (or None if dry_run).
    • Explicitly return None from write_shared_locations if dry run.
  • metadata

    • Added missing condition in todict().
  • scripts

    • Add variants and clobber flag for generation of foo/fooX/foo-X.Y.
    • Added .exe manifests for Windows.
  • util

    • Regularised recording of written files.
    • Added Configurator.
  • version

    • Tidyups, most suggested by Donald Stufft: Made key functions private, removed _Common class, removed checking for huge version numbers, made UnsupportedVersionError a ValueError.
  • wheel

    • Replaced absolute import with relative.
    • Handle None return from write_shared_locations correctly.
    • Fixed bug in Mounter for extension modules not in sub-packages.
    • Made dylib-cache Python version-specific.
  • docs

    • Numerous documentation updates, not detailed further here.
  • tests

    • Numerous test refinements, not detailed further here.
  • other

    • Corrected setup.py to ensure that sysconfig.cfg is included.

0.1.1

Released: 2013-03-22

  • database

    • Updated requirements logic to use extras and environment markers.
    • Made it easier to subclass Distribution and EggInfoDistribution.
  • locators

    • Added method to clear locator caches.
    • Added the ability to skip pre-releases.
  • manifest

    • Fixed bug which caused side-effect when sorting a manifest.
  • metadata

    • Updated to handle most 2.0 fields, though PEP 426 is still a draft.
    • Added the option to skip unset fields when writing.
  • resources

    • Made separate subclasses ResourceBase, Resource and ResourceContainer from Resource. Thanks to Thomas Kluyver for the suggestion and patch.
  • scripts

    • Fixed bug which prevented writing shebang lines correctly on Windows.
  • util

    • Made get_cache_base more useful by parameterising the suffix to use.
    • Fixed a bug when reading CSV streams from .zip files under 3.x.
  • version

    • Added is_prerelease property to versions.
    • Moved to PEP 426 version formats and sorting.
  • wheel

    • Fixed CSV stream reading under 3.x and handled UTF-8 in zip entries correctly.
    • Added metadata and info properties, and updated the install method to return the installed distribution.
    • Added mount/unmount functionality.
    • Removed compatible_tags() function in favour of COMPATIBLE_TAGS attribute.
  • docs

    • Numerous documentation updates, not detailed further here.
  • tests

    • Numerous test refinements, not detailed further here.

0.1.0

Released: 2013-03-02

  • Initial release.

Next steps

You might find it helpful to look at the Tutorial, or the API Reference.

Comments powered by Disqus