setuptools - python setup.py sdist and custom setup keywords don't play together -

subtitle: not sdist

i trying setup.py file of package i'm working on play nicely sdist. relevant parts of setup.py file are:

from setuptools.command.test import test [...] class tox(test):    "as described in      http://tox.readthedocs.org/en/latest/example/basic.html?highlight=setuptools#integration-with-setuptools-distribute-test-commands"    [...] def run_tests(self):     if self.distribution.install_requires:         self.distribution.fetch_build_eggs(             self.distribution.install_requires)     if self.distribution.tox_requires:         self.distribution.fetch_build_eggs(self.distribution.tox_requires)     # import here, cause outside eggs aren't loaded     import tox     import shlex     args = self.tox_args     if args:         args = shlex.split(self.tox_args)     else:         args = ""     errno = tox.cmdline(args=args)     sys.exit(errno)   entry_points ={} distutils_ext = {'distutils.setup_keywords': [                     "tox_requires = setuptools.dist:check_requirements", ]                  } entry_points.update(distutils_ext)  setup(       install_requires=['six', 'numpy', 'matplotlib', 'scipy', 'astropy>=1',                   'pillow', ],      cmdclass={         'test': pytest,  # run python setup.py test         'tox': tox,     },      # list of packages , data     packages=find_packages(),      # tests     tests_require=['pytest', 'pytest-cov'],     tox_requires=['tox'],     # other keywords, metadata ) 

if run python setup.py sdist, warning @ beginning:

/usr/lib/python2.7/distutils/dist.py:267: userwarning: unknown distribution option: 'tox_requires'   warnings.warn(msg) 

but sdist works fine , creates tar.gz file can use install package.

but if run second time, starts (it's beginning of pillow building):

warning: no previously-included files found matching '.editorconfig' building using 4 processes _imaging.c: in function ‘getink’: 

and begins building required packages .eggs directory.

if remove *egg-info directory can rerun command. if comment out tox_requires=[...] line, can build sdist many times want.

now according setuptools documentation command above should correct way run add new arguments setup function.

---

as per subtitle, problem not sdist it's due non-understanding on how setuptools , requirements work.

if run python setup.py tox in place without tox installed get, after installation of testing package should not install (namely pytest , pytest-cov):

traceback (most recent call last): [...] file "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 127, in with_project_on_sys_path func() file "setup.py", line 65, in run_tests if self.distribution.tox_requires: attributeerror: distribution instance has no attribute 'tox_requires'

---

[update] tox_requires confuse badly pip during installation. if commented out can install package without issue; otherwise begins compile source of packages , systematically fails because doesn't find numpy while building stuff scipy

---

how can setuptools recognize , use tox_requires?

once issue fixed, think can rid of spurious installations here doing better job @ implementing tox class, maybe overriding more things test or deriving directly command

complete (working) solution described below consist of 8 files (incl. short readme.rst) , has in total 43 lines of code. less code in original question.

despite of being short, supports many development , testing scenarios in convenient way.

anyway, not answer question, sure, fulfils requirements behind it.

three lines long setup.py

technically may possible put test command including tox automation setup.py, however, result may messy , difficult understand.

the same result can achieved in simpler way:

• for developer assume:

  • using git
  • having tox installed system

• for package user:

  • there no special requirements install resulting package

• (optional) if want users test package single command , keep test reports collected in central server:

  • install devpi-server , give users access it
  • ask users install $ pip install devpi

the solution builds on following tools , packages:

• pbr: simplify package creation incl. versioning via git tags , creation of authors , changelog git commit messages.
• pytest: excelent testing framework, other framework can used instead of it.
• tox: excellent build , test automation tool.
• coverage: tools measure test coverage (working simpler pytest-cov)

optionally may use:

• devpi-server: private pypi server password protected access. allows simple testing , provides test reports collection.
• devpi: tool similar pip. apart installation supports running tox defined tests (install, run tests, publish reports in on step).

authoring package

create new project directory , initialize git:

$ mkdir francesco $ cd francesco $ git init 

create package or module

here create single module francesco, same works more modules or packages.

francesco.py

def main():     print("hi, me, francesco, keeping things simple.") 

requirements.txt

create list of packages actual installation of package:

six 

test_requirements.txt

define packages required testing:

pytest coverage 

tests/test_it.py

initiate test suite:

from francesco import main   def test_this():     main()     print("all seems fine me")     assert true 

setup.py

did dream of stupid simple setup.py? here goes:

from setuptools import setup  setup(setup_requires=["pbr"], pbr=true) 

setup.cfg

metadata belong configuration file:

[metadata] name = francesco author = francesco montesano author-email = fm@acme.com summary = nice , installed python module supporting testing in different pythons description-file = readme.rst [files] modules=francesco [entry_points] console_scripts =     francesco = francesco:main 

tox.ini

to configure tox automated builds , tests:

[tox] envlist = py27, py34  [testenv] commands =     coverage run --source francesco -m pytest -sv tests     coverage report     coverage html deps =     -rtest_requirements.txt 

readme.rst

never forget readme.rst:

=========================================== complex package 3 line long `setup.py` ===========================================  can keep`setup.py` simple , still support automated testing?  ... 

tox: build sdist , run tests in supported python versions

being in project directory root, run single command tox:

$ tox glob sdist-make: /home/javl/sandbox/setuppy/setup.py py27 inst-nodeps: /home/javl/sandbox/setuppy/.tox/dist/francesco-0.0.0.zip py27 runtests: pythonhashseed='2409409075' py27 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests ============================= test session starts ============================== platform linux2 -- python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /home/javl/sandbox/setuppy/.tox/py27/bin/python2.7 cachedir: .cache rootdir: /home/javl/sandbox/setuppy, inifile:  collecting ... collected 1 items  tests/test_it.py::test_this hi, me, francesco, keeping things simple. seems fine me passed  =========================== 1 passed in 0.01 seconds =========================== py27 runtests: commands[1] | coverage report name           stmts   miss  cover ---------------------------------- francesco.py       2      0   100% py27 runtests: commands[2] | coverage html py34 inst-nodeps: /home/javl/sandbox/setuppy/.tox/dist/francesco-0.0.0.zip py34 runtests: pythonhashseed='2409409075' py34 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests ============================= test session starts ============================== platform linux -- python 3.4.2, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /home/javl/sandbox/setuppy/.tox/py34/bin/python3.4 cachedir: .cache rootdir: /home/javl/sandbox/setuppy, inifile:  collecting ... collected 1 items  tests/test_it.py::test_this hi, me, francesco, keeping things simple. seems fine me passed  =========================== 1 passed in 0.01 seconds =========================== py34 runtests: commands[1] | coverage report name           stmts   miss  cover ---------------------------------- francesco.py       2      0   100% py34 runtests: commands[2] | coverage html ___________________________________ summary ____________________________________   py27: commands succeeded   py34: commands succeeded   congratulations :) 

getting sdist

ls .tox/dist francesco-0.0.0.zip 

developing in python 2.7 virtualenv

activate python 2.7 virtualenv

$ source .tox/py27/bin/activate 

run tests

(py27) $ py.test -sv tests  ============================================================================================== test session starts =============================================================================================== platform linux2 -- python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /home/javl/sandbox/setuppy/.tox/py27/bin/python2.7 cachedir: .cache rootdir: /home/javl/sandbox/setuppy, inifile: collected 1 items  tests/test_it.py::test_this hi, me, francesco, keeping things simple. seems fine me passed  ============================================================================================ 1 passed in 0.01 seconds ============================================================================================ 

measure test coverage

(py27)$ coverage run --source francesco -m pytest -sv tests ..... (py27)$ coverage report name           stmts   miss  cover ---------------------------------- francesco.py       2      0   100% 

view coverage report in web browser

(py27)$ coverage html (py27)$ firefox htmlcov/index.html 

release new package version

(optional) install local devpi-server

the installation of devpi-server not covered here, simple, especially, if install local machine personal testing.

commit source code, assign version tag

make sure, source code commited.

assing version tag:

$ git tag -a 0.1 

rerun tests tox , build sdist

make sure, have deactivated virtualenv (otherwise conflicts tox):

(py27)$ deactivate 

run tox:

$ tox ..... ...it builds usual, may fail, if have forgotten commit changes or files... 

find sdist new version of package:

$ ls .tox/dist/francesco-0.1.0. .tox/dist/francesco-0.1.0.zip 

you done. may distribute new tested versions of package users usually.

(optional) upload sdist devpi-server , test locally

following steps assume, have devpi-server installed , running.

$ devpi login javl ...enter password... $ devpi upload .tox/dist/francesco-0.1.0.zip 

test package in clean environment

(deactivate virtualenv if active) :

$ cd /tmp $ mkdir testing $ cd testing $ devpi test francesco received http://localhost:3141/javl/dev/+f/4f7/c13fee84bb7c8/francesco-0.1.0.zip unpacking /tmp/devpi-test6/downloads/francesco-0.1.0.zip /tmp/devpi-test6/zip /tmp/devpi-test6/zip/francesco-0.1.0$ tox --installpkg /tmp/devpi-test6/downloads/francesco-0.1.0.zip -i all=http://localhost:3141/javl/dev/+simple/ --recreate --result-json /tmp/devpi-test6/zip/toxreport.json -c /tmp/devpi-test6/zip/francesco-0.1.0/tox.ini py27 create: /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py27 py27 installdeps: -rtest_requirements.txt py27 inst: /tmp/devpi-test6/downloads/francesco-0.1.0.zip py27 installed: coverage==4.0.3,francesco==0.1.0,py==1.4.31,pytest==2.8.7,six==1.10.0,wheel==0.24.0 py27 runtests: pythonhashseed='3916044270' py27 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests ============================= test session starts ============================== platform linux2 -- python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py27/bin/python2.7 cachedir: .cache rootdir: /tmp/devpi-test6/zip/francesco-0.1.0, inifile: collecting ... collected 1 items  tests/test_it.py::test_this hi, me, francesco, keeping things simple. seems fine me passed  =========================== 1 passed in 0.01 seconds =========================== py27 runtests: commands[1] | coverage report name           stmts   miss  cover ---------------------------------- francesco.py       2      0   100% py27 runtests: commands[2] | coverage html py34 create: /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py34 py34 installdeps: -rtest_requirements.txt py34 inst: /tmp/devpi-test6/downloads/francesco-0.1.0.zip py34 installed: coverage==4.0.3,francesco==0.1.0,py==1.4.31,pytest==2.8.7,six==1.10.0,wheel==0.24.0 py34 runtests: pythonhashseed='3916044270' py34 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests ============================= test session starts ============================== platform linux -- python 3.4.2, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py34/bin/python3.4 cachedir: .cache rootdir: /tmp/devpi-test6/zip/francesco-0.1.0, inifile: collecting ... collected 1 items  tests/test_it.py::test_this hi, me, francesco, keeping things simple. seems fine me passed  =========================== 1 passed in 0.01 seconds =========================== py34 runtests: commands[1] | coverage report name           stmts   miss  cover ---------------------------------- francesco.py       2      0   100% py34 runtests: commands[2] | coverage html ____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________   py27: commands succeeded   py34: commands succeeded   congratulations :) wrote json report at: /tmp/devpi-test6/zip/toxreport.json posting tox result data http://localhost:3141/javl/dev/+f/4f7/c13fee84bb7c8/francesco-0.1.0.zip posted tox result data 

you may check test results in web browser:

$ firefox http://localhost:3141 

then search "francesco" package, click package name, find in table column named "tox results", click there show environment set , test results.

let users test package

let's assume, devpi-server running , user has access it.

the user shall have devpi command installed:

$ pip install devpi 

(note, tool not installing devpi-server)

help user gain access devpi-server (not covering here).

then user runs test:

$ devpi test francesco 

after test run (it automatically using tox, user not have care that), find test results on same place on devpi web interface found yours before.