Skip to main content
Version: 0.8.0

Python SDK Development

This page provides general Python development guidelines and source build instructions

Prerequisites​

This is required for developing & testing changes, we recommend installing pysubmarine in its own conda environment by running the following

conda create --name submarine-dev python=3.7
conda activate submarine-dev

# Install auto-format and lints from current checkout
pip install -r ./dev-support/style-check/python/lint-requirements.txt

# Install mypy from current checkout
pip install -r ./dev-support/style-check/python/mypy-requirements.txt

# test-requirements.txt from current checkout
pip install -r ./submarine-sdk/pysubmarine/github-actions/test-requirements.txt

# Installs pysubmarine from current checkout
pip install -e ./submarine-sdk/pysubmarine

PySubmarine Docker​

We also use docker to provide build environments for CI, development, generate python sdk from swagger.

./run-pysubmarine-ci.sh

The script does the following things:

  • Start an interactive bash session
  • Mount submarine directory to /workspace and set it as home
  • Switch user to be the same user that calls the run-pysubmarine-ci.sh

Coding Style​

  • Use isort to sort the Python imports and black to format Python code
  • Both style is configured in pyproject.toml
  • To autoformat code
./dev-support/style-check/python/auto-format.sh
  • Use flake8 to verify the linter, its' configure is in .flake8.
  • Also, we are using mypy to check the static type in submarine-sdk/pysubmarine/submarine.
  • Verify linter pass before submitting a pull request by running:
./dev-support/style-check/python/lint.sh
  • If you encouter a unexpected format, use the following method
# fmt: off
"Unexpected format, formated by yourself"
# fmt: on

Unit Testing​

We are using pytest to develop our unit test suite. After building the project (see below) you can run its unit tests like so:

cd submarine-sdk/pysubmarine
  • Run unit test
pytest --cov=submarine -vs -m "not e2e"
  • Run integration test
pytest --cov=submarine -vs -m "e2e"

Before run this command in local, you should make sure the submarine server is running.

Generate python SDK from swagger​

We use open-api generator to generate pysubmarine client API that used to communicate with submarine server.

  1. To generate different API Component, please change the code in Bootstrap.java. If just updating java code for NotebookRestApi , ExperimentRestApi or EnvironmentRestApi, please skip step 1.

    SwaggerConfiguration oasConfig = new SwaggerConfiguration()
    .openAPI(oas)
    .resourcePackages(Stream.of("org.apache.submarine.server.rest")
    .collect(Collectors.toSet()))
    .resourceClasses(Stream.of("org.apache.submarine.server.rest.NotebookRestApi",
    "org.apache.submarine.server.rest.ExperimentRestApi",
    "org.apache.submarine.server.rest.EnvironmentRestApi")
    .collect(Collectors.toSet()));

    After starting the server, http://localhost:8080/v1/openapi.json will includes API specs for NotebookRestApi, ExperimentRestApi and EnvironmentRestApi

  1. swagger_config.json defines the import path for python SDK

    Ex:

    For submarine.client

    {
    "packageName" : "submarine.client",
    "projectName" : "submarine.client",
    "packageVersion": "0.9.0-SNAPSHOT"
    }

    Usage: import submarine.client...

  2. Execute ./dev-support/pysubmarine/gen-sdk.sh to generate latest version of SDK.

    Notice: Please install required package before running the script: lint-requirements.txt

Model Management Model Development​

For local development, we can access cluster's service easily thanks to telepresence. To elaborate, we can develop the sdk in local but can reach out to database and minio server by proxy.

  1. Install telepresence follow the instruction.
  2. Start proxy pod
telepresence --new-deployment submarine-dev
  1. You can develop as if in the cluster.

Upload package to PyPi​

For Apache Submarine committer and PMCs to do a new release.

  1. Change the version from 0.x.x.dev to 0.x.x in setup.py
  2. Install Python packages
cd submarine-sdk/pysubmarine
pip install -r github-actions/pypi-requirements.txt
  1. Compiling Your Package

It will create build, dist, and project.egg.info in your local directory

python setup.py bdist_wheel
  1. Upload python package to TestPyPI for testing
python -m twine upload --repository testpypi dist/*
  1. Upload python package to PyPi
python -m twine upload --repository-url https://upload.pypi.org/legacy/ dist/*