Python SDK Development
This page provides general Python development guidelines and source build instructions
Prerequisitesβ
This is required for developing & testing changes, we recommend installing pysubmarine in its own conda environment by running the following
conda create --name submarine-dev python=3.7
conda activate submarine-dev
# Install auto-format and lints from current checkout
pip install -r ./dev-support/style-check/python/lint-requirements.txt
# Install mypy from current checkout
pip install -r ./dev-support/style-check/python/mypy-requirements.txt
# test-requirements.txt from current checkout
pip install -r ./submarine-sdk/pysubmarine/github-actions/test-requirements.txt
# Installs pysubmarine from current checkout
pip install -e ./submarine-sdk/pysubmarine
PySubmarine Dockerβ
We also use docker to provide build environments for CI, development, generate python sdk from swagger.
./run-pysubmarine-ci.sh
The script does the following things:
- Start an interactive bash session
- Mount submarine directory to /workspace and set it as home
- Switch user to be the same user that calls the
run-pysubmarine-ci.sh
Coding Styleβ
- Use isort to sort the Python imports and black to format Python code
- Both style is configured in
pyproject.toml
- To autoformat code
./dev-support/style-check/python/auto-format.sh
- Use flake8 to verify the linter, its' configure is in
.flake8
. - Also, we are using mypy to check the static type in
submarine-sdk/pysubmarine/submarine
. - Verify linter pass before submitting a pull request by running:
./dev-support/style-check/python/lint.sh
- If you encouter a unexpected format, use the following method
# fmt: off
"Unexpected format, formated by yourself"
# fmt: on
Unit Testingβ
We are using pytest to develop our unit test suite. After building the project (see below) you can run its unit tests like so:
cd submarine-sdk/pysubmarine
- Run unit test
pytest --cov=submarine -vs -m "not e2e"
- Run integration test
pytest --cov=submarine -vs -m "e2e"
Before run this command in local, you should make sure the submarine server is running.
Generate python SDK from swaggerβ
We use open-api generator to generate pysubmarine client API that used to communicate with submarine server.
To generate different API Component, please change the code in Bootstrap.java. If just updating java code for
NotebookRestApi
,ExperimentRestApi
orEnvironmentRestApi
, please skip step 1.SwaggerConfiguration oasConfig = new SwaggerConfiguration()
.openAPI(oas)
.resourcePackages(Stream.of("org.apache.submarine.server.rest")
.collect(Collectors.toSet()))
.resourceClasses(Stream.of("org.apache.submarine.server.rest.NotebookRestApi",
"org.apache.submarine.server.rest.ExperimentRestApi",
"org.apache.submarine.server.rest.EnvironmentRestApi")
.collect(Collectors.toSet()));After starting the server,
http://localhost:8080/v1/openapi.json
will includes API specs forNotebookRestApi
,ExperimentRestApi
andEnvironmentRestApi
swagger_config.json defines the import path for python SDK
Ex:
For
submarine.client
{
"packageName" : "submarine.client",
"projectName" : "submarine.client",
"packageVersion": "0.9.0-SNAPSHOT"
}Usage:
import submarine.client...
Execute
./dev-support/pysubmarine/gen-sdk.sh
to generate latest version of SDK.Notice: Please install required package before running the script: lint-requirements.txt
Model Management Model Developmentβ
For local development, we can access cluster's service easily thanks to telepresence. To elaborate, we can develop the sdk in local but can reach out to database and minio server by proxy.
- Install telepresence follow the instruction.
- Start proxy pod
telepresence --new-deployment submarine-dev
- You can develop as if in the cluster.
Upload package to PyPiβ
For Apache Submarine committer and PMCs to do a new release.
- Change the version from 0.x.x.dev to 0.x.x in setup.py
- Install Python packages
cd submarine-sdk/pysubmarine
pip install -r github-actions/pypi-requirements.txt
- Compiling Your Package
It will create build
, dist
, and project.egg.info
in your local directory
python setup.py bdist_wheel
- Upload python package to TestPyPI for testing
python -m twine upload --repository testpypi dist/*
- Upload python package to PyPi
python -m twine upload --repository-url https://upload.pypi.org/legacy/ dist/*