This project has retired. For details please refer to its Attic page.
Development Guide | Apache Submarine
Skip to main content
Version: master πŸƒ

Development Guide

Project Overview

The document Submarine Quickstart shows how to deploy the Submarine service to your Kubernetes cluster. The Submarine service consists mainly of nine components, and you can check them with the following command:

kubectl get pods -n ${your_namespace}

A brief introduction about these components:

  1. tf-operator: Enable users to run TensorFlow jobs distributedly
  2. pytorch-operator: Enable users to run PyTorch jobs distributedly
  3. notebook-controller: Jupyter Notebook controller
  4. submarine-istio: Kubernetes Ingress controller
  5. submarine-database: A MySQL database to store metadata
  6. submarine-minio: An object store for machine learning artifacts
  7. submarine-mlflow: A platform for model management
  8. submarine-tensorboard: A visualization tool for distributed training experiments
  9. submarine-server: Handle API requests, and submit distributed training experiments to Kubernetes.
  10. submarine-agent: Listening to the status of experiments and notebooks

Submarine Development

Video​

  • From this Video, you will know how to deal with the configuration of Submarine and be able to contribute to it via Github.

Develop server​

Prerequisites​

  • JDK 11
  • Maven 3.3 or later ( < 3.8.1 )
  • Docker

Setting up checkstyle in IDE​

Checkstyle plugin may help to detect violations directly from the IDE.

  1. Install Checkstyle+IDEA plugin from Preference -> Plugins
  2. Open Preference -> Tools -> Checkstyle
    1. Set Checkstyle version:
      • Checkstyle version: 8.0
    2. Add (+) a new Configuration File
      • Description: Submarine
      • Use a local checkstyle ${SUBMARINE_HOME}/dev-support/maven-config/checkstyle.xml
  3. Open the Checkstyle Tool Window, select the Submarine rule and execute the check

Testing​

  • Unit Test

    For each class, there is a corresponding testClass. For example, SubmarineServerTest is used for testing SubmarineServer. Whenever you add a funtion in classes, you must write a unit test to test it.

  • Integration Test: IntegrationTestK8s.md

Build from source​

  • Before building

    1. We assume the developer use minikube as a local kubernetes cluster.
    2. Make sure you have installed the submarine helm-chart in the cluster.

  1. Package the Submarine server into a new jar file

    mvn install -DskipTests

  2. Build the new server docker image in minikube

    # switch to minikube docker daemon to build image directly in minikube
    eval $(minikube docker-env)

    # run docker build
    ./dev-support/docker-images/submarine/build.sh

    # exit minikube docker daemon
    eval $(minikube docker-env -u)

  3. Delete the server deployment and the operator will create a new one using the new image

    kubectl delete deployment submarine-server -n submarine-user-test

Develop workbench​

  1. Deploy the Submarine

    Follow Getting Started/Quickstart, and make sure you can connect to http://localhost:32080 in the browser.

  2. Install the dependencies

    cd submarine-workbench/workbench-web
    npm install

  3. Run the workbench based on proxy server

    npm run start
    1. The request sent to http://localhost:4200 will be redirected to http://localhost:32080.
    2. Open http://localhost:4200 in browser to see the real-time change of workbench.

  4. Frontend E2E test: IntegrationTestE2E.md

Develop database​

  1. Build the docker image

    # switch to minikube docker daemon to build image directly in minikube
    eval $(minikube docker-env)

    # run docker build
    ./dev-support/docker-images/database/build.sh

    # exit minikube docker daemon
    eval $(minikube docker-env -u)

  2. Deploy new pods in the cluster

    helm upgrade --set submarine.database.dev=true submarine ./helm-charts/submarine

Develop operator​

For details, please check out the README and Developer Guide on GitHub.

Develop Submarine Website​

Submarine website is built using Docusaurus 2, a modern static website generator.

We store all the website content in markdown format in the submarine/website/docs. When committing a new patch to the submarine repo, Docusaurus will help us generate the html and javascript files and push them to https://github.com/apache/submarine-site/tree/asf-site.

To update the website, click β€œEdit this page” on the website.

Add a new page​

If you want to add a new page to the website, make sure to add the file path to sidebars.js.

Installation​

We use the yarn package manager to install all dependencies for the website

yarn install

Build​

Make sure you can successfully build the website before creating a pull request.

yarn build

Local Development​

This command starts a local development server and open up a browser window. Most changes are reflected live without having to restart the server.

yarn start