This project has retired. For details please refer to its Attic page.
Project Architecture | Apache Submarine
Skip to main content
Version: 0.6.0

Project Architecture

1. Introduction​

This document mainly describes the structure of each module of the Submarine project, the development and test description of each module.

2. Submarine Project Structure​

2.1. submarine-client​

Provide the CLI interface for submarine user. (Currently only support YARN service)

2.2. submarine-cloud-v2​

The operator for Submarine application. For details, please see the README on github.

2.3. submarine-commons​

Define utility function used in multiple packages, mainly related to hadoop.

2.4. submarine-dist​

Store the pre-release files.

2.5. submarine-sdk​

Provide Python SDK for submarine user.

2.6. submarine-security​

Provide authorization for Apache Spark to talking to Ranger Admin.

2.7. submarine-server​

Include core server, restful api, and k8s/yarn submitter.

2.8. submarine-test​

Provide end-to-end and k8s test for submarine.

2.9. submarine-workbench​

  • workbench-server: is a Jetty-based web server service. Workbench-server provides RESTful interface and Websocket interface. The RESTful interface provides workbench-web with management capabilities for databases such as project, department, user, and role.
  • workbench-web: is a web front-end service based on Angular.js framework. With workbench-web users can manage Submarine project, department, user, role through browser. You can also use the notebook to develop machine learning algorithms, model release and other lifecycle management.

2.10 dev-support​

  • mini-submarine: by using the docker image provided by Submarine, you can experience all the functions of Submarine in a single docker environment, while mini-submarine also provides developers with a development and testing environment, Avoid the hassle of installing and deploying the runtime environment.
  • submarine-installer: submarine-installer is our submarine runtime environment installation tool for yarn-3.1+ and above.By using submarine-installer, it is easy to install and deploy system services such as docker, nvidia-docker, nvidia driver, ETCD, Calico network etc. required by yarn-3.1+.