This document mainly describes the structure of each module of the Submarine project, the development and test description of each module.
2. Submarine Project Structure
Provide the CLI interface for submarine user. (Currently only support YARN service)
The operator for Submarine application. For details, please see the README on github.
Define utility function used in multiple packages, mainly related to hadoop.
Store the pre-release files.
Provide Python SDK for submarine user.
Provide authorization for Apache Spark to talking to Ranger Admin.
Include core server, restful api, and k8s/yarn submitter.
Provide end-to-end and k8s test for submarine.
- workbench-server: is a Jetty-based web server service. Workbench-server provides RESTful interface and Websocket interface. The RESTful interface provides workbench-web with management capabilities for databases such as project, department, user, and role.
- workbench-web: is a web front-end service based on Angular.js framework. With workbench-web users can manage Submarine project, department, user, role through browser. You can also use the notebook to develop machine learning algorithms, model release and other lifecycle management.
- mini-submarine: by using the docker image provided by Submarine, you can experience all the functions of Submarine in a single docker environment, while mini-submarine also provides developers with a development and testing environment, Avoid the hassle of installing and deploying the runtime environment.
- submarine-installer: submarine-installer is our submarine runtime
environment installation tool for yarn-3.1+ and above.By using
submarine-installer, it is easy to install and deploy system services such as
Calico networketc. required by yarn-3.1+.