How to build and version control pipelines

Before using the pipeline, you need to build it. Building it merges the custom logic of the pipeline itself with the CLI and UI logic so that you can later use it.

Building a pre-trained pipeline

BojAI contains several pre-trained pipelines, which you do not need to code and you can just use to process, train, and deploy pipelines. You can follow the steps below to use them:

  1. Run bojai list --pipelines to see what pipelines are available. You can also optionally refer to the pipelines’ documentation to get details on what each pipeline does, how it works, and what models it uses.

  2. Once you know which pipeline to use, run bojai build --pipeline chosen-pipeline-name.

Now you have a built pipeline and you can use it either through command-line interface, or visual interface.

Building a custom pipeline

If you want to build your custom pipeline, you need to create it first by following the steps below:

  1. Run bojai create --pipeline your-custom-pipeline --directory directory-where-to-save-pipeline-folder. This creates a folder with the name you gave in --pipeline in the directory you specified in --directory.

  2. Access the pipeline folder in the directory you specified, and insert your data processing, training, evaluation, and model usage logic. Refer to How to code a custom pipeline for more details.

Once coding of one stage is finished, you can start using your pipeline. To use your pipeline you need to build it then start it. You can build your model by following the steps below:

  1. Run bojai list --build to see if you had already built your pipeline. If you did, you can rebuild it but the files will be overwritten.

  2. Run bojai build --piplein give-a-name --directory custom-pipeline-directory if this is the first time you build it, or
    Run bojai build --directory custom-pipeline-directory --replace if you had already built it and want to update it.

The directory when building your custom model is the directory you entered in the bojai create command plus the name of your pipeline. For example: if you run bojai create --pipeline mypip --directory ./pipelines then the build command should be bojai build --directory ./pipelines/mypip.

Now you have a built pipeline and you can use it either through command-line interface, or visual interface.

Building a modified pipeline or changed UI/CLI

If you modified any of the pre-built pipelines, and/or you changed the CLI or UI using bojai checkout, you can build your model by running the command below: bojai build --piplein pipeline-name --directory checkout-directiony,modifydirectory

Meaning, you can provide multiple directories, the build will prioritize the files in the provided directories before using the default ones. Meaning, your modified UI/CLI or modified pipelines will have priority.

If you are using your own pipeline and checkout out UI and CLI or both, you can follow the usual build a custom pipeline process along with providing the checkout directories separated by commas in the following style: bojai build --piplein give-a-name --directory custom-pipeline-directory,checkout-directiony

Version control through git

You can use git to version control your custom pipeline. This is especially useful if you’re actively improving the processor, trainer, or user logic over time, or working with a team.

Version control happens on the custom pipeline’s directory — the one you pass to --directory when building.

Follow the steps below to version control your pipeline:

  1. Go to your custom pipeline directory:
    cd path/to/your/custom-pipeline

  2. Initialize a git repository in the folder:
    git init

  3. Add the files you want to track:
    git add .

  4. Commit your changes:
    git commit -m "Initial version of my custom pipeline"

  5. (Optional) Connect to a remote repository (like GitHub or GitLab) to back up your work or collaborate:

    git remote add origin https://github.com/yourusername/yourrepo.git
    git push -u origin main
    
  6. Every time you make changes to your code (like improving your trainer or processor), you can repeat the following:

    git add .
    git commit -m "Describe what you changed"
    git push
    
  7. Finally, after each git update or changes to your pipeline, remember to rebuild your pipeline before using it:
    bojai build --directory path/to/your/custom-pipeline --replace

This makes sure the most recent version of your code is what gets run when you use BojAI.

You now have a version-controlled pipeline that is easier to manage, track, and share.