Deploy a pipeline with GitHub Actions

Before you can deploy a pipeline, you will need to:

1. Install dlt;
2. Create a pipeline;
3. Sign up for a GitHub account, since you will be deploying using GitHub Actions.

Add your dlt project directory to GitHub

You will need a GitHub repository for your project. If you don't have one yet, you need to initialize a Git repo in your dlt project directory and push it to GitHub as described in Adding locally hosted code to GitHub.

Ensure your pipeline works

Before you can deploy, you need a working pipeline. Make sure that it is working by running

python3 chess_pipeline.py # replace chess_pipeline.py with your pipeline file

This should successfully load data from the source to the destination once.

Initialize deployment

First you need to add additional dependencies that deploy command requires:

pip install "dlt[cli]"

then the command below will create a Github workflow that runs your pipeline script every 30 minutes:

dlt deploy chess_pipeline.py github-action --schedule "*/30 * * * *"

It checks that your pipeline has run successfully before and creates a GitHub Actions workflow file run_chess_workflow.yml in .github/workflows with the necessary environment variables.

Add the secret values to GitHub

Copy and paste each Name and Secret pair printed out by the dlt deploy command line tool to the GitHub UI located at the github.com/.../settings/secrets/actions link, which was also printed out by the dlt deploy command line tool.

Add, commit, and push your files

To finish the GitHub Actions workflow setup, you need to first add and commit your files:

git add . && git commit -m 'pipeline deployed with github action'

And then push them to GitHub:

git push origin

Monitor (and manually trigger) the pipeline

The pipeline is now running every 30 minutes as you have scheduled. The dlt deploy command line tool printed out a github.com/.../actions/workflows/run_chess_workflow.yml link where you can monitor (and manually trigger) the GitHub Actions workflow that runs your pipeline in your repository.

Known limitations

The GitHub cron scheduler has fidelity of ~30 minutes. You cannot expect that your job will be run at the exact the intervals or times you specify.

• The minimum official supported interval is 5 minutes.
• If you set it to 5 minutes, you can expect intervals between 5 and 30 minutes.
• From practical experience, any intervals above 30 minutes work on average as expected.