Prefect​

Prefect is a workflow management system that allows you to define, schedule, and monitor workflows.

Go to Prefect

Introduction​

In this post, we will learn how to deploy Prefect flows on a work pool, for triggering the flow via API.

Prerequisites​

• Python installed
• Prefect installed
• Prefect server running (cmd: prefect server start)

Steps​

1. Create a work pool
2. Transform function to flow
3. Deploy flow on work pool
4. Trigger flow via API

Create a work pool​

More about work pools

Create a Process work pool

prefect work-pool create --type process my-work-pool

Check the work pool

prefect work-pool ls

Start the work pool

prefect worker start --pool my-work-pool

Transform function to flow​

Create a folder code

In this folder, create 2 files with one function each, who will be transformed to flow.

from prefect import flow
@flow
def my_flow_1():
    print("This is flow 1")
my_flow_1.run()

from prefect import flow
@flow
def my_flow_2():
    print("This is flow 2")
my_flow_2.run()

With there files, if u want, you can run the flows with the command python code/flow1.py and python code/flow2.py.

Deploy flow on work pool​

More about deploying flows

More about YAML configuration

Now we will deploy the flows on the work pool.

For easy deployment, we will create a YAML file with the flows configurations.

Create a file prefect.yaml with the following content:

# Generic metadata about this project
name: test_flows_deployment
prefect-version: 3.1.0

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.deployments.steps.set_working_directory:
    directory: /path/to/your/project

# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: flow1
  version: null
  tags: []
  concurrency_limit: null
  description: null
  entrypoint: code/flow1.py
  parameters: {}
  work_pool:
    name: my-work-pool
    work_queue_name: null
    job_variables: {}
  enforce_parameter_schema: true
  schedules: []

- name: flow2
  version: null
  tags: []
  concurrency_limit: null
  description: null
  entrypoint: code/flow2.py
  parameters: {}
  work_pool:
    name: my-work-pool
    work_queue_name: null
    job_variables: {}
  enforce_parameter_schema: true
  schedules: []

For deploying the flows based on the YAML file, run the following command:

prefect deploy --all

You can deploy a specific flow with the command:

prefect deploy --name flow1 --name flow2

Trigger flow via API​

More about API

More about API Orchestration

Now we can trigger the flows via API.

import asyncio
from prefect.client.orchestration import get_client
from prefect.deployments import run_deployment

# with the client
async def call():
    client = get_client()
    await client.create_flow_from_name("flow1")

# OR with the deployment
async def call():
    await run_deployment("flow1")

if __name__ == "__main__":
    asyncio.run(call())

Nginx, HTTPS, SSL​

Requirements:

• Server with Ubuntu
• Python
• Git

Install Project​

Clone your project

git clone url_project

Create Python Env

python3 -m venv venv

Active Env

source venv/bin/activate

Install libs

pip install -r requirements.txt

Launch Streamlit

streamlit run app.py

Install Nginx​

Installation

sudo apt update
sudo apt install nginx

Commands Nginx

sudo systemctl stop nginx
sudo systemctl start nginx
sudo systemctl restart nginx
sudo systemctl reload nginx
sudo systemctl disable nginx
sudo systemctl enable nginx

Adjusting Firewall​

List the application configurations that ufw knows

sudo ufw app list

Output
Available applications:
  Nginx Full
  Nginx HTTP
  Nginx HTTPS
  OpenSSH

Allow traffic on port 80

sudo ufw allow 'Nginx HTTP'

Check Status

sudo ufw status

Output
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
Nginx HTTP                 ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
Nginx HTTP (v6)            ALLOW       Anywhere (v6)

Verify Web Server​

Check status

systemctl status nginx

Output
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active:active (running) since Fri 2022-03-01 16:08:19 UTC; 3 days ago
     Docs: man:nginx(8)
 Main PID: 2369 (nginx)
    Tasks: 2 (limit: 1153)
   Memory: 3.5M
   CGroup: /system.slice/nginx.service
           ├─2369 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           └─2380 nginx: worker process

Enter it into your browser’s address bar:

http://your_server_ip

Setting Up Server Blocks​

Create the directory for your_domain as follows, using the -p flag to create any necessary parent directories:

sudo mkdir -p /var/www/**your_domain**/html

Assign ownership of the directory with the $USER environment variable:

sudo chown -R $USER:$USER /var/www/**your_domain**/html

To ensure that your permissions are correct and allow the owner to read, write, and execute the files while granting only read and execute permissions to groups and others

sudo chmod -R 755 /var/www/**your_domain**

Create a server block with the correct directives. Instead of modifying the default configuration file directly,

let’s make a new one at /etc/nginx/sites-available/your_domain:

sudo nano /etc/nginx/sites-available/**your_domain**

Paste in the following configuration block

server {
    listen 80;
    listen [::]:80;

    server_name **your_domain**;

    location / {
        proxy_pass http://0.0.0.0:8501;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
    }
}

Next, let’s enable the file by creating a link from it to the sites-enabled directory, which Nginx reads from during startup:

sudo ln -s /etc/nginx/sites-available/**your_domain** /etc/nginx/sites-enabled/

Test to make sure that there are no syntax errors in any of your Nginx files:

sudo nginx -t

Restart Nginx

sudo systemctl restart nginx

You can test this by navigating to http://your_domain

Allowing HTTPS Firewall​

Allow the Nginx Full profile and delete the redundant Nginx HTTP profile allowance:

sudo ufw allow 'Nginx Full'
sudo ufw delete allow 'Nginx HTTP'

Your status should now look like this:

sudo ufw status

Output
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
Nginx Full                 ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
Nginx Full (v6)            ALLOW       Anywhere (v6)

Obtaining SSL Certificates​

Install certbot

sudo apt install certbot python3-certbot-nginx

Create Certificates

sudo certbot --nginx -d **your_domain** -d www.**your_domain**

Output
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/example.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/example.com/privkey.pem
   Your cert will expire on 2020-08-18. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Your domain's config file (created earlier) must have been modified automatically.

Update Nginx Configuration​

Open the configuration file for your domain

sudo cat /etc/nginx/sites-available/**your_domain**

Verify if the file has been modified

If not, replace with the following:

server {

    server_name **your_domain** www.**your_domain**;

    location / {
        proxy_pass http://0.0.0.0:8501;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
    }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/**your_domain**/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/**your_domain**/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {
    if ($host = www.**your_domain**) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    if ($host = **your_domain**) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    listen [::]:80;

    server_name **your_domain** www.**your_domain**;
    return 404; # managed by Certbot
}

server {
  listen 80;
  listen [::]:80;

  server_name **your_domain www.your_domain**;

  return 301 https://$http_host$request_uri;
}

Reload Nginx

sudo rm /etc/nginx/sites-enabled/**your_domain**
sudo ln -s /etc/nginx/sites-available/**your_domain** /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

You can test this by navigating to https://your_domain