PGMQ(PostgreSQL Message Queue) Setup ·

Table of Contents

PGMQ (PostgreSQL Message Queue)
#

A lightweight message queue. Like AWS SQS and RSMQ but on Postgres.

https://github.com/tembo-io/pgmq
PGMQ is a lightweight message queue based on PostgreSQL.
It is implemented using a PostgreSQL Extension.

Since it is based solely on PostgreSQL, it is very suitable for lightweight business needs that require an MQ.

Currently, the usage scenario is using the existing API Server + PGMQ + consumer to handle some asynchronous tasks and retry mechanisms with third-party services.

Installation
#

Tembo Docker Image
#

First, ensure your system has the following installed:

Docker
Docker Compose

The simplest way is to use the official Tembo Docker Image:

docker run -d --name postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 quay.io/tembo/pg16-pgmq:latest

Installing from Extension
#

If you want to install using the original postgres image as an extension:

Download the latest PGMQ SQL and Control files:

curl -o pgmq.sql https://raw.githubusercontent.com/tembo-io/pgmq/main/pgmq-extension/sql/pgmq.sql
curl -o pgmq.control https://raw.githubusercontent.com/tembo-io/pgmq/main/pgmq-extension/pgmq.control

Update docker-compose.yml to add the volume settings:

db.sql contains some initialization SQL files (depending on the project’s needs)
pgmq.sql and pgmq.control are the PGMQ SQL and Control files
- The version number in pgmq--1.3.3.sql should match the version number in pgmq.control
stateful_volumes/postgresql is the directory mounted for postgres container persistence

version: '3.8'

services:
  postgres:
    image: postgres:15.1
    container_name: postgres
    ports:
      - 5432:5432
    env_file:
      - postgres.env
    restart: always
    volumes:
      - ./db.sql:/docker-entrypoint-initdb.d/db.sql
      - ./pgmq.control:/usr/share/postgresql/15/extension/pgmq.control
      - ./pgmq.sql:/usr/share/postgresql/15/extension/pgmq--1.3.3.sql
      - ./stateful_volumes/postgresql:/var/lib/postgresql/data/
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "your_postgres_user", "-d", "your_postgres_db"]
      interval: 2s
      timeout: 5s
      retries: 3

Start the postgres container:

If you have previously run a postgres container, remember to delete the volume first!

docker compose up postgres -d

PGMQ Python Client
#

Install the pgmq Python Client:

pip install tembo-pgmq-python

Since tembo-pgmq-python directly uses psycopg (which is psycopg3, not psycopg2)
it may have issues when installed with poetry on Mac.
It is recommended to install it directly with pip.

Using the pgmq Python Client:

The tembo-pgmq-python README is quite clear.
The main components are the PGMQueue and Message data classes.
The vt stands for visibility timeout, which is the time duration during which a message will not be re-read after being read.

Here is a simple example:

from tembo_pgmq_python import PGMQueue, Message

queue = PGMQueue(
    host="0.0.0.0",
    port="5432",
    username="postgres",
    password="postgres",
    database="postgres"
)

msg_id: int = queue.send("my_queue", {"hello": "world"})

read_message: Message = queue.read("my_queue", vt=30)
print(read_message)

deleted: bool = queue.delete("my_queue", read_message.msg_id)

dropped: bool = queue.drop_queue("my_queue")