Skip to content

Worker Pipeline

The Inklet backend uses three SQS-based workers to process user content asynchronously. Each worker handles a distinct stage of the pipeline: understanding uploaded content, deciding what to push, and rendering the final bitmap.

Overview

flowchart LR
    A[Upload Confirm] --> B[Summary Worker]
    B --> C[Analyze Worker]
    C --> D[Render Worker]
    D --> E[Push QUEUE]
    E --> F[Device Downloads]

After a user uploads content through the Content Upload API, the pipeline processes it through three stages before the result is available for a device to display.

Summary Worker

SQS Queue: inklet-raw-upload-item

Triggered after an upload is confirmed. This worker uses AI to understand the uploaded content and produce a structured summary.

Trigger: Upload confirmation (the final step of the Content Upload flow)

Processing:

  1. Reads the raw uploaded content from the database
  2. For text notes: uses GPT-4o-mini to understand and summarize the content
  3. For image attachments: uses GPT-4o vision to interpret the image and generate a textual description
  4. Writes a summary_content JSON object back to the database

Output: A summary_content JSON record stored on the upload item, containing the AI-generated understanding of the content.

Analyze Worker

SQS Queue: inklet-analyzer

Triggered on a schedule. This worker runs an autonomous AI agent that decides what content should be pushed to which devices.

Trigger: Scheduled invocation (reason: SCHEDULED) or manual via API (reason: MANUAL)

Manual trigger: POST /api/analyze/trigger (auth required) — queues an analyze task for the current user.

Response: {"status": "queued"}

Agent Architecture:

The worker uses the OpenAI Agents SDK to run an agent loop. The agent is given a set of tools and autonomously decides how to act based on the user's content and devices.

Available Tools:

Tool Description
list_notes List the user's uploaded notes
read_note_content Read the full content and summary of a specific note
list_templates List available rendering templates
get_template_detail Get metadata and preview for a template
get_template_schema Get the JSON schema for a template's input variables
list_devices List the user's bound devices
get_push_history Get recent push history for a device
create_push Create a new push record targeting a specific device

Behavior:

The agent autonomously:

  1. Reviews the user's recent notes and their summaries
  2. Inspects available templates and their schemas
  3. Checks which devices are available and their push history
  4. Decides which content to push to which device using which template
  5. Calls create_push with the chosen template, input variables, and target device

The create_push tool creates a push record in PREPARE status and enqueues a message to the Render Worker.

Render Worker

SQS Queue: inklet-render

Triggered when a push record is created by the Analyze Worker's create_push tool. This worker renders the content into a bitmap image suitable for the e-ink display.

Trigger: create_push from the Analyze Worker, or a custom-image push created via POST /api/devices/{id}/custom-push/confirm

Processing:

  1. Obtain the source pixels:
    • HTML templates: load the Jinja2 template, render it with the task's params to HTML, convert HTML → PDF (WeasyPrint) → PNG
    • __custom_image__ template: download the user-uploaded image from S3 (by fileId in the render task params) — no HTML rendering
  2. Resize to the device resolution (800×480) and produce three bitmap variants:
    • image.png — 1-bit Floyd–Steinberg dithered PNG
    • image2.raw — 1 bpp packed bitmap (48000 bytes)
    • image4.raw — 2 bpp packed bitmap, 4 grayscale levels (96000 bytes)
  3. Upload all three to S3 under render/{userId}/{deviceId}/{pushId}/
  4. Create a file record pointing to image.png
  5. Update the push status from PREPARE to QUEUE

Output: Three rendered bitmaps in S3 and a push record in QUEUE status, ready for the device to download in its preferred format.

Push Lifecycle

Each push record transitions through four statuses:

stateDiagram-v2
    [*] --> PREPARE : create_push
    PREPARE --> QUEUE : Render complete
    QUEUE --> PUBLISHED : Device requests push
    PUBLISHED --> CONFIRMED : Device confirms display
Status Description
PREPARE Push record created; rendering is in progress
QUEUE Rendering complete; bitmap is ready for the device to download
PUBLISHED The push URL has been sent to the device (via API or MQTT)
CONFIRMED The device has confirmed the push was displayed on screen

Devices retrieve pushes using the GET /api/devices/{id}/push endpoint or the get_push / refresh_push MQTT topics documented in the IoT Protocol page.