FBQueue: Comprehensive Technical Overview

FBQueue (Forblaze Queue) is a robust, lightweight local job scheduler designed for researchers and developers in shared computing environments.

This documentation provides an in-depth technical guide to FBQueue, covering its architecture, resource management, advanced scheduling features, and compatibility with traditional HPC schedulers like PBS and Slurm.

0. Quick Installation

FBQueue is distributed as a single binary and can be installed with a few simple commands.

1. Download and Extract

Choose the package for your architecture (x86_64 or Arm64).

Linux (x86_64):

wget https://github.com/ForblazeProject/fbqueue/releases/download/v0.9.3/fbqueue-linux-x64.tar.gz
tar -xzvf fbqueue-linux-x64.tar.gz

Linux (Arm64/AArch64):

wget https://github.com/ForblazeProject/fbqueue/releases/download/v0.9.3/fbqueue-linux-arm64.tar.gz
tar -xzvf fbqueue-linux-arm64.tar.gz

2. Move to your bin directory

Ensure the directory (e.g., ~/bin) is in your system PATH.

mkdir -p ~/bin
mv fbqueue ~/bin/

If you prefer using familiar commands like qsub or qstat, create symbolic links as follows:

ln -s ~/bin/fbqueue ~/bin/qsub
ln -s ~/bin/fbqueue ~/bin/qstat
ln -s ~/bin/fbqueue ~/bin/qdel

1. Introduction & Philosophy

FBQueue addresses the gap between manual script execution and heavy enterprise schedulers. It is designed and implemented based on the following core principles:

2. Security & Efficiency

FBQueue is built for high-security environments where software installation and network access are restricted.

3. Resource Management (Capacity & Cost)

FBQueue manages computational resources through abstract values of capacity and cost.

4. Multi-Queue and Priority Support

Queues can be defined in ~/.fbqueue/config to control resource distribution and execution order.

# Example Configuration
capacity: 16          # Global maximum capacity
default_queue: batch

queue: batch
  capacity: 8         # Maximum capacity for this queue
  priority: 10        # Lower priority

queue: express
  capacity: 4         # Maximum capacity for this queue
  priority: 100       # Higher priority (processed first)

5. Advanced Scheduling Features

Walltime (-W)

Automatically terminates jobs that exceed the specified execution time limit, preventing runaway processes from consuming resources.

fbqueue sub -W 01:30:00 ./simulation.sh

Dependency Management (-hold_jid)

Holds job execution until the specified job ID(s) finish successfully (DONE). This allows for the execution of complex task-graphs (DAGs).

fbqueue sub -hold_jid 123 ./post_process.sh

Delayed Start (-a)

Schedules a job to start after a specific time. Supports relative time (+1h), specific clock times (18:00), or absolute dates.

fbqueue sub -a 18:00 ./nightly_task.sh

6. PBS/HPC Compatibility

FBQueue maintains high compatibility with the conventions of established HPC schedulers like PBS, SGE, and Slurm.

Directive Parsing

FBQueue automatically recognizes embedded script directives. You can use your existing job scripts without modification.

FeatureSupported Directives
Job Name#PBS -N, #SBATCH -J
Resource/Cost#$ -pe smp N, #SBATCH -c N, #PBS -l ncpus=N
Target Queue#PBS -q, #SBATCH -p
Stdout/Stderr#PBS -o, #PBS -e
Dependencies#PBS -hold_jid
Walltime#PBS -l h_rt=HH:MM:SS

On Linux, you can interact with FBQueue using familiar commands by creating symbolic links:

7. Configuration & Environment

Team Collaboration (Shared Mode)

Multiple users on a single machine can share a common queue by pointing the FBQUEUE_DIR environment variable to a local shared directory. Note: Network-mounted drives (NFS/SMB) are discouraged due to file-locking latency.

8. Windows Support

FBQueue provides native support for Windows with automatic interpreter selection:

Jobs are executed via their respective shells without modifying the original script’s file permissions, ensuring script integrity.