FAPEC: compression for genomics maritime text and logs image and video generic science data tailored data types

FAPEC Core

  • Resilience

    In presence of outliers and noise, many statistical (entropy) compressors reduce their coding efficiency significantly. FAPEC efficiency is typically over 90% of the theoretical limit even in very unfavourable conditions.

  • Adaptive

    Our algorithm builds an optimized histogram on the fly and performs a smart statistical analysis regularly, leading to a nearly-optimum coding configuration even with rapidly changing statistical properties.

  • Speed

    Thanks to its highly optimized implementation, FAPEC performs as fast as other simpler algorithms such as Adaptive Rice-Golomb.

  • Full-range

    FAPEC adequately handles very compressible data to take the most of it, as well as very uncompressible data to make sure we do not expand it. Overall, FAPEC typically outperforms other algorithms such as Huffman.

  • Hardware

    A simplified version of the FAPEC core is available for FPGA devices in VHDL language.

FAPEC framework

1
2
3
4
5
6
7
1

Auto-configuration: FAPEC has an autoconfiguration stage which selects the best compression strategy automatically.

2

Auto-configurable: You can choose between a manual configuration of the many FAPEC options to get the most from your specific data, or let FAPEC automatically choose among its several pre-processing stages and compression options.

3

Portable: FAPEC is completely written in ANSI C with minor variations for different platforms. It has been tested on Windows, Linux, Mac and VxWorks, on x86, PowerPC and ARM platforms.

4

Multi-threaded: For maximum speed even with complex pre-processing stages, you can use up to 63 compression or decompression threads, allowing to handle over 1 GB per second – as long as your disk can cope with that!

5

Encryption: OpenSSL is used for a reliable AES-256 encryption of your compressed data, all in one go while you compress it and with a negligible computing overhead.

6

Reliability: Data is compressed in independent chunks of typically 1 MB, minimizing data loss in case of file corruption or network cuts.

7

Multi-file: FAPEC can compress over a million files or folders into a single file, quickly and reliably, also minimizing lost files in case of compressed file corruption.

Key aspects:

Two-stage

FAPEC compresses data in two stages, meaning a first pre-processing or decorrelation (highly configurable) followed by data compression in itself (our FAPEC entropy coding core).

Auto-configurable

You can choose between a manual configuration of the many FAPEC options to get the most from your specific data, or let FAPEC automatically choose among its several pre-processing stages and compression options.

Portable

FAPEC is completely written in ANSI C with minor variations for different platforms. It has been tested on Windows, Linux, Mac and VxWorks, on x86, PowerPC and ARM platforms.

Multi-threaded

For maximum speed even with complex pre-processing stages, you can use up to 63 compression or decompression threads, allowing to handle over 1 GB per second – as long as your disk can cope with that!

Encryption

OpenSSL is used for a reliable AES-256 encryption of your compressed data, all in one go while you compress it and with a negligible computing overhead.

Reliability

Data is compressed in independent chunks of typically 1 MB, minimizing data loss in case of file corruption or network cuts.

Multi-file

FAPEC can compress over a million files or folders into a single file, quickly and reliably, also minimizing lost files in case of compressed file corruption.

FAPEC Pre-Proc Stages