WinFAPEC in the Microsoft Store, including patch release

To make even easier the installation and use of FAPEC on Microsoft Windows machines, today we have published WinFAPEC in the Microsoft Store.

This first publication in the Store corresponds to WinFAPEC 1.2 with the FAPEC core library 24.0.1 – a patch release that fixes some minor bugs, including an issue with genomics (FastQ) data compression.

As usual, FAPEC (for Linux, macOS and Windows) and WinFAPEC are available in our website, where you can also ask for free evaluation licenses.

The fapyc Python package will soon be updated as well, to use this 24.0.1 patch release of the core library.

We recommend our customers to move to this new release.

 

Release of FAPEC

FAPEC 23 logoicon

Today we have released the new version of our professional data compression software, FAPEC Archiver 24.0, as well as its graphical front-end for Windows, WinFAPEC 1.1.

As in previous versions, the software can be downloaded from our website, at this link. Without any cost, it will provide you free decompression features (with some minor limitations) of .fapec files, both in command-line and with WinFAPEC (for Windows platforms). The command-line compressor and decompressor is available for Windows, macOS (x86_64 and M1 processors) and Linux (x86_64 and ARM processors).

Today we have also released fapyc 0.6.0, the Python wrapper for FAPEC. You can easily get it with a simple command:

pip install fapyc

To compress files (either through the command-line, with WinFAPEC or with fapyc), you will need a valid FAPEC license. You can get free 30-days evaluation licenses from our website. For full licenses, integration in your systems and support, please contact us.

As usual, FAPEC Archiver (and now also WinFAPEC) includes our professional technology:

  • High-performance, multi-threaded, versatile two-stage data compression.
  • Automatic algorithm selection from a quick analysis of each file to be compressed.
  • Professional stages: Kongsberg’s MBES data, CILLIC image and video compression, tabular (CSV) data, FastQ genomics compression, Audio and RF compression…
  • In this version, we have added several improvements for KMALL and KMWCD files compression. Remarkably, FAPEC now offers the option to compress in “instrumentally lossless” mode, removing the noise bits from the data, significantly increasing the ratios while not affecting the quality data.
  • Ultra-fast compression for speeds in the GB/s regime.
  • Lossy compression options for most of the stages, configurable by the user (lossless is set by default).
  • Basic data analytics on-the-fly, allowing to obtain some statistics from each file. It includes, for example, a “flatness map” of images (with CILLIC), or the compressibility of Kongsberg MBES watercolumn datagrams – which allow detecting features in the watercolumn.
  • On-the-fly encryption with XXTEA (and AES-256 for full licenses). Error recovery from corrupted files.
  • API for third-party integration in C/C++, Python, Java, HDF5, FITS…

Have fun!

Release of FAPEC Archiver 23.0 and WinFAPEC 1.0

FAPEC 23 logoicon

We have just released the new version of our professional data compression software, FAPEC Archiver 23.0.

For the first time, we are also including a graphical front-end: WinFAPEC 1.0, for Microsoft Windows.

As in previous versions, the software can be downloaded from our website, at this link. Without any cost, it will provide you free decompression features (yet with some limitations) of .fapec files, both in command-line and with WinFAPEC (for Windows platforms). The command-line compressor and decompressor is available for Windows, macOS (x86_64 and M1 processors) and Linux (x86_64 and ARM processors).

To compress files (either through the command-line or with WinFAPEC), you will need a valid FAPEC license. Once again, you can get free 30-days evaluation licenses from our website. For full licenses, integration in your systems and support, please contact us.

These are some snapshots of WinFAPEC 1.0:

 

As usual, FAPEC Archiver (and now also WinFAPEC) includes our professional technology:

  • High-performance, multi-threaded, versatile two-stage data compression.
  • Automatic algorithm selection from a quick analysis of each file to be compressed.
  • Professional stages: Kongsberg’s MBES data, CILLIC image and video compression, FastQ genomics compression, tabular (CSV) data…
  • Ultra-fast compression for speeds in the GB/s regime.
  • Lossy compression options for most of the stages, configurable by the user (lossless is set by default).
  • Improvements in the Wave compression of multi-channel time series, including a “smart lossy” algorithm. As shown in this paper, it allows to achieve ratios as high as 100 on digital RF data with error-free demodulation.
  • On-the-fly data analysis, allowing to obtain basic statistics from each file. It includes, for example, a “flatness map” of images (with CILLIC), or the compressibility of MBES watercolumn datagrams.
  • On-the-fly encryption with XXTEA (and AES-256 for full licenses). Error recovery from corrupted files.
  • APIs for third-party integration in C/C++, Python, Java, HDF5, FITS…

Have fun!

Release of FAPEC Archiver 22.0 – codename “Merganser”

 

Today, DAPCOM has released the new version of our propietary, high-performance, professional, staged data compressor: FAPEC Archiver 22.0 (codename Merganser).

Anybody can request and download free evaluation licenses, and for the first time, free perpetual decompressors can be downloaded.

It includes several exciting features:

  • New professional stage to compress Kongsberg’s MBES data, including their brand new KMALL (and KMWCD) files, as well as .all and .wcd files.
    • FAPEC achieves the best compression ratios in the market for those files.
    • You can find more information in our brochure.
  • New image compression algorithm, CILLIC (Context-Interpolation Lossless and Lossy Image Compressor), supporting greyscale, colour and hyperspectral images in various coding formats (including Bayer pattern). It offers lossless, near-lossless and lossy options.
  • New compression algorithm for multi-channel time series, Wave, offering lossless and near-lossless options. It is ideal for audio and radio-frequency (RF) data, for example.
  • New ultra-fast compression core, FASEC, for extremely fast lossless compression of single- or multi-dimensional time series or images. It can achieve compression (and decompression) speeds well above 1GB/s on a modest laptop.
  • Our usual professional stages: FastQ (genomics data), Tabular text data (such as CSV or some LIDAR and point cloud formats), LZW and improved FAPECLZ for text data, etc.
  • On-the-fly generation of basic compression statistics for each data chunk and file, which can be extended to perform quick statistical analyses on the data.
  • Multiple file and directories archival (up to 8 million files or folders), keeping dates and permissions.
  • Multi-threaded operation.
  • AES256 and XXTEA-based encryption.
  • Public API to integrate FAPEC compression or decompression in your C/C++ software.
  • Python and Java bindings, to programmatically use FAPEC from your Python or Java code.
  • Integration in HDF5 and FITS.

The codename Merganser comes from the red-breasted merganser, which is claimed to be the fastest duck. We chose a duck name due to the release number and date (2022-02-22), which contain a lot of ducks.

You can get your personal copy of FAPEC here, where you can choose between a 30-day evaluation license to test the compressor, or the free decompression program.

You can find more information on FAPEC here.

First Earth Observation images compressed by the new FAPEC-CILLIC algorithm onboard ESA’s OPS-SAT

The European Space Agency (ESA) launched OPS-SAT on 18 December 2019 into a circular, polar orbit at 515 km altitude.
OPS-SAT is a technology demonstration nanosatellite based on the cubesat standard. The satellite is only 30cm x 10cm x 10cm, but it brings powerful technology and instrumentation onboard, such as a fine Attitude Determination Control System, a GPS receiver, an X-band transmitter (up to 50 Mbps downlink) or a Software Defined Radio front-end.
It also includes a high-performance data processing platform (based on a dual-core ARM Cortex-A9 processor) and a high-resolution camera (based on an RGB Bayer filter) with a bit depth of 12 bits per pixel and a ground resolution of up to 80m x 80m per pixel.

When preparing for this mission, ESA issued a call for experimenters willing to test their technologies in orbit. One of the submitted (and accepted) experiments is the FAPEC data compression software – a versatile and efficient solution for lossless and lossy compression of images and of a large variety of instrumental data.
FAPEC is already being used by a satellite constellation to compress payload data onboard. However, its image compression capabilities had not been demonstrated in orbit yet. Furthermore, FAPEC was recently improved with a new image decorrelation algorithm (named CILLIC), presented at the ESA/CNES On-Board Payload Data Compression workshop (OBPDC) on 22 September 2020. FAPEC, with its CILLIC configuration, offers lossless and lossy image compression performances similar to those achieved by wavelet transforms but at a fraction of their computational cost.

On 28 November 2020, FAPEC was uploaded to OPS-SAT. The day after, and for the very first time in orbit, it was invoked by the payload data processing system to compress two images of our planet taken from Space.
Each of the two images, with a resolution of 4 Mpix and weighting 7.6 MB, were reduced to nearly 800 KB in about 0.8 seconds in a single computing thread.
Once received on ground, they were “de-bayered” with the ImageJ software to obtain the original colour images, which are shown below.

First FAPEC-compressed images from ESA’s OPS-SAT. The colour pictures shown here have been obtained by applying a standard de-bayer algorithm on ground, without any colour correction or white balancing in this case.
Photos: ESA/OPS-SAT

 

Shortly after this, more images were acquired and, again, compressed with FAPEC to be later downlinked to the ground station. These ones better illustrate the high resolution of the camera:

Some of the Earth Observation images acquired by ESA’s OPS-SAT and compressed by FAPEC onboard. In this case, some colour correction (white balancing) was applied.
Photos: ESA/OPS-SAT

 

With this, FAPEC has been the first user non-ESA experiment to be activated onboard OPS-SAT, achieving two important goals. First, it demonstrates the feasibility of uploading and activating new experiments onboard this technology demonstration mission. And second, thanks to the efficient collaboration with ESA and OPS-SAT experts, it can provide an interesting data compression service to other experimenters having to download many images from the satellite.

FAPEC is an excellent data compression solution especially for cubesat-based missions, which use to have strong limitations in onboard computing capabilities and in the downlink. The highly optimized, versatile and portable FAPEC software allows for an agile integration in the payload data handling system while offering a compression throughput comparable to hardware-based solutions. Even satellites with just a low-range onboard computer can use it. Science return of EO missions can significantly be increased in this way.

More information:

Disclaimer: The view expressed herein can in no way be taken to reflect the official opinion of the European Space Agency.

Gaia EDR3 bulk catalogue available in FAPEC format

The Gaia group at the University of Barcelona (IEECICCUB), in cooperation with DAPCOM, has published an alternative copy of the bulk data files from Gaia EDR3 – the Early Data Release 3 from Gaia.

Gaia EDR3 was published yesterday, 3rd December 2020. Besides the on-line catalogue, bulk CSV files were also made available for download – an interesting option for exhaustive analyses. Such files are officially offered in “csv.gz” format, that is, compressed with the widely known gzip compressor.

On 6 February 2019, we released FAPEC Archiver 19.0, our professional data compression software offering high compression ratios at high speeds. One of the options provided is the compression of tabular (CSV-like) text files, such as those from the bulk Gaia EDR3. As a service to the worldwide astronomical community, and also as a demonstration of the capabilities of FAPEC, DAPCOM and the Gaia IEEC/ICCUB Group converted the GaiaSource files from the official Gaia EDR3 bulk CSV repository into the FAPEC format, reducing the total size from 613 GB to 495 GB – that is, 19% smaller than with gzip. Other data compressors like bzip2, rar, Zstandard or 7-zip cannot reach this mark.

You can now download Gaia EDR3 in csv.fapec format here:

     Gaia EDR3 csv.fapec bulk download

The additional tables available in the bulk Gaia EDR3 catalogue will also be converted and published during the coming days.

Free FAPEC decompression licenses can be obtained from our website. Besides, we are preparing a new FAPEC release, including a freely downloadable decompressor with Python bindings.

Have fun!

What is FAPEC?

Here at DAPCOM you have some pages presenting or describing what is FAPEC, such as here, here, in some of these publications, or in this flyer. But to make it short: What is FAPEC? And which are its benefits?

FAPEC is a data compression algorithm implemented as a software application.

That is: we simply reduce the size of your data to reduce the requirements on disk space or transfer time.
There are other solutions like Zip, BZip2, Rar, 7-Zip or (more recently) Zstandard which have been widely used for “generic” data compression since years. There are also other solutions aimed at “specific” types of data, such as JPEG (for images), MP3 (for sound and music) or MPEG4 (for video), some of which may introduce losses in the data to allow compressing better at the cost of a slight quality reduction. In general, there is, actually, a large variety of data compression software.

So what makes FAPEC different?

FAPEC is a staged data compressor:
It is based on a first pre-processing stage, which is adapted to the type of data being compressed, followed by a second entropy coding stage, based on our patented technology, performing a fast statistical analysis to select the most efficient binary codes.
Other solutions simply try to perform an exhaustive search for repeated strings or values, which can be significantly slow. Or they are restricted to a specific type of data, thus not applicable to other types.
With this staged approach, FAPEC is able to efficiently handle a wide variety of data – all in a single, lightweight, fast and multi-platform software program.
You can either let FAPEC detect the most adequate pre-processing stage and options, or you can fix it by yourself.

What are the real benefits?

Users basically care about two indicators: ratio and speed.
In many solutions, there is typically a balance between both: you can either compress better but slower, or compress worse but faster.
By applying the adequate algorithms to your specific data, FAPEC can break this restriction and achieve high compression ratios at high speeds.
What is even more important: by better adapting to your data, and depending on the case, FAPEC can achieve significantly higher ratios than other algorithms.

…more specifically?

These are some of the data types where FAPEC excels:

  • Binary files with time series, such as sensor measurements (temperature, pressure, brightness, energy flux, power…), either as integer or floating-point values
  • Multi-dimensional data (binary values arranged in a table or matrix)
  • Raw multi-band images, such as color pictures, and specially multispectral or hyperspectral imagery
  • Log files, such as those generated by data processing systems
  • Tabular text data, such as CSV files or text files with LIDAR or Point Cloud data
  • Genomics data, such as FastQ and VCF files
  • As an example of a tailored professional stage, watercolumn data files generated from Kongsberg Maritime multibeam echosounders

For some of these cases (such as images), FAPEC offers a lossy option, allowing to slightly reduce the data quality to achieve higher ratios. The default option is a lossless operation.

How much can we gain from FAPEC?

It depends on the specific kind of data, and it can also vary with each file or data block.
In general, at least on the mentioned data types, FAPEC can outperform other solutions like Zip by 10-20%, and in some cases it can even double it.
Speed is also important: even in single-threaded mode, FAPEC typically compresses much faster than other solutions. In some cases, FAPEC can compress 10 times faster than other solutions. Decompression speed is also excellent, exceeding 1 GB/s in some cases.
If you want to know for sure how much can you get from FAPEC, you can simply test it by yourself!

What else does FAPEC offer?

  • Chunk-based operation: if your compressed file gets corrupted, FAPEC will try to recover it, minimizing data loss.
  • Multi-file: you can store over 8 million files or folders in a single FAPEC archive.
  • Multi-thread: do you have a many-core processor? FAPEC can use up to 62 threads for a lightning speed.
  • Encryption: AES-256 (requiring OpenSSL libraries) or our own implementation of the XXTEA algorithm.
  • License-enforced privacy: you can generate FAPEC archives that can only be decompressed with your license.
  • On-the-fly statistics generation: while compressing each file, FAPEC can generate a log file with the partial ratios obtained for each data chunk. Some stages generate additional statistics on the data contents. It offers a kind of digest of the data complexity, allowing to quickly detect some features in the data, for example.
  • DAPCOM support: we will help you to achieve the best compression results on your data. We can also design and implement specific pre-processing stages for your case!

Where can I run FAPEC?

FAPEC is mostly implemented in ANSI C with some POSIX extensions.
You can run it on Linux, Mac OS and Windows; x86 (32 or 64 bits), ARM (32 or 64 bits) or Power PC; Little or Big Endian.
It is lightweight (less than 1MB), and you can run it on low-range computers with slow processors and small RAM. By selecting a small enough chunk size you can run it with less than 1MB of RAM.
FAPEC can actually run in almost any computer, from satellites to supercomputers.

How do I use FAPEC?

You can use it from the command-line, as an executable program. You can invoke it on files or on streams (standard input/output).
It can also be invoked through its C API, invoking it on files or memory buffers. Thus, you can integrate FAPEC in your own software.
We will soon offer the Java API (through JNI) and Python API, as well as the FAPEC integration in HDF5, NetCDF and FITS. We are also preparing a Graphical User Interface, implemented in Java for better portability.

 

Gaia DR2 bulk catalogue available in FAPEC format

The Gaia group at the Universitat de Barcelona (IEECICCUB), in cooperation with DAPCOM, has published an alternative copy of the bulk data files from Gaia DR2 – the second data release from Gaia, where DAPCOM has made significant contributions.

Gaia DR2 was published on 25 April 2018. Besides the on-line catalogue, bulk CSV files were also made available for download – an interesting option for exhaustive analyses. Such files are officially offered in “csv.gz” format, that is, compressed with the widely known gzip compressor.

On 6 February 2019, we released FAPEC Archiver 19.0, our professional data compression software offering high compression ratios at high speeds. One of the options provided is the compression of tabular (CSV-like) text files, such as those from the bulk Gaia DR2. As a demonstration of the capacities of FAPEC, we converted the full Gaia DR2 bulk CSV files to the FAPEC format, reducing the total size from 554 GB to 471 GB – that is, 15% smaller than with gzip. Other data compressors like bzip2, rar, Zstandard or 7-zip cannot reach this mark. Specifically, for the largest tables:

  • gaia_source has been reduced from 548 GB to 466 GB. We have also combined several CSV files into larger FAPEC archives to improve download transfer speeds.
  • gaia_source_with_rv, from 3.1 GB to 2.5 GB.
  • light_curves, from 2.3 GB to 1.9 GB.

You can now download Gaia DR2 in csv.fapec format here:

Gaia DR2 csv.fapec bulk download

There you will also find the scripts used for the gzip-to-fapec conversion, as well as the log files from the process, during which we checked each of the files to make sure no data was lost or corrupted.

Free FAPEC decompression licenses can now be obtained from our website.

Have fun!

 

Release of FAPEC Archiver 19.0

 

FAPEC

FAPEC Archiver 19.0 is out!

Today, DAPCOM has released the new version of our propietary, high-performance, professional, staged data compressor, FAPEC.

This version, called FAPEC Archiver 19.0, is the first public version in the sense that anybody can request and download free decompression or evaluation licenses.

It also includes some exciting improvements with respect to the previous release, such as:

  • New professional stages: FastQ (genomics data), Tabular text data (such as CSV or some LIDAR and point cloud formats), Kongsberg’s water column data.
  • LZW stage and improved FAPECLZ stage for text data, offering excellent ratios and outstanding decompression speeds on log files.
  • On-the-fly generation of basic compression statistics for each data chunk and file, which can be extended to perform quick statistical analyses on the data.
  • Multiple file and directories archival (up to 8 million files or folders), keeping dates and permissions.
  • Multi-threaded operation.
  • AES256 and XXTEA-based encryption.
  • Public API to integrate FAPEC compression or decompression in your software, available in C for now (Java/JNI and Python bindings are in the making).

Get your personal FAPEC copy here!

DAPCOM on TV (La 2, “Tinc una idea”)

DAPCOM appeared yesterday on the TV program “Tinc una idea” (“I’ve got an idea”) of La 2 (around minute 21:55), in catalan.

Watch it online