ThinkMeta LogoDeutsch
English
LAME: Asynchronous file processing

The encoding process of LAME on file level is shown in the following picture:

Dateiverarbeitung in LAME

The uncompressed file is read in blocks and the compressed file is written in blocks. Here, read and write operations alternate.

Although this approach and the use of synchronous I/O operations is not optimal the encoding speed of one single file is not affected.

The process can be described in pseudo code in an imperative programming language as follows:

MP3EncodeFile(wavFilename, mp3Filename)
{
  infile = OpenFile(wavFilename);
  outfile = OpenFile(mp3Filename);
  for (i = 0; i < maxBlocks; ++i) {
    inblock = ReadBlock(infile);
    outblock = EncodeBlock(inblock);
    WriteBlock(outfile, outblock);
  }
}

In general, it's possible to call the function "MP3EncodeFile" in parallel on different threads for several files. This is suitable for two or three threads.
With even more threads, too many I/O operations will compete for a serial data device, so that the effiency losses exceed the benefits of more processors.

Fiber Pool has an I/O scheduler to handle such problems. An arbitrary number of tasks can be passed to it that are processed with the best possible I/O performance, then.

The pseudo code in an asynchronous programming language could be as follows:

MP3EncodeFile(wavFilename, mp3Filename)
{
  Memory inMemory;
  Memory outMemory;
  ReadFile(wavFilename, inMemory);
  WriteFile(mp3Filename, outMemory);
  Encode(inMemory, outMemory);
}

By calling this function the three tasks "ReadFile", "WriteFile" and "En­code" are passed to the underlying system (here: Fiber Pool) for further processing.

When processing several files the function can be called arbitrarily, so that for n files 3n tasks will be passed to the system.

Unlike the imperative solution in which the sight distance of a thread is the currently executed program line and no knowledge about further threads exists, the asynchronous system obtains sufficient information in order to develop strategies on how to use CPU, I/O and memory up to the next synchronization point.

Implementation in fpMP3:
The asynchronous function "MP3EncodeFile" was implemented as C++ class "MP3EncodingTask". The I/O strategy was defined to prefer read operations rather than write operations (see documentation of "SetDefaultFileIOSchedulingStrategy").
The CPU and memory strategy can be set through the command line.