Xman Drupal

I found that current dos2unix will produce an intermediate file and stop when it encounters a directory in the arguments. And it cant process folders recursively. Hence, I wrote these scripts to help me to convert all text files in a list of files and folders into DOS or UNIX format.

Explore directories and files recursively.

explore()
{
        local file=""
        ls -1 | while read -r file
        do
                # Recursively explore if we found a directory.
                if [ -d "$file" ] ; then
                        echo "Entering $file"
                        pushd "$file" > /dev/null
                        explore
                        echo "Leaving $file"
                        popd > /dev/null
                # Process it if we found a text file.
                elif [ "`${FILE} ${file} | ${GREP} text`" != "" ] ; then
                        # echo "Processing: $file"
                        $DOS2UNIX "${file}"
                fi
        done
} # end of explore()

Process each of the files and folders given in the command line.

for file in $@ ; do
        # FIXME: I should have put these into a subroutine to reduce code duplications in explore().
        # However, I'm not familiar with subroutines in BASH yet.
        if [ -d "$file" ] ; then
                echo "Entering $file"
                pushd "$file" > /dev/null
                explore
                echo "Leaving $file"
                popd
        elif [ "`${FILE} ${file} | ${GREP} text`" != "" ] ; then
                $DOS2UNIX "$file"
        fi
done

For the entire file, see xman_dos2unix in xman utility (latest) under GPL.

Get svn
Get git

Pros of svn:

• Good GUI tortisesvn for running svn commands.
• Partial checkout: Choose any sub-dir to checkout as a working copy.
• Centralized server: Save space by maintaining single global repository.
• Portable to Windows and Linux/Unix.
• File-based tracking: More detailed tracking of files.

Cons of svn:

• Centralized server:
  • When the server/network is down or damaged, you cant access your repository, or loss your repository forever.
  • Potential access conflicts to shared files. Frankly, I dont like people manipulating my repository.
  • Flat checkout: Checkout from remote repository transfers file by file, which can be quite slow through the network, especially dealing with many small files.
• Limited move/rename file: You'll need to commit moved/renamed files before manipulating them.
• No good mechanism for pulling or pushing updates to other repositories.
• .svn folders are everywhere in a working copy. Hate this when I want to copy a folder to somewhere.
• File-based tracking: More overhead in tracking.

Pros of git:

• Good GUIs qgit, citool available for visualizing branches and histories.
• Decentralized: If a server is gone, you have nothing to loss in your local copy.
• Space efficient: Single git repository is typically much smaller than a svn repository.
• Compressed checkout: Checkout from remote repository transfers data in compressed format, hence more network friendly.
• Better ways of file moving/renaming/deleting, but this is subjective.
• Simple branch & merge mechanism, for both local and remote repositories.
• Easy branch switching: Short command to switch a working copy from a branch to another branch (with little CPU), requires no folder duplications too. Compared to svn, you need to know which folder to look for for a particular branch. To create a branch in svn, you basically duplicate entire branching folder.
• Requires no user account. Simply publicize your repository, and it's up to anyone to pull your updates. Not really a feature, but git mode of working supports this well.
• Tree-based tracking: For low tracking overhead.

Cons of git:

• Full checkout: Must checkout entire working copy, as well as the entire repository (including all history).
• Multiple repositories: A full repository for each checkout.
• Lonely merging: I have to resolve conflicts by myself when pulling updates from remote repositories. (In svn, the one causing the conflicts will fix the problems.) You can settle conflicts with the authors if you can command them :).
• Use Linux/Unix File System specific features such as hard link in some cases, arguably a pro feature for space efficiency, but trading off portability. Needs Cygwin for Windows, which results in poor performance.
• Missing file-based tracking: Doesn't track file copy operations. We may not know the source copy of a file. It probably takes more time to obtain file specific information, e.g. history of a file.

Upgrading from single core processor to dual-core processor does not give you double performance for free most of the time. Software written to run single threaded has no idea how to fully utilize a dual-core processor. Fortunately, it is not too difficult to parallelize certain software such as Reversi, because:

1. Large amount of time is spent in small evaluation function.
2. Different evaluations can be computed independently.
3. Many evaluations are to be computed.

Let's see how do I quickly parallelize the Reversi in C++ under GPL license using merely a few lines of OpenMP and minor changes to existing codes.

Following is the code snippet that we want to parallelize, to get multiple iterations computed in parallel.

    for(iter = vmoves.begin() ; iter != vmoves.end() ; iter++) {
      m = *iter ;     
      buffer_b = b ; 
      buffer_b.move_at(m, value) ;      
      s = e.eval(buffer_b, value, (value+1)%2) ; 
      cout << "(" << m.x << "," << m.y << "): " << s << endl ;
      if(s > best_s) {
        best_s = s ;
	best_move = m ;
      }          
    }             

Ooops, we have to be able to compute iter statically (iter points to different possible moves), so that OpenMP can schedule and compute different moves in parallel. So we need to change iter a bit. We have:

DMA:

Prototype: co_memory co_memory_create(const char *name, const char *loc, size_t size);  
// "Name" is used by external application to identify this memory.
// Create 64 bytes memory from the heap. loc is architecture dependent.
co_memory_create("name", "heap0", 64);

Prototype: void co_memory_readblock(co_memory mem, unsigned int offset, void *buf, size_t buffersize);
Prototype: void co_memory_writeblock(co_memory mem, unsigned int offset, void *buf, size_t buffersize);
Prototype: void *co_memory_ptr(co_memory mem); // Return a C pointer to the buffer of the shared memory.

Fixed-Point Arithmetic:
Example of format specification - 1s8.23 => 1 sign bit, 8 bit integer, 23 bit fraction.
We can choose several modes: saturation, floor, ceil.

co_int16 a = (co_int16) FXCONST16(96,7); // a <- 96 in 1s8.7 format.
c = FXADD16(a,b,8); // Add a and b in 1s7.8 format.
c = FXMUL32(a,b,8); // Multiply a and b in 1s7.8 format.
c = FXDIV16(a,b,10); // c <- a/b in 1s6.9 format.

Macros: FXADD8(), FXADD16(), FXADD32(), FXMUL8(), FXMUL16(), FXMUL32(), ...

Sorting is an old topic in Computer Science. With the advent of new processors, how's the sorting performance looked like? Following graphs plot the total running time of inserting elements into the containers and sorting the elements. Different number of elements are experimented.

C++ list is implemented as a linked-list. Without random access capability, we expect the sorting time to be very bad. From the experiment, multiset is even worse. Perhaps, if we must use multiset, we probably can sort the elements in vector, and use range insert to insert into multiset? I observed that list and multiset use much much more amount of memory (> 10x)!!! Hence, running 100M elements with 4GB RAM will make swap space into operations.

Vector is clearly the winner for simple sorting applications. It's even better than standard textbook Quick Sort. :) I shall implement other algorithms or variants to compare with this C++ vector implementation.

Did a benchmark on Yafray, a rendering software, recently on several cases. See the plot below,

Running in 64bit helps quite a bit in the performance. However, it's not clear if it is due to the use of different compilers.

For AMD64 running 64bit, I use GCC 4.1.1 64bit.
For AMD64 running 32bit, the compiler is unknown as I use the rpm provided by yafray.org.
For Pentium IV, I use GCC 4.1.1 32bit.

Overall, it should be fare enough to compare AMD64 64bit and Pentium IV 32bit since they are using the same compiler.

After reading some documents and hacking on the scons .py codes. I found an easy way to compile Yafray in AMD64 FC6.

In the file SConstruct, merely edit the following line

common_env=Environment(ENV=os.environ, CXXFLAGS = config.cxxflags);

to

common_env=Environment(ENV=os.environ, CXXFLAGS = config.cxxflags, SHLINK = "g++", LINK = "g++");

The compilation will then use g++ as the linker with shared library.

If the compilation cannot find the 64bit OpenEXR library, edit the line in file linux-settings.py

def get_libpath(args): return [ exr.PATH + "/lib" ]

to

def get_libpath(args): return [ exr.PATH + "/lib64" ]

Then compile as usual: scons -j 2

Given yafray-0.0.9.tgz:

# Install scons as needed by yafray. Yafray no longer uses GNU make.
$ yum install scons
$ tar zxvf yafray-0.0.9.tgz
$ cd yafray

# Start compilations
$ scons
ld -shared -no_archive -o src/yafraycore/libyafraycore.so src/yafraycore/bound.os src/yafraycore/buffer.os src/yafraycore/yafsystem.os src/yafraycore/tools.os src/yafraycore/camera.os src/yafraycore/color.os src/yafraycore/filter.os src/yafraycore/matrix4.os src/yafraycore/object3d.os src/yafraycore/triangletools.os src/yafraycore/mesh.os src/yafraycore/kdtree.os src/yafraycore/triclip.os src/yafraycore/reference.os src/yafraycore/renderblock.os src/yafraycore/scene.os src/yafraycore/forkedscene.os src/yafraycore/threadedscene.os src/yafraycore/ipc.os src/yafraycore/ccthreads.os src/yafraycore/noise.os src/yafraycore/background.os src/yafraycore/sphere.os src/yafraycore/texture.os src/yafraycore/metashader.os src/yafraycore/targaIO.os src/yafraycore/triangle.os src/yafraycore/vector3d.os src/yafraycore/photon.os src/yafraycore/params.os src/yafraycore/HDR_io.os src/yafraycore/spectrum.os -lpthread
ld: /usr/lib64/libpthread.a(pthread_create.o): relocation R_X86_64_32S against `a local symbol' can not be used when making a shared object; recompile with -fPIC
/usr/lib64/libpthread.a: could not read symbols: Bad value
scons: *** [src/yafraycore/libyafraycore.so] Error 1
scons: building terminated because of errors.

I recompile yafray with -fPIC does not help at all. Replace 'ld' by 'g++' solves the linking problem. But I dont know how to do that using scons! Even if I create libyafraycore.so manually, running scons will still try to ld!

Out of no choice, I use a quick and dirty workaround. Generate the list of commands with dry run option. Replace the 'ld' commands with 'g++'. Then execute the list of commands.

$ scons -n > run
$$ Edit run, replace 'ld ' by 'g++ '. Using vi, run ':%s/ld /g++ '
$$ Remove scons outputs and -no_archive in run if you dont like to see warning outputs.
$ chmod +x run
$ ./run
$ ls src/yafray

I got the 'yafray' executable file created. Installation is not done yet. We still have to install yafray .so library.

# You can install the .so into /usr/lib or /usr/local/lib.
$ mkdir /usr/local/lib/yafray
$ cp src/backgrounds/*.so src/lights/*.so src/shaders/*.so  /usr/local/lib/yafray/
$ cp src/yafraycore/*.so src/interface/*.so /usr/local/lib/
# Remember to add /usr/local/lib into ld.so.conf.d/
$ ldconfig
# You can install yafray into /usr/bin or /usr/local/bin
$ cp src/yafray /usr/local/bin

DONE!!! scons disappointed me.