Performance improvement after code refactoring

I took over an existing project recently. Unfortunately, the code is difficult to understand and maintain: the code was written in procedural with many duplicated codes. It’s really painful to add new features, even modify a simple testing condition, I have to scroll up and down in a single code file with more than thousands of lines. So the primary objctive of refactoring is to improve the code design, in order to make it more easier to understand and maintain. I spent a few days to analyze the existing code, and rewritten the code in object-oriented paradigm.

After refactoring, I tested and compared the performance of both codes. The result is encouraging, the new code is running more stable and has a overall improvement in executing time. For this project is not providing simple web page services, the executing time highly depends on the specific user request context, so it’s difficult to write testing cases for benchmark. For this reason, I extracted the statistical executing time from real running environment for both version of code.

I will call the old version code and the new version code as V1 and V2 respectively. Following data and plots show the response times for server running at relatively low and high load. Every set of data was generated in one hour.

Low load situation

V1 response time

Resp. Time Range(ms) Resp Count Percentage Accu. Per.
0-10      334,970 4.72% 4.72%
10-20    1,383,553 19.48% 24.19%
20-30   3,511,391 49.43% 73.62%
30-40    1,534,209 21.60% 95.22%
40-50      129,206 1.82% 97.04%
50-60       45,071 0.63% 97.67%
60-70       30,191 0.42% 98.10%
70-80       15,793 0.22% 98.32%
80-90        8,785 0.12% 98.44%
90-100       6,728 0.09% 98.54%
100-110        5,266 0.07% 98.61%
110-120        4,149 0.06% 98.67%
120-130        3,402 0.05% 98.72%
130-140        3,135 0.04% 98.76%
140-150        3,155 0.04% 98.81%
150-160        3,116 0.04% 98.85%
160-170        2,908 0.04% 98.89%
170-180        3,280 0.05% 98.94%
180-190        3,961 0.06% 98.99%
190-200        4,786 0.07% 99.06%
200-300       35,690 0.50% 99.56%
300-400       17,350 0.24% 99.81%
400-500        1,630 0.02% 99.83%
500-600        1,028 0.01% 99.84%
600-700        1,187 0.02% 99.86%
700-800          623 0.01% 99.87%
800-900          150 0.00% 99.87%
900-1000           46 0.00% 99.87%
1000+        9,080 0.13% 100.00%

16-v1

16-v1-accu

V2 response time

Resp. Time Range(ms) Resp Count Percentage Accu. Per.
0-10 20,258 0.28% 0.28%
10-20 4,475,788 61.01% 61.29%
20-30 2,623,493 35.76% 97.05%
30-40 76,568 1.04% 98.09%
40-50 30,340 0.41% 98.51%
50-60 12,895 0.18% 98.68%
60-70 5,498 0.07% 98.76%
70-80 5,129 0.07% 98.83%
80-90 8,560 0.12% 98.94%
90-100 10,141 0.14% 99.08%
100-110 9,741 0.13% 99.21%
110-120 7,714 0.11% 99.32%
120-130 5,519 0.08% 99.39%
130-140 3,668 0.05% 99.44%
140-150 2,297 0.03% 99.48%
150-160 1,615 0.02% 99.50%
160-170 1,176 0.02% 99.51%
170-180 918 0.01% 99.53%
180-190 666 0.01% 99.53%
190-200 489 0.01% 99.54%
200-300 14,112 0.19% 99.73%
300-400 18,831 0.26% 99.99%
400-500 131 0.00% 99.99%
500-600 126 0.00% 99.99%
600-700 77 0.00% 99.99%
700-800 27 0.00% 100.00%
800-900 25 0.00% 100.00%
900-1000 30 0.00% 100.00%
1000+ 288 0.00% 100.00%

16-v2

16-v2-accu

High load situation

V1 response time

Resp. Time Range(ms) Resp Count Percentage Accu. Per.
0-10 125,181 0.99% 0.99%
10-20 2,003,708 15.83% 16.82%
20-30 766,219 6.05% 22.87%
30-40 2,759,891 21.80% 44.68%
40-50 2,793,375 22.07% 66.74%
50-60 1,460,567 11.54% 78.28%
60-70 711,881 5.62% 83.91%
70-80 351,485 2.78% 86.68%
80-90 174,165 1.38% 88.06%
90-100 88,230 0.70% 88.76%
100-110 48,826 0.39% 89.14%
110-120 30,767 0.24% 89.39%
120-130 20,802 0.16% 89.55%
130-140 15,563 0.12% 89.67%
140-150 12,479 0.10% 89.77%
150-160 10,540 0.08% 89.86%
160-170 9,458 0.07% 89.93%
170-180 8,818 0.07% 90.00%
180-190 8,447 0.07% 90.07%
190-200 8,763 0.07% 90.14%
200-300 78,584 0.62% 90.76%
300-400 49,607 0.39% 91.15%
400-500 12,389 0.10% 91.25%
500-600 9,764 0.08% 91.32%
600-700 10,466 0.08% 91.41%
700-800 8,982 0.07% 91.48%
800-900 7,357 0.06% 91.54%
900-1000 5,113 0.04% 91.58%
1000+ 1,066,330 8.42% 100.00%

22-v1

22-v1-accu

V2 response time

Resp. Time Range(ms) Resp Count Percentage Accu. Per.
0-10 11,619 0.09% 0.09%
10-20 3,423,981 27.37% 27.46%
20-30 7,841,554 62.68% 90.14%
30-40 862,265 6.89% 97.03%
40-50 119,273 0.95% 97.99%
50-60 54,839 0.44% 98.43%
60-70 23,996 0.19% 98.62%
70-80 12,736 0.10% 98.72%
80-90 13,441 0.11% 98.83%
90-100 17,846 0.14% 98.97%
100-110 17,427 0.14% 99.11%
110-120 15,161 0.12% 99.23%
120-130 11,330 0.09% 99.32%
130-140 7,900 0.06% 99.38%
140-150 5,229 0.04% 99.43%
150-160 3,720 0.03% 99.46%
160-170 2,581 0.02% 99.48%
170-180 1,994 0.02% 99.49%
180-190 1,592 0.01% 99.51%
190-200 1,210 0.01% 99.51%
200-300 31,230 0.25% 99.76%
300-400 25,771 0.21% 99.97%
400-500 1,069 0.01% 99.98%
500-600 575 0.00% 99.98%
600-700 164 0.00% 99.98%
700-800 64 0.00% 99.99%
800-900 19 0.00% 99.99%
900-1000 1 0.00% 99.99%
1000+ 1,807 0.01% 100.00%

22-v2

22-v2-accu

The plots show the V1 was fluctuant at high load situation, and V2 keep consistent at both situation.

This post is just a experience note, not intended to compare procedural and OOP, both paradigm can be used to write excellent code, but I personally prefer OOP and believe it’s easier to write maintainable code.

Order problem of awk foreach

In PHP, the order the elements of array are always displayed as it created when traverse it with foreach, but in AWK it’s not the case, it seems in random order.

AWK version of foeach:

for (var in array)
  body

For example, when I run following awk script, the data display in random order, but not from 1 to 11.

awk '{i=1; while(i<=NF){arr[i]+=$i;i+=1}}END{for(i in arr){printf("%d=%d\n", i,arr[i]);}}' demo_data.txt
4=78913294
5=72533221
6=151446515
7=5690950
8=7163155
9=12854105
10=600865293
11=1071720068
1=193122485
2=113431670
3=306554155

Use following method instead in order to avoid confusing if the order is required:

for(i=1; i <= length(ARRAY); i++) 
    body

demo_data.txt

9632108 5659241 15291349 3937536 3620112 7557648 283966 355788 639754 29978495 53467246 
9657403 5669637 15327040 3946110 3625140 7571250 284820 358214 643034 30050377 53591701 
9655299 5674226 15329525 3945152 3626141 7571293 284555 358477 643032 30048665 53592515 
9658195 5671810 15330005 3950449 3629763 7580212 284571 357575 642146 30039707 53592070 
9655504 5675401 15330905 3944973 3623397 7568370 285966 358200 644166 30049206 53592647 
9658533 5674614 15333147 3941120 3630867 7571987 284417 358763 643180 30043939 53592253 
9663355 5670377 15333732 3945271 3627271 7572542 284767 357710 642477 30043298 53592049 
9661154 5674127 15335281 3946795 3626219 7573014 284636 358345 642981 30040659 53591935 
9655119 5670961 15326080 3947739 3627210 7574949 285055 358086 643141 30048501 53592671 
9654181 5668588 15322769 3947021 3627608 7574629 284603 358980 643583 30050409 53591390 
9664004 5672546 15336550 3946492 3627060 7573552 285867 357617 643484 30038455 53592041 
9653571 5670797 15324368 3945694 3628555 7574249 283991 357938 641929 30051367 53591913 
9655395 5671722 15327117 3946222 3624930 7571152 283839 358734 642573 30051274 53592116 
9655780 5672003 15327783 3942720 3625895 7568615 285084 357638 642722 30053798 53592918 
9655786 5675341 15331127 3948867 3626483 7575350 284089 358135 642224 30044271 53592972 
9655718 5672592 15328310 3946863 3629646 7576509 284260 359655 643915 30044217 53592951 
9652834 5673940 15326774 3949866 3624831 7574697 284791 357703 642494 30048256 53592221 
9656857 5670560 15327417 3947632 3629071 7576703 284152 358580 642732 30045527 53592379 
9665378 5668883 15334261 3941470 3625433 7566903 283750 358483 642233 30048808 53592205 
9656311 5674304 15330615 3945302 3627589 7572891 283771 358534 642305 30046064 53591875

Run custom shell scripts in Android

It’s not secure and inconvenient to do some low-level testing task depending on third party APPs. Shell is especially useful for such purpose.

1. Install Android Debug Bridge(ADB)

Download Android SDK and run SDK Manager to install the platform-tools, as following image shows:

android-adb

2. Connect to android device with ADB

In order to push file into /system file system, root mount is required, so run following commands:

$ adb root
$ adb remount

If no error occurred, you should see “remount succeeded” on terminal.

3. Write shell scripts

It’s almost the same as writing general shell scripts for Linux except the first line, for the android shell is located at /system/bin directory.

#!/system/bin/sh

# DO SOMETHING

4. Push script to android

Push the script, test_script for example, to android:

$ adb push /path/to/test_script /system/xbin/test_script

And change permission:

$ adb shell chmod 6755 /system/xbin/test_scripts

5. Run from Android

If you wanna run the scripts on Android, you’d better install Android Terminal Emulator.

Windows batch script to start VMWare guests

The GUI of VMWare Workstation seemed quite slow and always run into “not responding” state when I try to boot a guest. I tried the vmrun command and it works very smoothly, so I wrote a simple batch script which can be used to run a guest automatically on windows start.

@echo off

if "%1" == "" (
    set vm_name=X:\\default\\path\\to\\linux.vmx
) else (
    set vm_name=%1
)

vmrun list | findstr /E "%vm_name%" 1>NUL

if errorlevel 1 (
    echo Starting %vm_name% ...
    vmrun -T ws start %vm_name% nogui
) else (
    echo %vm_name% is running
)

pause

Windows batch script is rarely used (at least for me) so I make a note here in case I or other guys need such script in the future.

CMD reference: http://ss64.com/nt/.

Use fsockopen to read chunked page

PHP provides many different ways to download a page(file) through HTTP, the simplest way is using file_get_contents function which is suitable for relatively small files. If try to download large file with file_get_contents the PHP allowed memory (configured by memory_limit directive) may be exhausted with a fatal error. What’s why we need a portable way to deal with large file, where fsockopen comes in.

There are two main conditions we should consider: Content-Lenght specified or Chunked data.

0. Initiate socket connection

Use fsockopen() to initiate the socket connection. You’d better specify the error number, error message and timeout parameters, and process the error if exists.

$url = 'http://test.example.com/fetch_file.php?file=testfile.iso';

if (preg_match_all('#http://([^/]+)(/.+)#i', $url, $matches)) {
    $host = $matches[1][0];
    $path = $matches[2][0];
} else {
    die('Invalid URl');
}

$fp = fsockopen($host, 80, $errno, $error, 30);

//Open a file pointer for write
$wfp = fopen('file-write-to', 'w');

//specify the block size to read
$readBlockSize = 512;

1. Content-Length

If the Content-Length is specified by HTTP response header, the reading is straightforward just as reading general files.

Snippet use to read response body:

$data = fread($fp, $readBlockSize);
fwrite($wfp, $data);

2. Chunked

For chuncked encoding, there is a different data format, here is a quotation from WikiPedia:

Each chunk starts with the number of octets of the data it embeds expressed in ASCII followed by optional parameters (chunk extension) and a terminating CRLF sequence, followed by the chunk data. The chunk is terminated by CRLF. If chunk extensions are provided, the chunk size is terminated by a semicolon followed with the extension name and an optional equal sign and value.

The last-chunk is a regular chunk, with the exception that its length is zero.

The encoded data looks like this:

4
Wiki
5
pedia
E
 in

chunks.
0

So we have to address the response chunk by chunk. Snippet to do so:

if ($chunk_length === false) {
    $data = trim(fgets($fp, 128));
    $chunk_length = hexdec($data);
} else if ($chunk_length > 0) {
    $read_length = $chunk_length > $readBlockSize ? $readBlockSize : $chunk_length;
    $chunk_length -= $read_length;
    $data = fread($fp, $read_length);
    fwrite($wfp, $data);
    if ($chunk_length <= 0) {
        fseek($fp, 2, SEEK_CUR);
        $chunk_length = false;
    }
} else {
     break;
}

The full script:


 * @copyright (C) 2013 James Tang.
 */

set_time_limit(600);
ignore_user_abort(true);

//$url = 'http://test.example.com/fetch_file.php?file=testfile.iso';
//$saveToFile = 'tmp.iso';
$url = 'http://test.example.com/fetch_file.php?file=tmp.gz';
$saveToFile = 'tmp.gz';

if (preg_match_all('#http://([^/]+)(/.+)#i', $url, $matches)) {
    $host = $matches[1][0];
    $path = $matches[2][0];
} else {
    die('Invalid URl');
}

$fp = fsockopen($host, 80, $errno, $error, 30);
$readBlockSize = 512;

if ($fp) {

    $wfp = fopen($saveToFile, 'w');

    if ($wfp) {
        $request = "GET $path HTTP/1.1\r\n";
        $request .= "Host: $host\r\n";
        $request .= "Connection: close\r\n";
        $request .= "User-Agent: php-download/1.0\r\n";
        $request .= "\r\n";

        fwrite($fp, $request);

        $body_start = false;
        $md5sum = '';
        $content_length = false;
        $chunk_length = false;

        $startLine = fgets($fp, 128);

        if ($startLine && preg_match('#^HTTP/1.\d?\s+200\s+#', $startLine)) {
            while (!feof($fp)) {
                if (!$body_start) {
                    $header = fgets($fp, 128);
                    echo $header;
                    $colon_pos = strpos($header, ':');
                    $header_name = strtolower(trim(substr($header, 0, $colon_pos)));
                    $header_value = trim(substr($header, $colon_pos+1)); 
                    if ($header_name == 'content-md5') {
                        $md5sum = bin2hex(base64_decode($header_value));
                    } else if ($header_name == 'content-length') {
                        $content_length = (int) $header_value;
                    }
                    if ($header == "\r\n") {
                        $body_start = true;
                        echo "Reading data...\n";
                    }
                } else {

                    if ($content_length !== false && $content_length > 0) {
                        $data = fread($fp, $readBlockSize);
                        fwrite($wfp, $data);
                    } else {
                        if ($chunk_length === false) {
                            $data = trim(fgets($fp, 128));
                            $chunk_length = hexdec($data);
                        } else if ($chunk_length > 0) {
                            $read_length = $chunk_length > $readBlockSize ? $readBlockSize : $chunk_length;
                            $chunk_length -= $read_length;
                            $data = fread($fp, $read_length);
                            fwrite($wfp, $data);
                            if ($chunk_length <= 0) {
                                fseek($fp, 2, SEEK_CUR);
                                $chunk_length = false;
                            }
                        } else {
                            break;
                        }
                    }
                }
            }
        } else {
            echo "Failed to read data: " . $startLine . "\n";
        }

        fclose($wfp);
        if ($md5sum && strlen($md5sum) > 0) {
            $md5sum_check = bin2hex(md5_file($saveToFile, true));
            if ($md5sum_check != $md5sum) {
                echo 'MD5 checksum does not match: ' . $md5sum_check . "\n";
            } else {
                echo "MD5 checksum match\n";
            }
        } else {
            echo "No MD5 checksum detected\n";
        }
        //unlink($saveToFile);
    }

    fclose($fp);
} else {
    echo 'Error: ' . $errno . '#' . $error . "
\n"; }

3. Problems

The $readBlockSize value is critical, if too large it may cause problem. When I test on remote server with $readBlockSize=4096, the downloaded file was not identical to source file. This problem must be caused by transfer rate: when you try to read 4096 bytes from the response body, but if less than 4096 bytes was prepared, then the reading sequence is disrupted. At last I found 512 works fine for me.

4. Reference

1. http://en.wikipedia.org/wiki/Chunked_transfer_encoding

2. http://tools.ietf.org/html/rfc2616#page-118