| Blog |
|
| Releases |
| Site Traffic Spike | |
|---|---|
|
Date: 1/4/2006 | Traffic was up 22% this month over the previous busiest month. But it wasn't only just popular with regular visitors, but it was popular with the bots as well. With robots now accounting for over half of all hits on this site. Maybe in 10 years time no-one will read memecode except millions of automated software. |
| (0) Comments | Add Comment | |
| Memory Leak Removal | |
|---|---|
|
Date: 1/4/2006 | With the code I talked about in the previous blog entry I've been able to start removing lots of memory leaks from Scribe. I've added the leak check dumping that I talked about. It's so easy to use I'm just amazed and it's so low impact on the software during use. It was inspired somewhat by valgrind's leak check tool, but this is (much?) better in that it's runtime impact is much lower than valgrind. Of course valgrind has memcheck, which I can't compete with.
However there is some work ahead, because to clear the app of leaks you have to exercise the code over and over to get coverage, i.e. all of the paths through the code getting executed. But now that the tools are so good, I'll leave it running for the next few weeks and get the codebase nice and tight with memory. I also want to reduce the block count, as lots of small blocks of memory are inefficent, in that each block has a number of bytes in overhead and takes time to allocate and free. And beyond that it also slows application shutdown because if the app is swapped out to disk and you shut it down, lots of small memory blocks take much longer to free than a few large blocks. Each block has to be read/written to free it, and if it has to be loaded from virtual memory first it takes longer. Yeah? The number one offender in the billions of tiny blocks of memory is the XML parser. So that'll get some string pooling options like the hash table. |
| (1) Comment | Add Comment | |
| Memory Usage Optimization | |
|---|---|
|
Date: 29/3/2006 | I added an optional memory usage debugger to Lgi over the last 2 days, just in case that Btree memory hog thing turns up again. What it does is override the normal new/delete/malloc/free functions to all go through 2 functions lgi_malloc and lgi_free. These functions add some tracking data onto the start of each block and also keep everything in a big list as each block has a next/prev pointer making it a doubly linked list (easy O(1) insert/delete). So when lgi_malloc is called it allocs memory enough for the data and the stats block. The stats block has the size of the data and a stack trace. The stack trace stores the EIP pointer for the top 8 functions on the call stack. This means later on we can track down who allocated the memory. Then it adds the block to the linked list. lgi_free obviously just removes the block from the list and calls "free" on it.
Thus at any point in the execution you can call LgiDumpMemoryStats which walks the current linked list of memory blocks and dumps it to file "stats.mem". While it does that it also looks up the module and source file/line information for each EIP in the stack trace for the block. Thus the dump file looks like this: Block 65536 bytes Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1434 Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1644 Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1878 Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:1544 Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:817 Block 64 bytes Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1644 Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1878 Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:1544 Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:817 Block 272 bytes Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:600 Lgid.dll include\common\GContainers.h:103 Lgid.dll C:\Code\Lgi\src\win32\Gdc2\Gdc2.cpp:2191 Lgid.dll GApp15::GApp15+0x16 Lgid.dll C:\Code\Lgi\src\common\Gdc2\15Bit.cpp:71 So you can see that this is interesting but it's also not very easy to digest. For example the dump from Scribe just after starting up has 140,000 or so blocks taking up 64mb of dump file. Good luck sorting through that! So of course you need something to summerize that into "information" instead of "data". And that tool is built on to LgiIde: 
As you can see this is much easier to digest. You can see some solid stats on where the memory is being allocated and where some optimization is needed. The great thing about having the stack trace is that we can jump over the container classes and get the real point that the allocation happened. I've already identified some areas for improvement in Scribe, in particular the XML parser and the storage sub-system. Initially this system put quite a bit of extra load on the program while it was running but I've managed to get it to a point where you barely notice the performance overhead during runtime, although obviously you use more memory than normal. So it can be left on during normal debugging so that you can at any time dump the current memory usage and analyse it in LgiIde. I intend to also make this functionality able to dump leaked memory at program exit (using the atexit method). The other good thing about this code is that it's not too closely tied to Lgi, it should be possible to separate it out and use it in any software. With a little work. Update: atexit is what I was looking for getting the memory leak detection going. |
| (1) Comment | Add Comment | |
| Scribe v1.89 Test1 Release Notes | |
|---|---|
|
Date: 28/3/2006 | Well it's done. I'm not making any promises on code quality or memory usage though. I've seen some problems in the few days of testing the final codebase but no bugs, just optimization issues. However thats mostly tied to the use of the new incremental bayesian word database updates. This feature is off by default, due to concerns about speed but you can turn it on in the bayesian settings dialog. It updates the word databases on the fly as you read your email and bin stuff as spam.
It's a little slow at times on older hardware, on my 1.4ghz/512mb machine it gets in the way of usability but on the 3.06ghz/2gb machine it's not noticable at all. The backend of the word DB's is currently a Btree but I'm thinking of playing with Sqlite as the backend and see what the performance is like. Sometimes it uses upwards 40mb of memory too, but I havn't reproduced this yet. Usually it uses about 3-4mb. Let me know your experiences with the new bayesian filter. You need to rebuild your word lists at least once with v1.89 to initialize the new format files, and then either use the incremental updates or the regular way of rebuilding the lists every so often. There is also a number of API changes that have bubbled changes throughout the code base, and while it's been running well for me in the last month it could mean that rarely used functions are still broken. It's a monumental task to try every menu option, command, filter condition/action and type of email and connection available in the software. So I'm relying somewhat on the user base for feedback. I ripped out the old XML parser which has a side effect of meaning the format of the lgi.conf has changed a little. All the lgi.conf options that had a "." in the element name should be converted to "-", so "font.fixed" becomes "font-fixed" etc. Looking forward to your comments. |
| (8) Comments | Add Comment | |
| PayPal Robots | |
|---|---|
|
Date: 27/3/2006 | PayPal seem to be employing robots to do their support work. Their menus are displaying incorrectly in Firefox 1.5.0.1 so I thought I'd be a good net citizen and report the problem to them. Firstly there is no section in the support options for "website". Which was the first ominous sign that this wasn't going to go well. So I send off a brief but polite description of the problem and hope for the best:
Hello my name is Brian, I will be happy to assist you with your question. In review you will want to clear cache and cookies and log in using www.paypal.com.au. If you are experiencing problems with our website that have not been encountered before, it is likely that you have a corrupted cookie. The easiest way to clear cookies is to remove them from the PC rather than the browser. Follow these simple steps to clear your cookies: [snip] Yeah ok, just to placate you I'll do all that. So I clear the cache and all the cookies and re-login to PayPal. Unsurprisingly the same problem appears and I email them back: I've cleared my cache and cookies and logged in again to www.paypal.com.au and I'm still getting the same issue. Attached is a screen shot of the problem. regards -- Matthew Then I get this back from them: Thank you for contacting PayPal. I was unable to open the attachment you had sent due to security precautions. Please contact PayPal in text form at https://www.paypal.com.au/wf/f=default and we will be happy to assist you further. Thank you for being part of the PayPal community. Community satisfaction and your experience with PayPal are very important to me. You may receive a survey from our third party vendor, Benchmark Portal, about the service you received. Awesome! I will club them to death with my PNG of malware! It was a PNG image file... it's not going to bite. PNG files can't have virus' or malicious payloads. Screw this, if you don't want help fixing your own damn site then fine I can spend my time elsewhere. -- Matthew That reset the robot: Hello my name is Mahak, I will be happy to assist you with your question regarding account problem. I do apologize for the inconvenience that you are experiencing with this situation. To better assist you please be specific about any one of your transaction that you are looking for. Thank you for being part of the PayPal community. Community satisfaction and your experience with PayPal are very important to me. You may receive a survey from our third party vendor, Benchmark Portal, about the service you received. Haha. So I guess I should get used to the bustedness of PayPal.com.au, all I have to do is repeat to myself "I do not care. I do not care. I do not care." |
| (0) Comments | Add Comment | |
| Scribe | |
|---|---|
|
Date: 23/3/2006 | The bug in the Btree code that I was using valgrind to find wasn't there. It was merely some old debugging code that was getting in the way. Doh!
So now there is no reason not to release v1.89-test1. I just need to clean up some bits and peices and we're good to go. The effort spent on valgrinding the bayesian code did turn up some bugs which I've fixed. So it wasn't a fruitless waste of time. But I'm sure it would've been nicer to have a release out last week eh? |
| (1) Comment | Add Comment | |