| Blog |
|
| Releases |
| Memory Usage Optimization | |
|---|---|
|
Date: 29/3/2006 | I added an optional memory usage debugger to Lgi over the last 2 days, just in case that Btree memory hog thing turns up again. What it does is override the normal new/delete/malloc/free functions to all go through 2 functions lgi_malloc and lgi_free. These functions add some tracking data onto the start of each block and also keep everything in a big list as each block has a next/prev pointer making it a doubly linked list (easy O(1) insert/delete). So when lgi_malloc is called it allocs memory enough for the data and the stats block. The stats block has the size of the data and a stack trace. The stack trace stores the EIP pointer for the top 8 functions on the call stack. This means later on we can track down who allocated the memory. Then it adds the block to the linked list. lgi_free obviously just removes the block from the list and calls "free" on it.
Thus at any point in the execution you can call LgiDumpMemoryStats which walks the current linked list of memory blocks and dumps it to file "stats.mem". While it does that it also looks up the module and source file/line information for each EIP in the stack trace for the block. Thus the dump file looks like this: Block 65536 bytes Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1434 Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1644 Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1878 Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:1544 Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:817 Block 64 bytes Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1644 Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:1878 Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:1544 Lgid.dll C:\Code\Lgi\src\common\Gdc2\Font\GFontCodePages.cpp:817 Block 272 bytes Lgid.dll C:\Code\Lgi\src\common\General\GContainers.cpp:600 Lgid.dll include\common\GContainers.h:103 Lgid.dll C:\Code\Lgi\src\win32\Gdc2\Gdc2.cpp:2191 Lgid.dll GApp15::GApp15+0x16 Lgid.dll C:\Code\Lgi\src\common\Gdc2\15Bit.cpp:71 So you can see that this is interesting but it's also not very easy to digest. For example the dump from Scribe just after starting up has 140,000 or so blocks taking up 64mb of dump file. Good luck sorting through that! So of course you need something to summerize that into "information" instead of "data". And that tool is built on to LgiIde: 
As you can see this is much easier to digest. You can see some solid stats on where the memory is being allocated and where some optimization is needed. The great thing about having the stack trace is that we can jump over the container classes and get the real point that the allocation happened. I've already identified some areas for improvement in Scribe, in particular the XML parser and the storage sub-system. Initially this system put quite a bit of extra load on the program while it was running but I've managed to get it to a point where you barely notice the performance overhead during runtime, although obviously you use more memory than normal. So it can be left on during normal debugging so that you can at any time dump the current memory usage and analyse it in LgiIde. I intend to also make this functionality able to dump leaked memory at program exit (using the atexit method). The other good thing about this code is that it's not too closely tied to Lgi, it should be possible to separate it out and use it in any software. With a little work. Update: atexit is what I was looking for getting the memory leak detection going. |
| (1) Comment | Add Comment | |
| Scribe v1.89 Test1 Release Notes | |
|---|---|
|
Date: 28/3/2006 | Well it's done. I'm not making any promises on code quality or memory usage though. I've seen some problems in the few days of testing the final codebase but no bugs, just optimization issues. However thats mostly tied to the use of the new incremental bayesian word database updates. This feature is off by default, due to concerns about speed but you can turn it on in the bayesian settings dialog. It updates the word databases on the fly as you read your email and bin stuff as spam.
It's a little slow at times on older hardware, on my 1.4ghz/512mb machine it gets in the way of usability but on the 3.06ghz/2gb machine it's not noticable at all. The backend of the word DB's is currently a Btree but I'm thinking of playing with Sqlite as the backend and see what the performance is like. Sometimes it uses upwards 40mb of memory too, but I havn't reproduced this yet. Usually it uses about 3-4mb. Let me know your experiences with the new bayesian filter. You need to rebuild your word lists at least once with v1.89 to initialize the new format files, and then either use the incremental updates or the regular way of rebuilding the lists every so often. There is also a number of API changes that have bubbled changes throughout the code base, and while it's been running well for me in the last month it could mean that rarely used functions are still broken. It's a monumental task to try every menu option, command, filter condition/action and type of email and connection available in the software. So I'm relying somewhat on the user base for feedback. I ripped out the old XML parser which has a side effect of meaning the format of the lgi.conf has changed a little. All the lgi.conf options that had a "." in the element name should be converted to "-", so "font.fixed" becomes "font-fixed" etc. Looking forward to your comments. |
| (8) Comments | Add Comment | |
| PayPal Robots | |
|---|---|
|
Date: 27/3/2006 | PayPal seem to be employing robots to do their support work. Their menus are displaying incorrectly in Firefox 1.5.0.1 so I thought I'd be a good net citizen and report the problem to them. Firstly there is no section in the support options for "website". Which was the first ominous sign that this wasn't going to go well. So I send off a brief but polite description of the problem and hope for the best:
Hello my name is Brian, I will be happy to assist you with your question. In review you will want to clear cache and cookies and log in using www.paypal.com.au. If you are experiencing problems with our website that have not been encountered before, it is likely that you have a corrupted cookie. The easiest way to clear cookies is to remove them from the PC rather than the browser. Follow these simple steps to clear your cookies: [snip] Yeah ok, just to placate you I'll do all that. So I clear the cache and all the cookies and re-login to PayPal. Unsurprisingly the same problem appears and I email them back: I've cleared my cache and cookies and logged in again to www.paypal.com.au and I'm still getting the same issue. Attached is a screen shot of the problem. regards -- Matthew Then I get this back from them: Thank you for contacting PayPal. I was unable to open the attachment you had sent due to security precautions. Please contact PayPal in text form at https://www.paypal.com.au/wf/f=default and we will be happy to assist you further. Thank you for being part of the PayPal community. Community satisfaction and your experience with PayPal are very important to me. You may receive a survey from our third party vendor, Benchmark Portal, about the service you received. Awesome! I will club them to death with my PNG of malware! It was a PNG image file... it's not going to bite. PNG files can't have virus' or malicious payloads. Screw this, if you don't want help fixing your own damn site then fine I can spend my time elsewhere. -- Matthew That reset the robot: Hello my name is Mahak, I will be happy to assist you with your question regarding account problem. I do apologize for the inconvenience that you are experiencing with this situation. To better assist you please be specific about any one of your transaction that you are looking for. Thank you for being part of the PayPal community. Community satisfaction and your experience with PayPal are very important to me. You may receive a survey from our third party vendor, Benchmark Portal, about the service you received. Haha. So I guess I should get used to the bustedness of PayPal.com.au, all I have to do is repeat to myself "I do not care. I do not care. I do not care." |
| (0) Comments | Add Comment | |
| Scribe | |
|---|---|
|
Date: 23/3/2006 | The bug in the Btree code that I was using valgrind to find wasn't there. It was merely some old debugging code that was getting in the way. Doh!
So now there is no reason not to release v1.89-test1. I just need to clean up some bits and peices and we're good to go. The effort spent on valgrinding the bayesian code did turn up some bugs which I've fixed. So it wasn't a fruitless waste of time. But I'm sure it would've been nicer to have a release out last week eh? |
| (1) Comment | Add Comment | |
| Cygwin/GCC | |
|---|---|
|
Date: 22/3/2006 | Lgi now builds and runs on cygwin with GCC. I tested this by building the IDE with Visual C++ and then loading the XML project files that LgiIde uses, and building it again with gcc. After a while I got it to a point where the GCC version doesn't crash. It's not fully tested but at least something is working. Most of the issues were with using mis-matched DLL's, ie some that were compiled with VC and some were GCC. This obviously "Does Not Work(TM)".
A release of Lgi will be out shortly. |
| (0) Comments | Add Comment | |
| Linux/Lgi | |
|---|---|
|
Date: 18/3/2006 | It builds! After several months of inactivity, I used Kubuntu under Vmware to get Lgi + Scribe building again, mostly cause I need to valgrind the new Btree/Bayesian filtering code but nice to have it ready for the first v1.89 release "any time now". |
| (0) Comments | Add Comment | |