Blog
Page: 0 ... 5 ... 10 ... 15 ... 20 21 22 23 24 25 26 27 28 29 ... 30 ... 35 ... 40 ... 45 ... 50 ... 55 ... 60 ... 65 ... 70 ... 75 ... 80 ... 85
Temp files vs in memory.
Date: 28/11/2008
Most application programmers will eventually come to a point where they need to choose whether to store some data in memory or whether to write it to a temporary file. In these sorts of cases you generally don't know how much data you have to handle so the safe thing to do it to write it to disk. However that has it's downsides, it's slower, you have to make sure you clean up the file afterwards and it can get intercepted by all sorts of other processes like anti-virus.

I'm currently working on the code that handles email download in Scribe v2 and basically there needs to be a container that stores the email as it's streaming in over the socket. Initially I just used a standard in memory container (Lgi object GStringPipe) which is a FIFO sort of thing that puts it's data in fixed sized blocks. Which is kinda cool because it doesn't have to re-allocate a single block of memory to "grow" it when it runs out of space. It just allocates another block and writes to that. Anyway thats fine for about 99.8% of email. Then there is the edge case. The really insanely large email... 111mb? 456mb? I've seen some whoppers. You usually don't see those on the internet, but in a LAN situation it's not unreasonable to email a large file to someone. You know they are not particularly bandwidth limited. And I've always made it a point to handle that gracefully, both for sending and receiving. In the receiving case I stream into memory until I hit 4mb, then I create a temp file and write all the data in memory to disk, and continue streaming remaining data straight to disk. At the end of the email download I parse it in the worker thread. The cool thing about the MIME parser is that it accepts a generic stream class as input and I can give it a memory stream or a disk stream without the MIME parser caring where the data is stored. The graceful part of that system is that if you have a large email, it's not hanging around in memory, or worse virtual memory, making your system slow down.

I have had bugs before where I wrote an email to disk and for a moment closed the file, then a second later attempted to open it and it's gone. Deleted. This happens when some zealous anti-virus process decides that the email is infected. So I do take writing email to disk seriously, and in general I choose to keep the file handle open to avoid losing the data. I'm not particularly worried about infected email considering Scribe's fairly good at detecting executable attachments and blocking or deleting them.

The 4mb cut over limit is somewhat arbitrary. But I think it covers the normative case nicely.
(0) Comments | Add Comment

Scribe v2
Date: 26/11/2008
So over the last week or so I've brought online a new mail storage back end that I'm calling "Mail3". There are already some release notes in the v2 history page about it but I thought I'd talk a little bit about it in the blog. Basically a whole bunch of things are happening in the v2 code base all at the same time.

Firstly I'm really hammering on the memory leaks, trying to get them under control. I've fixed a lot already but the new code that I've written this week still leaks a fair bit. However the leak detection and fixing is working really well so it's almost fun.

Secondly I've finally rid myself of the hideous mime parser from Scribe v1.xx (and early alphas/betas of v2.00). Basically it sat inside the Mail object class and was well not well written. It's probably some of the oldest code in Scribe, dating back to when I sucked more at coding. Anyway I wrote a new MIME parser back in 2006 or something but couldn't work out how to integrate it without breaking everything so it stayed unused for years. Anyway I though while I was doing major earthworks in the code I'd blow the old parser away at the same time. So I did that and moved the ugly MIME flattening code into the Mail2 back end where it belongs.

Thirdly, Mail3. What is Mail3? Well Mail3 is the new mail store back end that I'm working on as the default mail store for Scribe v2. It's primarily a Sqlite database in a folder. I've chosen Sqlite for several reasons; it's small, it's cross platform (both the data format and the code), it's reasonably fast/robust and finally it doesn't have a restrictive license. Also you can also upgrade the format of objects (tables) easily enough. So it fulfills all the requirements for a new mail store. I may still choose to store large attachments outside the Sqlite database, I'm some what undecided on that at this point. Hence having the database in a folder.

So the Mail3 back end is working, you can create the mail store, manage the folder hierarchy, store mail into it etc. But the code doesn't support other types of objects yet, just folders and mail. One of the great things about Mail3 is that finally I'll have a 1:1 mapping between an incoming rfc822 email "image" and how it's stored in the database. So I'll be able to write out an exact (or functionally the same) version of a received email. Which is something that the Mail2 back end couldn't do because it munged the MIME segment hierarchy into a flat list of attachments. The Mail3 back end stores the MIME segments as rows in a table, with ParentId pointers to reconstruct the tree of segments.

Forthly, I've moved the parsing of email from the GUI thread into the worker thread doing the email download.



This should reduce the amount of GUI lockup during mail downloads.

Oh and lastly, the current code base can load more than one mail store at the same time. For instance you can have a mail2 file and a mail3 database open at the same time and they both appear in the folder tree as separate hierarchies of folders. There is only one problem with that, and that is the addressing of folders via paths is going to have to change. At the moment I'm not sure how that is going to play out but basically now that you can have multiple folders loaded things like filters and so on that reference a folder by it's path need to pre-pend a prefix on to the path to define which mail store the folder is in.

A v1 folder name:
/Inbox/Subfolder


Now needs a prefix to determine which store the Inbox belongs to:
/Store2/Inbox/Subfolder


However I'm not totally sure what that 'Store2' part should be. Should it be a complete path and file name to the store? or some arbitrary name that the user calls each mail store? or some UID that Scribe itself generates?

Yeah not so simple (to me at least).
(1) Comment | Add Comment

Svn Repository
Date: 21/11/2008
Something has corrupted my Svn Repository and it's unrecoverable. Most recent backup is 3 months old, and I still have recent checkout's of most of my apps so I don't think I'll actually lose anything other than the hours of sorting out the mess.
  • Checkout old copy.
  • Merge changes from working copy (I love you WinMerge!).
  • Check for new files/folders, add them to svn.
  • Check that it builds.
  • Commit up to date copy.
  • Repeat for all known checkouts (there are a 100 or so!)
*sigh*

Not what I wanted to do today. I plan to make more frequent backups too. In fact I have mostly finished writing a backup application which I'll be launching before year's end. I should've made sure that was live last week, but who knew?
(1) Comment | Add Comment

Valgrind.... MacOSX.... OH YES OH YES OH YES!
Date: 11/11/2008
Someone has done a preliminary port of Valgrind to MacOSX. It's kinda incomplete and buggy but it exists, which means other devs should get on the wagon and get it to production level soon enough.

This is just awesome news. Valgrind is a great tool and seeing it come to Mac is very good news for software quality on the Mac. Mine included.

Last night I was looking at a bug in Scribe/Mac, wishing that I had Valgrind. Well now I do! Well... almost, it's Leopard only at this point so it looks like I'll have to finally upgrade from Tiger. I really love Tiger, in my experience it's a lot more stable than Leopard. So I'm hoping the more recent builds of Leopard are better because I really didn't care for the earlier ones.

Valgrind is something I'd pay for. It's that good.
(0) Comments | Add Comment

Rocket Science
Date: 28/10/2008
There has been two recent events in rocket science that have really advanced the state of the art for private enterprise getting into space. This is exciting on a number of levels, both from the little kid point of view, "wow rockets!" and the more mature, "finally someone other than the government doing it right" point. Generally governments are not good at long term development, and when politics get involved things go sideways (e.g. the shuttle debacle) for a long time. Anyway, here they are:

Armadillo won the level 1 prize of the Lunar Lander Challenge this week. I've been watching Armadillo for a very long time and they finally won something, which is well deserved for all the blood, sweat and tears they put into their rockets. Well done Armadillo!

Secondly, SpaceX put their Falcon1 into orbit, which is a great step forward towards cheap privately funded space access. SpaceX got it right on try number 4 after going so close on the previous 2 attempts. And it's really good to see them get to orbit.

Obviously Scaled Composites won the X Prize a while back, but great things are still happening in the space industry.
(0) Comments | Add Comment

Looking for Scribe v1.88 Users
Date: 24/10/2008
I've had some bug reports from people upgrading between v1.88 to v1.89 or v1.90 where the folder names disappear or other problems with accessing their mail and contacts. I want to look into this issue and sort it out, so I've been running v1.88 in parallel with v2 just to collect some mail and try the upgrade myself. So far so good, the upgrade is perfect, all email, folders and so on usable in later version (although I can't open the upgraded folders in v1.88 anymore).

If you have some mail folders from v1.88 that you haven't opened in a more recent version that doesn't have anything too confidential that you wouldn't mind donating a copy of to improve the Scribe software I'd be interested. Especially if it doesn't upgrade well. To test if it upgrades well or not, MAKE A COPY of your Scribe folder, install a later version (i.e. the latest v1.90 build) into that copied folder and run the new version. If there are problems then you can help! Drop me an email and we'll sort out something in terms of file transfer. If you zip the OLD v1.88 folder you'll probably be able to upload it to a free file sharing site (google it) and post me a URL to download.

That way I can sort out the bugs and stop others from experiencing them.
(1) Comment | Add Comment