I have been working this week in a new custom model for one of the applications we are developing here at ICT Consulting. We found a problem with many many rows in gtk.ListStore. This is a common problem with Gtk's ListStore and TreeStore. Both are easy to implement and convenient if you are not going to work with too many values. I'm thinking in a couple of thousands. As far as this, the TreeModel is quite usable, but if you use a TreeFilterModel and/or TreeModelSort then you will experience some performance issues.

If what you want to do is just map the main application and then fill the model you can use the idle_add gobject function to iterate over your data and then store it in your model without freezing your application, but it is at the same way, the slowest. You can put chunks of 20 or so, then every time you application runs the callback function will run a small but faster insertion. Anyway, this is still a bit slow.

I have been doing tests with 100,000 rows, in the easy way there are just numbers using a for cycle via a range() result.I think in Gtk, the smaller is the data stored in the model, the faster it goes, because I can insert 100,000 rows in a gtk.ListStore in 7 seconds aprox. while in christine, inserting 2600 row in christine (with more data per cell) takes a lot more.

Anyway, the application that I'm developing shows a list of customers, this is a growing list as new customers are added, in our tests we need to show up to 17,000 customers. Using gtk.ListStore takes at least 17 seconds on my machine just to show it, but we also have a filter model becuse search in a list of 17000 items is not easy if you go looking every row by hand. So, usig the FilterModel takes almost the same time in searching something. Obviusly, waiting 17 seconds for the result is not an option, even worse, you made a mistake and wrote LOPEX instead LOPEZ.

So, I was searching over internet what can I do. I have already use the common 'tweaks' to improve the gtk.ListStore performance and I have reached the 8 - 9 seconds over those 17,000 items, but even with this, its too much time. The search points me to the gtk.GenericTreeModel. This generic model allows you to create a custom model where you can make your own tweaks to improve the performance of your model. In the link above gives you the way to use it. I have alreade created my own custom model and it really relly made the things better.

Now, one of the problems that gtk.GenericTreeModel have are the memory leaks. Using this model you may use lots of memory. This is because the GTM use to create a reference to the value that you are returning as Iter, e.g. you are using a list of lists, where you store the values, every item in the list is a row, then, the index if the iter wich references to the list you are storing in that row. The problem is that the GTM increase the reference count of you node (the list inside of the list) to aviod row destruction and then use the iter. This, in the best will make your values been refered until you destroy your model (and any reference to the data), in the worse the reference count of your data will never been unrefered and then the memory will never be freed.

You can use the invalidate_iters every time you modify the rows (add or delete) to make the rows invalid and then decrease the reference count of the row, and when your data have no more references then be deleted. But, I have the problem that the reference count was never decreased when I destroy the window or the model. Where is the trick here, use the Python's garbage collector, and your memory will be freed.

So, the new model based on PyGtk's GenericTreeModel actually works for us. Inserting 17,000 takes just 1 second, and the search (filter) is the same thing, 1 second is an option. I would like to see everything just appear when you request it, but one second is good for now. And testing with my current christine music list it takes just 0.06 seconds to show the list. Inserting 100,000 items (the easy way, just numbers) takes just 1.164 seconds to get filled with four columns and 2.36 seconds to show itself (creating a filter model and setting it to a treeview), so, for now this kind of optimizations will work for us, there are other optimizations on the Filter model and the treeview, My goal is to create something similar to wat Aaron bockover do with banshee.

Note: Test times are approximate, cache, cpu usage, load average and other may cause the times change, but at the end give us an idea of what is going on.