I’ve spent the better part of two months learning and integrating with Magento eCommerce software. During this time I’ve bumped into several limitations and scaling issues because of the particular problem domain I’m working in but, at the end of it all, there are a few tips and tricks I’ve discovered which have helped. I’ll just present them here in no particular order. Hopefully they help someone else in future.

Avoid text attributes as much as reasonably possible. The text attributes you create for your products correspond to TEXT and/or BLOB data types in the MySQL database. When MySQL processes a query which generates a result set with a TEXT or BLOB column, it doesn’t use in-memory temporary tables because the MEMORY engine does not support those column types. These temporary tables are created on disk. This increased disk usage obviously results in increased disk IO which slows down everything on your server. So if you have significant amounts of Text attributes, you should make sure you have some quality high-speed disk systems or specify that temporary tables be created on a virtual file system backed up by memory. (Is ramdisk still the correct term? Am I getting old?)

Implement your own RPC interface to speed up the SOAP or XML-RPC API interface. Why? Because it’s sloooow. My problem involved importing about 500,000 products into the Magento system and it was decided to use the API provided. Unfortunately, it took a long long time but that provided me with a lot of time to learn about Magento in my attempts to speed it up. Technically, you have two choices here: 1) delve straight into the database and build queries yourself; 2) Implement a small wrapper about the API calls to replace the existing API calls.

Choice #2 is the obvious one since it’s maintainable. What I did was to implement a small XML-RPC server in PHP with the same method signatures as the Magento API for the calls I needed (catalog_product.create, etc). My little stubs then imported the Mage system, instantiated an API class (Mage_Catalog_Model_Product_Api) and called the existing Magento logic for product creation. Rather oddly, this sped things up dramatically.

Keep as little data within Magento as possible. Keep enough to display your products the way you want them to be displayed but no more. The Magento database schema is a bit unexpected; it has products split into several attribute tables based on the data type of the attribute. So there’s an integer table for integer attributes, a varchar table for varchar attributes, etc. This seems reasonable but it has a performance impact when your physical hardware can’t back it up. If you don’t have sufficient memory and quality disk systems, a single product lookup can result in an awful lot of disk access. Searches through all products can become impossible when you tend towards large amounts of data. So try to keep it down to a minimum or throw lots of hardware at the problem.

Eventually, our data was kept in a separate database designed specifically for that purpose and our internal systems queried and maintained that as the authoritative source – Magento served as the customer front-end and relative data was pushed to it when required. Much better for long-term possibilities and plans.

Allocate as much memory as you have data and then some. The storage engine used by Magento is InnoDB. InnoDB needs a lot of memory. If you don’t have a lot of memory then disk access once again becomes and issue and your system starts becoming unusable. Make sure you have at least as much memory as you have data within your database. Allocate it all properly to the InnoDB engine. Use helper scripts like mysqltuner.pl to diagnose problems.

Uploaded files and save locations. Not sure why but I guess the makers behing Magento assumed there would be a limited amount of images per product. So they used a 2-layer/depth file system for uploaded images. If your product image is called “myimage.jpg”, it would be uploaded to the “m/y/myimage.jpg” location beneath your media root. Each of the products in my local Magento could have 3 images and that totaled a possible 1.5 million files at least. If there was no file naming bias in the product image files, then you would be looking at directories with maybe 1K files. However, if your image files are all coded according to a logical scheme – as mine are – then there’s a significant bias and Magento’s 2-depth system results in several thousand files in a single directory. I manually increased this to a 3-depth layer system by modifying the getDispretionPath function in (ROOT)/lib/Varien/File/Uploader.php to bring the count back down to sanity. Ideally, though, Magento makers should fix this and allocate directory trees based on a random number Magento generates (or hash, sha1, whatever). It won’t be easy to find your files again but it will allow Magento to scale a bit better.

Keep your database as close as possible to your Magento installation. When it became obvious that Magento needed a lot of memory and power at my required usage levels, we tried to use Amazon RDS – which is really awesome and worked well – but it didn’t work out. Magento actively uses the database and each RPC call or operation performs several operations on the database. So the latency between the server and Amazon was suddenly the bottleneck. So keep your data close on a high-spec’d machine.

Is Magento scalable? If you have the money to throw at hardware, I’m sure it will scale okay-ishly. I think it’ll scale vertically to a point. I believe the caching system they have was probably a response to the slow speed of the database system they built. I believe the caching is there to hide the speed issues when you have a lot of products. (I suppose that’s what all caching is for?)

Do I recommend Magento?. Yes. My problems were just due to my large number of products and limited hardware due to budget constraints. If you don’t have both of those things, then I’m sure you’ll be fine.

For the last several months, I’ve been developing in Python for a Django project. Python is new to me but there’s something about it which just feels right. It’s clean, dynamic, powerful, comes with a full complement of tools and has already proven itself useful in the field with large proponents such as Google as well as my Linux distribution of choice – Gentoo – basing their system package manager off of it.

While I don’t post here often, I decided to share the Enum class I’ve been using with Django which is specifically used for the Django ModelFields when you specify the choices for an attribute of your Model.

For those who don’t know what the above means, Django uses a model approach in which you specify the data model you will be using – for example a User which could contain a username, password, etc – to represent the database schema. So, you design a single Python class, say User, and you add the attributes of a username and password. Django then loads this class and creates the database schema for you. Also, because the schema is ultimately a Python class and based off of Django Model classes, you can use the class and manipulate it and the Django subsystem will know how to save, load and update the data stored in the database without you ever having to worry about SQL. Kinda like:

current = User.objects.get( id = 1 )
print "Hello {}".format( current.username )
current.username = 'NewName'
current.save()

Viola. Extremely convenient.

Some of the fields in your schema – or attributes of your Model – will have a limited amount of valid options. For example, an User account may be INACTIVE, ACTIVE or DISABLED. These states are the only possible states. So the User.state attribute should only be limited to those possible choices. You can specify this in django with the following:

USER_STATES = ( ( 0, 'Inactive' ), ( 1, 'Active' ), ( 2, 'Disabled' ) )

class User( models.Model ):
    ...
    state = models.IntegerField( choices = USER_STATES, default = 0 )
    ...

So now that Integer field will be validated against the allowed states. The problem is, you now have to refer to the states numerically from now on in your code:

if myuser.state == 0:
    do this

Or alternatively:

if myuser.state == USER_STATES[0][0]:
    do this

Both of which is really messy and bad. You want to be able to define the user states in a similar manner you do with other languages – using an Enum. Turns out it’s quite easy if you write your own Enum class. Here is the class I wrote:

ChoicesEnum Class for Django

class ChoicesEnum( object ):
    def __init__( self, *args, **kwargs ):
        super( ChoicesEnum, self ).__init__()
        vals = {}
        for key,val in kwargs.iteritems():
            vals[ key ] = val
        object.__setattr__( self, "_vals", vals )

    def choices( self ):
        cho = []
        vals = object.__getattribute__( self, "_vals" )
        for key, val in vals.iteritems():
            cho.append( val )
        cho.sort()
        return cho

    def __getattr__( self, name ):
        return object.__getattribute__( self, "_vals" )[ name ][ 0 ]

    def __setattr__( self, name, value ):
        object.__setattr__( self, "_vals" )[ name ][ 0 ] = value

    def __delattr__( self, name ):
        del object.__setattr__( self, "_vals" )[ name ]

And so usage would be as follows:

UserState = ChoicesEnum(
      INACTIVE = ( 0, 'Inactive' ),
      ACTIVE = ( 1, 'Active' ),
      DISABLED = ( 2, 'Disabled' ),
)

class User( models.Model ):
    ...
    state = models.IntegerField(
                     choices = UserState.choices(),
                     default = UserState.INACTIVE
                    )
    ...

And then you can use the UserState object as you would a normal Enum on most languages:

if myuser.state == UserState.INACTIVE:
    do this

It’s really quite nice and cleans up your code substantially.

Reason for Problem:

There are a few reasons for this:

• Some servers run a battle.net server clone using PvPGN. PvPGN has a “missing feature” when it comes to game advertising. The real Battle.Net servers are smart enough to recognize when two or more machines are trying to connect from a single IP address (ADSL connection, for example) and send back the internal IP of the host to its neighbours. The PvPGN server just sends back the Internet IP address to the host’s neighbours.
• Routers have to be smart enough or configured to perform “NAT loopback” or “Reverse NAT” or “PAT” for connections originating from inside the local network to the router’s Internet addressable IP address and port.

So if you are unfortunate enough to combine these two problems, your friends on your LAN won’t be able to join any Internet games you’re hosting.

Typical example in which your host’s IP address is 10.0.0.5 and your Router’s Internet IP is 66.1.1.2.

• Your host creates a game on the server.
• The server lists the game and all other people joining are told to connect to 66.1.1.2:6112.
• A buddy on your LAN (10.0.0.6) tries to join and is told to connect to 66.1.1.2:6112 as well. (A real Battle.Net server would tell it to go to 10.0.0.5:6112).
• Your buddy’s game tries to connect… which means it tries to connect to your Router’s Internet IP.
• The Router doesn’t have “NAT loopback” or “Reverse NAT” or “NAT on the inside” to forward internal requests (from 10.0.0./24) back into the network (to 10.0.0.5).
• Your buddy’s connection attempt times out, is rejected or fails.
• Your buddy goes back to his house to connect over the Internet. Everyone is sad.

Confirmed Working Router

I’ve found a good router which does NAT loopback out-of-the-box without any configuration.

It’s the Billion 7300G (RA) – ADSL 2+ Wireless 802.11g ADSL Modem/Router, 4 Port

I can confirm that this router works perfectly. It’s also quite a nice router.

Just Extra Stuff

Some routers will work automatically. Some will need to be configured. Some just won’t work. The thing with this is that it’s not really a feature that a router will advertise as having (like on the box or on it’s product page). You just have to know or test that it works.

Also, there are other methods for doing this – like setting your ADSL router into bridged mode – but this both increases the complexity of your network and exposes your internal machines directly to the Internet… so your security is compromised.

With computer security, you not only need to protect against incoming attacks but you also need to detect when an intrusion has taken place. AIDE is good for detecting successful intrusions within a server.

It works by creating an initial database of checksums for the files you want to monitor. You can choose from a myriad of hashing algorithms including MD5 and SHA1.  From then on, you can have AIDE check all your current system files against the compiled database.  If a file has changed it will be detected and you can take appropriate action if any is required.

This is useful if your machine becomes infected with a trojan or virus – the kind that modifies the binaries to embed itself within the usual commands. For example, coreutils’ ls, mkdir, etc.

The installation and setup is trivial, give or take the time you need to understand the configuration file and list the directories and files to monitor. So give it a go.

Here in South Africa, we have a monopolistic telecommunications provider named Telkom which overcharges and underdelivers basic services such as ADSL. I have to pay R 70 (Approximately USD $7  at the moment) for each gigabyte.  Besides that, I have a monthly ADSL cap of 3 GB which is shared amongst 4 people in my household. Apart from stifling the development of internet-based services in South Africa (imagine YouTube having to pay $7 for every GB a user uploads or downloads to them), it also makes simple things like transferring my meagre 100 MB backups locally a little unpleasant.  Telkom also hasn’t seen the need to introduce speeds over 4 Mbps. I’m sitting on 386 Kbps which suites me just fine because, if I went any faster, I might be tempted to use my bandwith cap within a day.

So I saw an advert for rsync.net which advertised rather cheap and convenient storage space. They basically offer disk space and provide ssh, ftp, rsync, webdav and *insert your protocol here* access to it. Can it be more convenient? I signed up with them on Wednesday and modified my backup system to scp all backups to their servers in Switzerland.  The only issue with the sign up was the +- 12 hour wait for the system to become active. If they automated signup and I had had instant access to the space, it would have been terrific.  Otherwise, everything worked first time as it should.

If you need some space for backups and you enjoy geographic safety, go with them.  I’d recommend them.

You will probably want to use this technique if:

• Your firewall is preventing you from accessing what you need to access outside your network.
• You don’t have control over the firewall to open up the ports you need.
• You DO have access to an external server via ssh.
• The ssh daemon on the remote server has been configured to allow this or you can configure it yourself.  (GatewayPorts = yes)

I’ll just quickly go over the steps which will get you ready to run:

1. Install tsocks.
2. Configure tsocks.  My configuration looks like this:

   local = 192.168.0.0/255.255.0.0
   server = 127.0.0.1
   server_type = 4
   server_port = 1080
   
   Which means that for all local addresses, it will go directly over the LAN. However, for everything else, it will tunnel to 127.0.0.1:1080 over SOCKS version 4.

Now the only thing left is to create the SSH tunnel on a local port of 1080 (as configured in tsocks.conf) to your remote server:

bash#  ssh -D 1080  <username>@<server.com>

Leave that running. It’s your link to the outside world. Now open up another terminal and do whatever you need to do. In this example, we’ll start firefox:  (make sure your firefox isn’t already running so that it’s a complete restart.)

bash# source /usr/bin/tsocks on
bash# firefox

And…. that’s pretty much it.

Tsocks preload’s itself as a library and implements it’s own network operations. (connect,  etc..).   So it works like magic.

It turns out that it works quite well.

I used a combination of qlist, find, cat, sort and uniq to determine which files were orphans on my system.

For all files which are supposed to be there: qlist -IC | xargs -n 1 qlist

Apart from some cache files, tmp files, a few configuration files and some portage related stuff, nothing was out of place.

So it’s a very clean management system. Hooray.

If you mount your iPod under linux, you’ll notice straight away that they have some funny database system for storing the songs. This database version and format actually changes between revisions of the iPod software release. The open-source solutions are all capable of reading and writing the older versions because they’ve reverse engineered the format. But they need some more time to get the new ones working. Unlucky for me.

As a result, most of the open-source solutions would actually copy the songs to the iPod but failed to update the iPod database. Oddly enough they didn’t complain about this. So I would just assume that it worked but the music didn’t show up in the iPod menu.

In the end, I tried the following solutions:

• Amarok – failed to install ‘cos I don’t have KDE installed and it was looking for something called kbuildsycoca on startup. Apparently it will wipe out your cover art on some iPods if you’re not careful.
• Banshee – I tried the stable 0.12.1 release in the portage tree. It’s okay. It’s the only open-source solution which actually downloaded cover art for my library. The next unstable release is 1.2.1-rc2 which was brilliant compared to the previous release. Unfortunately it didn’t work with my iPod version.
• gtkpod – Nice, convenient and small. Not too feature-full at all. I would have preferred to install and use this one because everything else required hundreds of other dependencies. Pity it didn’t work.
• Rhythmbox – Similar to banshee, smaller. Would have been nice if it had worked. I think this is the default manager on Ubuntu so it’s probably well supported and has a future.
• Songbird – Now this is my ideal manager. It’s very similar to iTunes and it didn’t require hundreds of dependencies. You can just download the binaries from their site and run it. Fantastic. It’s a professional product as well. All shiny. When they support my iPod, I’m switching to this.

My conclusions and next steps:

1. I’m going to use iTunes as a stop-gap until Songbird supports the new iPod Classic on Linux.
2. I need to convert my entire FLAC library to ALAC so that I can import my songs losslessly into iTunes until I get a linux client. (They all support my FLAC ‘cos I compiled it in everywhere.)
3. I polluted my nice clean gentoo install with hundreds of new dependencies and applications which didn’t work. I’m now spending the day cleaning up all orphaned files which don’t belong. It gives me something to do.

The tricky part is the extreme lock-in by Apple which you only tend to discover afterwards and especially if you happen to be running Linux! Here are some poor actions and decisions on Apple’s part designed to lock you in, as well as some of the obstacles an awesome Linux user like me has encountered:

• An iPod registers itself as external USB storage when I plug it into the machine. Great. However, you can’t just copy your music files over. I have all my CDs saved in FLAC format on a file server with ID3 tags embedded. I was really just assuming that I could convert them to something more supported (MP3 or AAC or Apple’s Lossless aka ALAC) and copy them across. However, the iPod keeps all the music within a proprietary database format and you need specialized open-source applications which have reverse engineered the format to copy them in.
• I like my open-source applications lean and mean. (I run Gentoo. I’m weird that way). The open-source applications available aren’t too lean and mean at all. I’m still trying them all out, one at a time, until I find one that a) works and b) is easy to use.
• iTunes only runs on Windows and Mac. This is a huge problem, as far as I’m concerned because the authority on Apple formats and their own proprietary stuff is Apple themselves. Because they don’t support Linux, the support you can find on a Linux platform is all reverse engineered and not up to date. So, to do an iPod justice and get it working out the box, you kinda need iTunes here. Unless you’re willing to experiment with the open-source software mentioned above.
• The iTunes store for South Africa doesn’t have any music! What the heck?
• I imported several common CDs of mine into iTunes and the cover art wasn’t loaded even though I asked nicely. Bah humbug. What’s the point of the pretty screen without cover art? I think this may have to do with the fact that iTunes SA doesn’t have the music in the first place.

Uh.. and that’s all. It’s a very pretty device. It would have been amazing if they didn’t lock it down the way they have. I’ll let you know what I find out about the open-source products.