ECI: Server Processing Time

A common question is about the time it takes to process X amount of records. We need to know if Easy CSV Importer will manage to process all of our records within the time our server allows. I should say, some users would like to know. That is why I created the ability to avoid importing data or creating posts all in one attempt or event as I refer to it. We don’t need to process 50 MB of data in a single event, we can spread it out a little to avoid upsetting our servers. In most cases the maximum execution time will be 30 seconds, not really a lot of time to do what ECI and WordPress need to do when mass creating posts. I’ve spent 2 years considering how to make it quicker and fit more post creation into each second but it’s not easy. Something a lot of users don’t realise is that ECI uses WordPress own functions to create posts, every part of the post. It uses WordPress functions wherever possible, makes sense right? I’ve not counted exactly but I estimate around 20 database queries per post, that is WordPress not Easy CSV Importer. WordPress needs to check if the post permalink exists or not, checks if tags exist or not, it does a lot of querying when creating categories and each custom field creates it’s own record in the database also.

My Point?

My point is, it is not really possible to predict processing time and I’ve not even mentioned the features in Easy CSV Importer that will effect how many posts you can create per second. There is also your data to consider, I’ve seen files with 5 columns and I’ve had CSV files sent to me with 36 columns. An extra 29 columns is a lot of processing, a lot of memory and if you make use of a lot of the columns in your design the loop to build a posts content is a great deal longer than the loop time for a project of 5 columns.

There are many variables to take into consideration. If you want to create 2000 posts in a blog that already has 2000 posts, it is a lot more work for WordPress than if the blog is a fresh installation. The reason being is when WordPress or ECI queries the database to check existing data the returned values are smaller. When posts exist in the blog, there is more likely to be values returned from queries and then arguments need to be processed as a result.

Seconds vs File Size

I’ve not spent a lot of time focusing on how long a specific file size takes but this is the closest and quickest approach to getting average times. If a 50 MB file takes 100 seconds then we can just say a 100 MB file will take 200 seconds, although it would probably be more like 220 seconds but that is another estimate from experience. It is still not an accurate approach. Take feature images for example. If you use feature images and import images to your WordPress library the use of that feature along adds a great deal of work to your server.

I’m not sure I have the years experience required to guess the number of microseconds each feature will add to the processing time of a single post. If someone does, there are many ECI users who would like to read about your estimates.