Skip to main content

Importing properties

AgentBox#

Klemich uses AgentBox as their Real Estate CRM and property management system. This is the source of truth for all property data. From here properties are exported to various platforms such as real estate listing websites, print advertising, and the Klemich website.

Access#

I find I have no need for access to AgentBox in order to maintain the website. Regardless, if you would like a login to familiarise yourself get in touch with someone at Klemich.

XML dumps#

When any properties are updated in AgentBox, AgentBox use a restricted FTP account to upload XML files containing only the changes to klemich.snarkiedesign.com.au/XML/feedsync/input. These XML files are in REAXML format; a standard for Real Estate property data.

This is an old-fashioned method of transferring data, however at the time this was the only method provided to me by AgentBox.

REST API#

AgentBox have a JSON REST API. Unfortunately there is no publicly available information about the existence of this API, so I only discovered this after pressing AgentBox support. If I had been aware of this API before setting up the XML property import process, I would have used it. I believe it would be much faster and less complicated.

You can get access to this API by contacting AgentBox support, which involves submitting an integration partner application (yes... really). The API is not well documented and by default they will not provide you with access to all endpoints, let alone inform you other endpoints exist! After some investigation I put a request in to enable the Listings endpoint: api.agentboxcrm.com.au/listings/. This provides data for property listings. I was able to request properties, however the format was not ideal as individual requests needed to be made for every property in order to scrape all required data for an import. To solve this I built a Node.js API over the top of this, which aggregates this data for you into single responses. I found this to be quite powerful and more robust then the current XML import process. I built this with the intent of updating the import process however I did not complete this due to time restraints. I can supply this if you would be interested in it. One huge advantage of this REST API is that it would negate the need for the FeedSync middle-man importer described below.

Images#

AgentBox policy is that API consumers can not provide public access to the property images hosted on AgentBox. It's not ideal, but I assume this is to reduce server load on AgentBox servers. Because of this the import process must download all property images and host them elsewhere.

Support#

You can log tickets with AgentBox support team by emailing support@agentbox.com.au. On first contact they may need to verify your relationship with Klemich Real Estate.

FeedSync#

A WordPress plugin which is installed on the server at klemich.snarkiedesign.com.au/XML/feedsync. You can access the admin panel by visiting this url http://klemich.snarkiedesign.com.au/XML/feedsync/core/login.php.

FeedSync allows you to import REAXML formatted XML files into WordPress. Triggered by a cron job, it will regularly process any XML files that are uploaded to klemich.snarkiedesign.com.au/XML/feedsync/input. All data is stored in a seperate FeedSync database. FeedSync properties are exported to WordPress over the supplied REST API which ouputs XML. You are then able to query and filter properties over the API, which is requried by the WordPress XML import tool WP All import pro in the next step.

Please see the FeedSync documentation for more details.

WP All import pro#

A WordPress plugin that allows you to import data in a variety of formats, including XML. Triggered by cron jobs, WP All import pro will process FeedSync data into WordPress as Properties posts. Please see the WP All import pro documentation for more details.

There are multiple imports setup, as there are varying requirements for different property types and statuses. The cron jobs for these imports are staggered to avoid processing congestion.

Problems#

Performance issues#

I have noticed significant performance isssues with this plugin. Unfortunately this seems to be the best method of importing XML data into WordPress and I have struggled to find alternatives. The imports are very slow, especially on new property imports as it downloads new images from AgentBox. It uses a significant amount of memory and processing power on the server, which has meant the server specs need to be unusually high for a WordPress instance. At times, large imports have taken the server offline. The server automatically restores, and Cloudflare caching usually makes this unnoticable on the frontend. Despite band-aid fixes I have not been able to completely rule out or remedy the issues.

Bugs#

On rare occasions, the imports have missed certain data fields. I have not been able to trace the cause of these problems but it has lead me to believe the plugin is too unreliable for continued use. To resolve missing data issues I have to delete the property in WordPress and allow the plugin to re-import it.

Alternative solutions#

Inside the WordPress ecosystem I have struggled to find any reliable import solution for XML. Instead I opted to look for more performant and stable content management systems. I have built a proof of concept with the CraftCMS FeedMe plugin, sourced data from the Node.js property API discussed earlier. I have successfully ran property imports with enormous speed savings compared to the current XML process. I have also found this to be much more reliable. I can provide this to you if you would like. The disadvantage of changing CMS is obviously having to rebuild all other WordPress data structures and content again, so I'm unsure this is the best solution, however once setup I imagine it would make management of the website much less problematic.

Summary#

In short, the overall import process is as follows:

  1. AgentBox dumps XML in REAXML format containing property data onto the server via FTP
  2. Triggered by a cron job, FeedSync regulary processes these XML files into it's own database
  3. Triggered by cron jobs, WordPress plugin WP All import pro pulls data from FeedSync over the Feedsync XML REST API and imports each property as a post in WordPress
  4. The properties are now available in WordPress over the WordPress REST API for consumption by the React application