Getting data into your site

There's little doubt that data makes the difference between a 'me-too' brochureware site, and an active and useful internet presence. However that data isn't always where you need it to be – on the site. What we need is an easy way to import, lookup, or synchronise data from other sources, and make it available to the table/query/view system that powers the data handling in clearString neatComponents.

Today I'm talking about the component that manages that process, to give you a feel for what's possible. The 'External Lookup Data Feed' component can be pointed at an external source – typically an XML file or an RSS or Atom feed – and will pull the data found there into one or more tables within the system.

A simple example would be where you are doing a one-off import, to migrate a blog from a legacy platform like Wordpress: point the component at the Wordpress blog's RSS feed, map the feed fields to the ones in the table, and trigger the lookup. All done in moments.

If it's not a one-off import, but a regular pulling in of data you need, simply add a schedule, and it will repeat the process every so often – you decide the schedule – and it will take care not to duplicate any records already imported, but will update any which need refreshing.

Sometimes the dataset is too big to import, or simply unavailable in complete form. In these cases you need to perform an on-demand lookup, and it will pull in the particular records you need. For example, on a car parts website, you might want to let the user enter their car registration plate, and the lookup asks the national database which make and model it is – and thus present a catalog of the parts suitable for that particular vehicle.

Again, the lookup component can handle this, passing forward the search terms entered by the user in the request to the external database, having first checked to make sure the details aren't already known (to save time, and, if charged per request, money).

Sometimes things go wrong with external databases, so we also gracefully handle request timeouts, allowing for failover to backup feeds, and of course some lookups need to be to secure sites or require authentication to access them.

This has only given a quick sample of what's possible; for more information see: www.neatcomponents.com/external-lookup