HTML Logo by World Wide Web Consortium (www.w3.org). Click to learn more about our commitment to accessibility and standards.
Proud to be a PHP Open Source CMS
Learn more

ocPortal Tutorial: Importing data into ocPortal

Written by Chris Graham, ocProducts
Thumbnail: Importing will generally use the contents of database tables designed for one product to create equivalent data suitable for ocPortal

Importing will generally use the contents of database tables designed for one product to create equivalent data suitable for ocPortal
{!DOC_IMPORT}


Table of contents

  1. ocPortal Tutorial: Importing data into ocPortal
    1. Importers
    2. Using importers
    3. Converting to OCF
    4. Switching forums, without converting to OCF via an importer
    5. Specifics of importers
      1. ocPortal merge
      2. phpNuke
    6. Discussion Forum importing
      1. PhpBB
      2. vBulletin
      3. Invision Board
    7. After importing
    8. Getting a new importer written for you
    9. Additional help
    10. Writing a new importer
    11. See also

Importers

At the time of writing, the following importers are available:
  • phpNuke 6.5 (with possibly other versions and forked-products partially compatible)
  • phpBB 2
  • vBulletin 3
  • Invision Board 1.3.x
  • Invision Board 2.0.x
  • Merge from another copy of the latest version of ocPortal

Using importers

Thumbnail: A configuration file may be required. This screen-shot illustrates what they are and where they tend to be.

A configuration file may be required. This screen-shot illustrates what they are and where they tend to be.
Thumbnail: The list of importers

The list of importers
At the time of writing, all importers work by interface to the database of the product being imported. In addition, some require the presence of a configuration file for the product at an accessible path on the server, and will auto-detect database settings from this file. It is strongly recommended that you leave your old site installed and running, although perhaps at a moved location, so that the importer can find all the associated files that it may want to import.

The importer system uses a concept of 'import sessions'. These are built on top of the ocPortal login sessions, and are an important feature in allowing you to merge multiple sites into ocPortal: they keep the progress and "ID remap table" from each import separate. The "choose your session" interface exists so that if your ocPortal session is lost, you can still resume a previous import.

Thumbnail: Choosing an importer session

Choosing an importer session
Thumbnail: Import options

Import options
Importers define a list of features they can import, along with a dependency system to ensure that a feature can only be imported once any features that it is dependent upon have already been imported (for example, forum posts are always dependent on forum topics, and forum topics are always dependent on forums).

The importer system is designed to be robust, and is programmed as 're-entrant' code; this means that if installation is halted, via failure, timeout, or cancellation, it can continue from where it left off. This is of particular use if there is an incompatibility between your data and the importer, which is not very unlikely due to the wide variation in data for any single product across different versions and usage patterns.

It is recommended that you backup your site files and database before running an importer, in case the importer fails in some way (perhaps an incomplete, or unsatisfactory import, or duplication of data by a poorly written third-party importer).

Thumbnail: After importing some data a success screen is shown. Often special messages will be included on this screen.

After importing some data a success screen is shown. Often special messages will be included on this screen.
Usually an importer will provide information for further actions that must be taken. The following forms of further action are a common requirement:
  • stats recalculation (especially for forum importers)
  • moving of on-disk files from the imported products upload directory, to ocPortal's (this is sometimes done automatically, depending on how the importer was written [which itself depends on the expected data patterns of the imported product]).

ocPortal is designed such that forms of redundancy, such as thumbnails, parsed Comcode, and various forms of tally, can be recalculated dynamically as ocPortal runs. In order to remove load from the importer itself, these are rarely produced by an importer itself.

Converting to OCF

Thumbnail: ocPortal forum drivers are specially coded PHP files stored in the sources/forum directory (or sources_custom/forum).

ocPortal forum drivers are specially coded PHP files stored in the sources/forum directory (or sources_custom/forum).
Thumbnail: The forum driver can be changed using the base-config editor, but you [b]should not do this[/b] unless you know exactly what you are doing

The forum driver can be changed using the base-config editor, but you [b]should not do this[/b] unless you know exactly what you are doing
If you have been running ocPortal and a third-party forum, and wish to switch to using a complete ocPortal solution (ocPortal with OCF), this is possible if there is a forum importer for your current forum product. The opportunity is presented to move to OCF as the last importable feature of a forum import, and the function will 'jump' forum drivers for you and re-map any group and user id's. It is still strongly advised to check your permissions after performing this.

Important note: If you have installed ocPortal, and interfaced to a third-party forum, but want to switch to OCF without an import (because your forum is essentially empty still), then it is possible but we would discourage it for anyone other than an expert user. To do this you need to use the config editor to 'jump forum driver', but you also need to reset all your permissions for the new forum drivers user-groups, which quite possibly leaves a window of vulnerability if privileges are present which should not be. In addition, any ocPortal systems that reference users will reference different users after switching, as user-id's will have changed: for example, point transactions and admin logs will reference the wrong users.

Switching forums, without converting to OCF via an importer

ocProducts does not in any way support moving between different forum vendors/databases, after ocPortal has been installed. This is due to the member and usergroup IDs ocPortal uses being tied to the member and usergroup IDs of the linked forum. Because the actual usergroup and member data is not held or managed by ocPortal, ocPortal is entirely sensitive to changes to this data, yet has no way of properly detecting or synching against changes made to it.
All this said, it is possible to move between forums, if manual database changes are made to correct the aforementioned problems.

Specifics of importers

ocPortal merge

When merging with another copy of ocPortal, you should make sure the other copy must be running the same major version. Whilst other versions may merge succesfully, ocProducts does not support this officially.

The 'ocp merge' importer can merge multiple ocPortal websites together that either:
  • each run on OCF (and thus, OCF data gets merged)
  • or, each share a forum database (what we call a "multi-site-network" situation)
The importer only handles ocPortal (including OCF) data.

The 'ocp merge' importer cannot:
  • Work with anything other than ocPortal data (third-party forum data can not be merged, for instance)
  • Merge an ocPortal site into an ocPortal site that does not use the same forum database
    • unless you are highly technically proficient and capable of manually changing member and usergroup IDs, using a tool such as phpMyAdmin (because these IDs could not be mapped correctly for data that used a 'foreign' forum)
    • or unless both sites run on OCF (because in this situation, the importer can import everything, and correctly remap any member and usergroup IDs)
  • Import OCF data directly into a third-party forum
    • because the imported data would end up in the OCF database tables, regardless of whether they are currently being used for the ocPortal site's active forum.

Remember:
  • you must specify to import from an ocPortal database, not a forum database.

phpNuke

Tip

Many products have been 'forked' from phpNuke ('nukes') and may be compatible with this importer, with minimal or no changes. Products include: e107, Xoops, PostNuke, OpenPHPNuke, and many others.
Generally speaking, ocPortal does a lot more than phpNuke, so a phpNuke import is fairly straight-forward.

Compatibility notes:
  • ocPortal has a combined news system, so journals, editorials, stories, news, and reviews are all imported together. When I wrote the phpNuke importer, I was surprised what a "copy&paste mess" phpNuke was in this area.
  • The encyclopaedia is imported into CEDI, which is much more powerful, but different.
  • FAQS get placed in a catalogue, as do Links, ephemerides, and contacts.
  • Pages are turned into Comcode pages, with the HTML embedded in an HTML tag. They are are indexed through the pnindex page.
  • Statistics are not imported.
  • Permissions are not imported, as phpNuke doesn't have a permission system anything like ocPortal's
  • You need to copy the current news category (topic) images into themes/default/images/newscats
  • No files are moved to the ocPortal directory: this needs doing manually.

Discussion Forum importing

Compatibility notes in general:
  • personal/private message will be glued together to form personal topics. This is a very useful feature, and really cleans up an inbox.

PhpBB

Compatibility notes:
  • phpBB does not support attachments by default, but there is an attachment importer. Only use this if you have the popular attachments mod installed and running. Attachments are moved to the ocPortal attachment upload directory.
  • phpBB uses a very strange group configuration, so it is necessary to check your user-groups, permissions and group membership after import.

vBulletin

Compatibility notes:
  • the vBulletin calendar recurrence system is very different to the ocPortal calendar recurrence system, so recurrences may not be imported perfectly
  • forms of rating, such as topic rating, karma, and "goes to coventry", are not imported. However reputation is imported as points.
  • vBulletin has an enormous array of options, and many are not supported (we consider the product over complex in terms of configuration)
  • attachments, photos and avatars are extracted from the database to the appropriate ocPortal uploads directory. It is best to use the live database for the import, because there is a mySQL/vBulletin bug in some mySQL versions that causes binary database data to be corrupted in SQL dumps.

Invision Board

Compatibility notes:
  • Invision Board has an enormous array of options, and many are not supported (we consider the product over complex in terms of configuration)
  • attachments, photos and avatars are moved to the appropriate ocPortal upload directory

After importing

If the importer you used copied all relevant files, like avatars, photos and attachments, into ocPortal's directories, then you can remove the imported product directory in whole.
However, it is advisable to keep both the directory, database, and import session, around for a few weeks, just in case any data was not correctly imported and extra maintenance required to put things right: importing is a technically complex process, so it is always best to keep your doors open.

Getting a new importer written for you

ocProducts is always happy to hear from you if you have a website running a moderately-to-highly popular product, and you would like to switch to running our software. This is not just for the obvious reason of increasing our user-base, but also presents us an opportunity: to write and test an importer to a working website. Programming an importer to a database is relatively easy for us, but getting access to test data is not. Please contact us and we may be able to arrange a free conversion!

Writing a new importer

This is an advanced section, and should be ignored by non-programmers.

Importers are implemented using the ocPortal 'hook' system, meaning that a new importer may be added by just copying the importer-file into the sources_custom/hooks/modules/admin_import directory.
The importer-file consists of an object that defines an info section that exports information in a standard format, and feature-import functions, which are named according to the code-names of the import features ocPortal has been coded to recognise. These features are hard-coded, but it is possible to change them, by minor modification of the array at line 238 of the adminzone admin_import module (an array is defined here that links import code-names and language strings).
Once the hook is put in place, the importer will appear on the list of importers available for use on the first import module screen.

The importer info function returns an array which contains the following:
  • 'product', the name and versions of the product(s) the importer supports
  • 'prefix', the default table prefix for the database of the product
  • 'import', a list of feature-codename's that are supported (all of which have feature_<codename> associated functions in the importer).

The import functions themselves are passed details that will allow them access to the imported database, and they generally just loop over the data for the feature being imported, and use the ocPortal API's to add the data. Often special API parameters are used to disable cache-regeneration and input-validity-checks, to allow for a smoother and more error-free import. The 'import' source module defines a number of functions that make re-entry, and id-remapping easily achievable (id remapping is necessary, as the id's used to reference rows in the old database will be changed in ocPortal: for example, a post referencing it's topic-id will need to be remapped to the new topic-id once imported into ocPortal [and this is why there would be a dependency for topics to be imported first, so that the remap-id could be known and stored before-hand]).

Often the easiest way to write an importer is to copy and paste the importer code for the most similar product/database-source and work through the code function by function, adjusting field names and other code as appropriate.

Important note

Always test a new importer and keep backups before actually running it on your data. It is surprisingly easy to make a typo in an update query, for example, which will trash a whole table (and if also accidentally executed on the source database, perhaps more disastrous).
There is a very complex situation for importers that use forum-driver functions during import, in order to find data to associate with database records they are creating. For example, an importer for a simple download system which stores 'usernames' of download-submitters, might wish to try to bind these usernames to actual user-ids, using the forum-database that ocPortal normally uses. However, by default the forum-database is tied to the local-OCF-install during installation, as when forum data is imported, this OCF database is required (as this is where the information will be piped to). Therefore there are two 'importer' source file functions that allow switching to and from local and M.S.N. OCF installations:
  • ocf_over_local
  • ocf_over_msn
Note that the importer system does not change the actual forum driver used. Therefore it cannot be assumed that the global FORUM_DRIVER is an OCF forum driver, not can it be assumed it holds a local database. The OCF system has been programmed to never use FORUM_DRIVER itself, it uses OCF_DRIVER (which you may use yourself if writing a forum importer), which is always guaranteed to be linked to OCF.



See also