ocPortal Tutorial: Importing data into ocPortal
Written by Chris Graham, ocProducts
Importing will generally use the contents of database tables designed for one product to create equivalent data suitable for ocPortal |
Table of contents
- ocPortal Tutorial: Importing data into ocPortal
Importers
At the time of writing, the following software importers are available:- Advanced Electron Forum
- Invision Board 1.3.x
- Invision Board 2.0.x
- Joomla 1.5
- MkPortal
- MyBB
- phpBB 2
- phpNuke 6.5 (with possibly other versions and forked-products partially compatible)
- Simple Machine Forum
- vBulletin 3
- Wordpress
- WowBB
- HTML website importer
- Merge from another copy of the latest version of ocPortal
Memory limits
Importers may use a lot of memory in order to transfer large amounts of data, so you may need to raise the memory limit on your server or import on a different server then copy your site over.Information on PHP memory limits is included in our FAQ.
Using importers
A configuration file may be required. This screen-shot illustrates what they are and where they tend to be. |
The list of importers |
The importer system uses a concept of 'import sessions'. These are built on top of the ocPortal login sessions, and are an important feature in allowing you to merge multiple sites into ocPortal: they keep the progress and 'ID remap table' from each import separate. The 'choose your session' interface exists so that if your ocPortal session is lost, you can still resume a previous import.
Choosing an importer session |
Import options |
The importer system is designed to be robust, and is programmed as 're-entrant' code; this means that if installation is halted, via failure, timeout, or cancellation, it can continue from where it left off. This is of particular use if there is an incompatibility between your data and the importer, which is not very unlikely due to the wide variation in data for any single product across different versions and usage patterns.
It is recommended that you backup your site files and database before running an importer, in case the importer fails in some way (perhaps an incomplete, or unsatisfactory import, or duplication of data by a poorly written third-party importer).
After importing some data a success screen is shown. Often special messages will be included on this screen. |
- stats recalculation (especially for forum importers)
- moving of on-disk files from the imported products upload directory, to ocPortal's (this is sometimes done automatically, depending on how the importer was written [which itself depends on the expected data patterns of the imported product]).
ocPortal is designed so that forms of redundancy, such as thumbnails, parsed Comcode, and various forms of tally, can be recalculated dynamically as ocPortal runs. In order to remove load from the importer itself, these are rarely produced by the importer.
Converting to OCF
ocPortal forum drivers are specially coded PHP files stored in the sources/forum directory (or sources_custom/forum). |
![Thumbnail: The forum driver can be changed using the base-config editor, but you [b]should not do this[/b] unless you know exactly what you are doing](http://ocportal.com/data/images/docs/importer_change_forum_driver.png)
The forum driver can be changed using the base-config editor, but you [b]should not do this[/b] unless you know exactly what you are doing |
Important note: If you have installed ocPortal, and interfaced to a third-party forum, but want to switch to OCF without an import (because your forum is essentially empty still), then it is possible but we would discourage it for anyone other than an expert user. To do this you need to use the config editor to 'jump forum driver', but you also need to reset all your permissions for the new forum drivers usergroups, which quite possibly leaves a window of vulnerability if privileges are present which should not be. In addition, any ocPortal systems that reference users will reference different users after switching, as user-IDs will have changed: for example, point transactions and admin logs will reference the wrong users.
Switching forums, without converting to OCF via an importer
ocProducts does not in any way support moving between different forum vendors/databases, after ocPortal has been installed. This is due to the member and usergroup IDs ocPortal uses being tied to the member and usergroup IDs of the linked forum. Because the actual usergroup and member data is not held or managed by ocPortal, ocPortal is entirely sensitive to changes to this data, yet has no way of properly detecting or synching against changes made to it.All this said, it is possible to move between forums, if manual database changes are made to correct the aforementioned problems.
Specifics of importers
ocPortal merge
When merging with another copy of ocPortal, you should make sure the other copy must be running the same major version. Whilst other versions may merge succesfully, ocProducts does not support this officially.The 'ocp merge' importer can merge multiple ocPortal websites together that either:
- each run on OCF (and thus, OCF data gets merged)
- or, each share a forum database (what we call a "multi-site-network" situation)
The 'ocp merge' importer cannot:
- Work with anything other than ocPortal data (third-party forum data can not be merged, for instance)
- Merge an ocPortal site into an ocPortal site that does not use the same forum database
- unless you are highly technically proficient and capable of manually changing member and usergroup IDs, using a tool such as phpMyAdmin (because these IDs could not be mapped correctly for data that used a 'foreign' forum)
- or unless both sites run on OCF (because in this situation, the importer can import everything, and correctly remap any member and usergroup IDs)
- Import OCF data directly into a third-party forum
- because the imported data would end up in the OCF database tables, regardless of whether they are currently being used for the ocPortal site's active forum.
Remember:
- you must specify to import from an ocPortal database, not a forum database.
Please note that URL and page-link links will not be altered during the import, meaning it is likely they will need updating (because resource IDs change).
phpNuke
Tip
Many products have been 'forked' from phpNuke ('nukes') and may be compatible with this importer, with minimal or no changes. Products include: e107, Xoops, PostNuke, OpenPHPNuke, and many others.
Compatibility notes:
- ocPortal has a combined news system, so journals, editorials, stories, news, and reviews are all imported together. (When I wrote the phpNuke importer, I was surprised what a 'copy&paste mess' phpNuke was in this area.)
- The encyclopaedia is imported into CEDI, which is much more powerful, but different.
- FAQS get placed in a catalogue, as do Links, ephemerides, and contacts.
- Pages are turned into Comcode pages, with the HTML embedded in an HTML tag. They are are indexed through the pnindex page.
- Statistics are not imported.
- Permissions are not imported, as phpNuke doesn't have a permission system anything like ocPortal's
- You need to copy the current news category (topic) images into the themes/default/images/newscats directory
- No files are moved to the ocPortal directory: this needs doing manually.
Discussion Forum importing
Compatibility notes in general:- personal/private message will be glued together to form personal topics. This is a very useful feature, and really cleans up an inbox.
PhpBB
Compatibility notes:- phpBB does not support attachments by default, but there is an attachment importer. Only use this if you have the popular attachments mod installed and running. Attachments are moved to the ocPortal attachment upload directory.
- phpBB uses a very strange usergroup configuration, so it is necessary to check your usergroups, permissions and usergroup membership after import. Forum permissions will not import properly.
vBulletin
Compatibility notes:- the vBulletin calendar recurrence system is very different to the ocPortal calendar recurrence system, so recurrences may not be imported perfectly
- forms of rating, such as topic rating, karma, and 'goes to coventry', are not imported. However reputation is imported as points.
- Many vBulletin will not be imported
- attachments, photos and avatars are extracted from the database to the appropriate ocPortal uploads directory. It is best to use the live database for the import, because there is a mySQL/vBulletin bug in some mySQL versions that causes binary database data to be corrupted in SQL dumps.
Invision Board
Compatibility notes:- Many Invision Board options will not be imported
- attachments, photos and avatars are moved to the appropriate ocPortal upload directory
HTML website importer
The HTML website importer is a special importer that can import an HTML site that is stored on disk. It is designed for migrating existing static HTML websites into ocPortal.The importer will try and establish what your HEADER, FOOTER and GLOBAL templates should be, but it cannot be perfect at this. It is also not able to extract panels or menus in a particularly clever way (they all go as static markup in the aforementioned template files), so you should consider your imported site as a base that will require some cleaning.
If you do not have access to the files of your site, other than from the live server, you can download a website using the 'wget' tool. This tool exists on most Linux installs by default, but can also be installed for Mac and Windows.
You run wget using a command like:
Code
wget -nc -r <yoururl>
The HTML website importer will try to do the following:
- Create zones
- Create Comcode pages
- Copy over PHP files as pages (mini-modules)
- Create the HEADER/FOOTER/GLOBAL templates
- Try and fix links and file paths to be workable ocPortal links
- Copy over other files that are referenced (such as image files), to underneath uploads/website_specific, and fix the URLs accordingly
- Work out your website name
- Meta keywords and description, for each page
The importer uses a sophisticated algorithm to detect what your header and footer is. It isn't 100% perfect however (it is very CPU intensive, and may lock onto markup similarities between comparison pages that should not be universal). If you have a header.txt and/or footer.txt file in your source directory, the importer will consider these the header/footer instead, an use them when it comes to stripping down the pages.
After importing
If the importer you used copied all relevant files, like avatars, photos and attachments, into ocPortal's directories, then you can remove the imported product directory in whole.However, it is advisable to keep the directory, database, and import session, around for a few weeks – just in case any data was not correctly imported and extra maintenance required to put things right: importing is a technically complex process, so it is always best to keep your doors open.