HTML Logo by World Wide Web Consortium (www.w3.org). Click to learn more about our commitment to accessibility and standards.

Moving forward with Composr

ocPortal has been relaunched as Composr CMS, which is now in beta. ocPortal 9 will be superseded by Composr 10.

Head over to compo.sr for our new site, and to our migration roadmap. Existing ocPortal member accounts have been mirrored.


SMF to OCF :)

Login / Search

 [ Join | More ]
 Add topic 
Posted
Rating:
#95331 (In Topic #18973)
Avatar

Fan in training

want to transfer all my posts and users to OCF

Greetings all,

I discovered OCF just a while ago, and have been testing it since! And all I can say is excellent job. I even made some modifications and tried to create some stuff on my own. Really a CMS that I can honestly say good job!

Now, I have a rather large forum with more than 10 000 useres and about 350 000 posts per year. I was wondering if OCF can handle that much? And if it can, is there any way to transfer all my smf2.0.2 forums to OCF?

I didn't find any other solution but transfering my smf to another forum like phpbb and then transfering it to OCF. But I really don't wan't to do that.

And if it isn't possible, how well do ocPortal and SMF work together after installaton of ocPortal? Can smf users log in to ocPortal? When they comment a article, can their avatar be shown in comments section of the article? Where does the link of users that posted a comment lead? To oc user page, or to SMF user page?

Thank you in advance for your replies, looking forward to using ocPortal. Sooner the better :)
Back to the top
 
Posted
Rating:
#95341
Avatar

Hi,

350k posts per year, wow that's one active forum.
I wonder how many in total?

I am more concerned with how long it would take to import. When I created 100's of thousands of test records for our support of huge sites, it literally took days just to create that much random data, and putting in random records is computationally simpler than translating them. I could imagine having to leave the importer running for days which obviously would be a huge PITA if it failed (and the more data there is, the more chance of some inconsistency in it breaking the import). I don't know if you have any funds, but I would recommend getting a programmer to run the import; I think ideally for this much data some kind of script would be written that could run through a terminal (using the Linux 'screen' command).

I'm slightly concerned about performance with that amount of data, but if there was an issue I'd help resolve it.

We do have an SMF2 importer in ocPortal now, courtesy of Duck.

We do support forum integration too (shared logins, avatars, profile links, etc), but we don't like it much as it's a poor user experience to span two worlds like that (unless you spend a lot of time themeing).


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#95347
Avatar

Fan in training

Well, last night I saw importer option so I started importing. It's really slow, and with really slow, I really mean really slow :)

Already had to continue session for at least 5 times.

I'm doing it locally, and I am a programmer :)

Forum has about 500 000 posts, but it's a new forum and it's growing every day. Previous forum on another domain had about 2milion posts in 5-6 years.

I'm gonna try to import it. I tried combinig oc with smf, but I failed. Right after installation of OC and choosing smf as my forum, I can't login with any username.

Probably problem with cookies, but I didn't even try to work it out. First I'm gonna try converting, if that fails, then I'm gonna try using oc with smf.

But I'd rather use ocf, since templating is so much easier. I hate smf templating :)
Back to the top
 
Posted
Rating:
#95350
Avatar

Okay, if it were me, I'd be using phpMyAdmin to see how many rows are in the table for whatever is importing, to get a sense for how it's going. e.g. look at the number of ocp_f_posts.

It should auto-refresh on its own without need for an explicit continue session, so long as the refresh time option given (I think that's what we call it) is low enough to avoid some kind of timeout happening first.


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#95352
Avatar

Fan in training

Problem is I get errors while importing. Can't remember what they were since I'm really not tracking records. Just testing importer out locally.

If I set refresh time to 5secs or 10, I get those errors. But since I'm in no hurry, I have set the refresh time to be 40secs.

And atm 198080 posts have been transfered. So it seems so far it's working like a charm. I'll leave it be for a day or even two and see does it actually work :)
Back to the top
 
Posted
Rating:
#95353
Avatar

They may be in the error log (data_custom/errorlog.php).


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#95355
Avatar

Btw, as a programmer you may want to know the import_id_remap table in ocPortal is really key to the process. It is this that knows that say topic #34 on SMF has been imported to topic #30 on ocPortal. That tracks the links for fixing relationships (so an imported post knows what the topic ID must be set to), but it also tracks progress and is what allows resumption (if something has been marked as mapped, it implies it has been imported).


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#95356
Avatar

Fan in training

Indeed they are. And a lot of them, about 50+ :)

[15-Feb-2013 16:14:58 UTC] PHP Fatal error:  Maximum execution time of 50 seconds exceeded in /Applications/MAMP/htdocs/reboot/sources/hooks/modules/admin
_import/smf2.php on line 1139


I'll see if this continuing sessions work. If it doesn't, I'll try importing when I have the time to monitor the import session.

Atm I just left it on my mac to run and I check it when I get a chance to do so. But I guess it'll have to do for now :)
Back to the top
 
Posted
Rating:
#95357
Avatar

Fan in training

And thx for the advices :)
Back to the top
 
Posted
Rating:
#95359
Avatar

Actually, try putting in as large a number as possible to the refresh time. I just examined the code and I can see it actually uses PHP timeouts intentionally (I'd forgotten this). The way it works is a timeout triggers a PHP exception, and ocPortal handles the exception with a meta refresh. This is done because we'd rather trigger a timeout we can catch, than let something else do the timeout (e.g. the CGI gateway on some servers).

You don't really want it to timeout, so it is better to tune your own settings if some other timeout is happening. Timeouts aren't good because it's having to skip over what is already imported each time and that will take longer and longer the further it gets into the process.


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#95361
Avatar

Fan in training

I have been increasing it every time I got an error. atm it's been on 70, and since then no errors occured at all.

I'll report here when the conversion has finished :)
Back to the top
 
Posted
Rating:
#95362
Avatar

I would try something like 99999999999.

If you've imported say 300k posts, I think it is going to do 300k remap table queries while skipping existing posts, before it actually resumes any real work. That could potentially take longer than the 70s refresh time, so it could just stall completely.


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#95514
Avatar

Fan in training

Well, I've given up on trying to convert. No errors, but it just takes too long. I'm gonna try converting some other time when I get a chance to write my own converter.

I'm gonna try using ocPortal with smf. But I have a problem there as well. After installing ocPortal and setting smf as my forum software, I can't login to ocPortal at all. Checked cookies and all, and everything is ok, but login simply doesn't work. I can login to smf without a problem.

Checked logs for errors, none are shown.
Back to the top
 
Posted
Rating:
#95522
Avatar

Community saint

I am not at home and unable to look into this at all right now but a quick thought came to mind that I will pose for Chris to add to if possible.

Chris would it be possible to modify the remap situation to perhaps store (prob best in another table of course) the latest mapped post and instead of checking if each post has been mapped coding in a continue from last mapped post type thing?

It has been awhile since I studdied the importer code so this may not be possible but if something like this is then it would save the 300k queries from happening. It may have the side effect of a less acurate importing (possible some missed imports) but may make very large imports more managable? I dunno just thinking out loud here?
Back to the top
 
Posted
Rating:
#95531
Avatar

Fan in training

If that is even possible, it would be great. I would gladly convert my forum to ocf and have one admin panel for everything, one templating for everything, and one API. SMF is a great peace of software, but their templating is really awfull :(
Back to the top
 
Posted
Rating:
#95577
Avatar

I'm gonna try using ocPortal with smf. But I have a problem there as well. After installing ocPortal and setting smf as my forum software, I can't login to ocPortal at all. Checked cookies and all, and everything is ok, but login simply doesn't work. I can login to smf without a problem.

Is it a normal SMF password, or maybe the password was set in some other software then imported into SMF? ocPortal might not support the hashing schemes that SMF might have for legacy passwords. Or maybe it's an old version of SMF that set that password? Does it work on a new account? SMF 1.1 switched to standard sha1 password hashes, and we tested in SMF 2 RC's, I guess it is possible they've switched to something else somewhere.


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Important!
Posted
Rating:
#95578
Avatar

Automated fix message

c0ppo said

Well, I've given up on trying to convert. No errors, but it just takes too long. I'm gonna try converting some other time when I get a chance to write my own converter.

I'm gonna try using ocPortal with smf. But I have a problem there as well. After installing ocPortal and setting smf as my forum software, I can't login to ocPortal at all. Checked cookies and all, and everything is ok, but login simply doesn't work. I can login to smf without a problem.

Checked logs for errors, none are shown.
This issue has been filed on the tracker as issue #1057, with a fix.


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Important!
 
Posted
Rating:
#95579
Avatar

Chris would it be possible to modify the remap situation to perhaps store (prob best in another table of course) the latest mapped post and instead of checking if each post has been mapped coding in a continue from last mapped post type thing?

It would need to store its position through the result set to do that properly efficiently, which is not something we currently store or have anywhere to store, or start against. It's also a continuation on a path I'm not happy with, so my approach above is quite different and is to get it all done in one (non-web) request.


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#95580
Avatar

Okay c0ppo,

To keep you on board (or on forum? lol) I made a substantial effort to help you out here.

I wrote an OcCLE command to finish off an import. What you do, is you start an import as normal, but when you get to the screen where you choose what content to import, you stop there, then bring up a normal command line on the server.

Of course, what a command line is really depends on your environment. If you're running on Windows, it would be a command prompt. If you're running on Windows but have a Linux server, it might be a PuTTY session. If you're on a Mac or Linux, it'd be a terminal.

You'd cd into the ocPortal dir and type…

Code

php data/occle.php

This tunnels you into an actual command-line version of OcCLE, rather than a web version.

You can then type…

Code

continue_import "<password>"

where password is the database password for the database you're importing. (ocPortal needs this because it hasn't stored it in the import session for security reasons)

In theory the import will then work to completion, and you'll be doing it from a far less flaky environment.

This is all of course very experimental.

And I have turned off MySQL index maintenance during import, as I am aware this can slow down importing a lot too.

If you have a remote Linux server, and have 'screen' installed, you'd actually start OcCLE with:

Code

screen php data/occle.php
That makes it resistent even if your own internet connection dies: screen makes a command's execution not die if your terminal session does.


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#95591
Avatar

Fan in training

Well, all I can say is thank you for your assistance. I'm gonna try importing again later this night using your advices and afterwards I'm gonna report here with the status of import.

Thanks Chris :)
Back to the top
 
1 guests and 0 members have just viewed this: None
Control functions:

Quick reply   Expand