HTML Logo by World Wide Web Consortium (www.w3.org). Click to learn more about our commitment to accessibility and standards.

Moving forward with Composr

ocPortal has been relaunched as Composr CMS, which is now in beta. ocPortal 9 will be superseded by Composr 10.

Head over to compo.sr for our new site, and to our migration roadmap. Existing ocPortal member accounts have been mirrored.


Catalogue limitations

Login / Search

 [ Join | More ]
 Add topic 
Posted
Rating:
#107193 (In Topic #20895)
Avatar

Fan in action

Going round and round with these catalogues. I see limitations listed on the optimization tutorial so I'm trying to work around it but ran into another wall. Tutorial says OCP has been tested to a million in quantity of various things including catalogs however only a few hundred should have entries. Ok. My site would end up with a thousand plus catalogs done in one way and naturally they would all have entries at some point to make them actually useful. Structured a different way I could do it with 20-30 or so catalogs by using categories and one level of sub categories. Made up a text document with all my pipe symbols and backslashes to create the categories the fast way, tried to paste it into the "Auto-fill with categories" field when creating a catalog and evidently that field has a character limit. 10,000 or so seems to be the limit. Maybe 9,999. Anyone know if there's a way to change it temporarily? I need more like 55,258. Is a catalog with 51 categories and over 3000 subcategories beyond the catalog system limits? This would be done for 20-30 catalogs. The entries themselves don't amount to much. It's almost all "brief line of text with no comcode". One or two words per field for the most part.

btw there seems to be a typo in the catalogues tutorial. In the sample for adding categories quickly.
Canada\Newfoundlandand Labrador|Canada


should be Newfoundland and Labrador. Missing a space.


Last edit: by John Pollard
Back to the top
 
Posted
Rating:
#107197
Avatar

Thanks for pointing out that typo.

To be frank, anyone trying to push catalogues super-hard in order to avoid a programmer, is probably going down the wrong path. I highly advise reading the "what ocPortal is not" page which tries to spell out the practical realities of how to work with ocPortal (or any CMS really):
What ocPortal is not - ocPortal

We have indeed tested catalogues with huge amounts of data (along with the rest of ocPortal), but that doesn't mean anyone should practically do it. I think almost certainly you're going to soon be thinking about data relations, and custom editing interfaces, and efficient searching – all of which catalogues are not a good solution for. This kind of thing needs proper database design, with an understanding of normalisation and linker tables, and indexing, and efficient query patterns. Another issue is if you later want to add some custom code features for interacting with catalogue data, it is a lot harder programming with the level of abstraction that catalogues have. Another issue is a single catalogue entry can easily have 20 database rows associated with it (the number of fields plus one) – that's a lot in the database.


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#107198
Avatar

Fan in action

Ok, Thanks
Back to the top
 
Posted
Rating:
#107200
Avatar

I've put together a blog post to try and better highlight my thoughts on this kind of thing:
How to approach complex projects - ocPortal.com

It comes up a lot, so I wanted to properly describe my point of view and hopefully not come across as too much of a curmudgeon in the process.


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#107213
Avatar

Community saint

@John, Based on one of your previous posts, would it be a safe assumption you want to have some sort of database of recipes? A couple of options you might have other than using catalogues would be to find an existing recipe database manager and integrate that into ocPortal or write your own stand-alone script that does the job and then integrate that into ocPortal. Depending on the script you use, what type of permissions you need, and how tightly integrated you want it in ocPortal, that will determine the level skill you need to accomplish it on your own. If that is something you want to look into, you should check out the tutorial that covers the integration of external scripts and applications:

ocPortal.com - ocPortal Tutorial: Integration of ocPortal and other installed scripts/applications

You might also find the Making an Addon tutorials parts 1 and 2 useful for integrating external scripts into ocPortal:

ocPortal.com - ocPortal Tutorial: Making an addon (part 1)
ocPortal.com - ocPortal Tutorial: Making an addon (part 2)

Part 1 gives an example of creating a mini-block and part 2 gives an example of creating a mini-module. Both can be done without having to know much about the ocPortal API. By going the 'mini' route, you can get started more easily by not having to know how to use and implement the templating, language, caching, installation and removal functions.

If you still feel catalogues will suit your purpose, (and at the risk of me giving you more rope here) the 10000 character limit on the Auto-Fill form field can be changed in the cms/pages/modules/cms_catalogues.php file. On or near line 1767, you will find this line:

Code

if ($name=='') $fields->attach(form_input_line(do_lang_tempcode('AUTO_FILL'),do_lang_tempcode('DESCRIPTION_AUTO_FILL'),'auto_fill','',false,NULL,10000));

Just change the 10000 at the end of the line to the number you need. Be aware, I haven't tested it with a higher limit so there may be other limitations what may still cause it to fail and there still may be other areas of code that might need to be changed as well. Also, you will want to make this edit using the ocPortal Code Editor (AdminZone>Tools>Code Editor) or, if you do it manually outside of ocPortal, be sure to upload the changed cms_catalogue.php file to the cms/pages/modules_custom directory instead of overwriting the existing modules/cms_catalogues.php file. Once you have created all of your catalogues, it would be best to go in and delete that modules_custom/cms_catalogues.php file since it will completely override the existing modules/cms_catalogues.php file. Not deleting the override when you are done will prevent your site from benefiting from any enhancements or bug fixes to the original file.
Back to the top
 
Posted
Rating:
#107223
Avatar

Fan in action

Hi Jason, Thanks again. I'll probably not go the route of that many categories and subcategories after setting up a smaller sample and adding a few entries. The menu tree to choose where the entry should go would be huge with over 3000 categories and subs. Mistakes waiting to happen as it allows entries to be put in higher up the tree than where they should go. This is for genealogical data. surname, given name(s), gender, dob etc. I'll probably go with many smaller catalogs but make them as needed, not 1000+ all at once. This will get us by until I can gain the knowledge to do something more proper. Reading the comparison of catalogues to MS access in the catalogue tutorial lead me to believe that catalogues would be a good fit but naturally anything custom tailored will be a better fit. We may have old recipes at some point but I figured the wiki+ would be a good candidate for that. I'm still working out the overall structure of the site and what we will have for content. I'm still finding out about new content types all the time. LOL My sister lives 1000 miles away and she's the genealogist. Started with our family tree and became a hobby.

This site's not going to be live anytime soon so if I break things while learning, it's not a big deal. I'm sure Composr will be out long before we even think about opening the site. It's mostly our hobbies put together. I dabbled in web design years ago and am wanting to pick it up again. I thought ocp would be a good candidate to learn on and with all the warm welcomes at the top of the forums, I thought I could ask my stupid questions here and not tick anyone off. Of course when I dabbled in it before, all you had to know was cpanel, html and this new thing called css and you were a webmaster. Webmonkey was new and all cutting edge stuff. Somewhere I still have the very first edition of the webmonkey html cheatsheet I printed out. Wonder if it's worth anything? Ebay. I've built a few sites with wordpress, plugging things in. Tried out all the other big ones, Joomla, Drupal etc but didn't care for them. Barebones frameworks like zend seem to be the thing for some cases but would be too much to try and build something along the lines of ocp from scratch but I'll probably do the tutorial on a couple of frameworks just to learn. Tikiwiki could be something but for all the half finished modules and outdated documentation. EdX has a series of courses on SQL and that's definitely on my to do list. And of course php. Hard to find courses for that but plenty of tutorials. I'm not sure what's behind OCP's search so there is the possibility of needing to install solr or lucene and learn that. Quite a bit to it from my reading.
It may take a few years but I'll get there. It's only time invested, I've done enough to cover hosting, I'm learning and it's probably a better thing to do with my spare time than watching Simon Cowell's latest creation. For now my only goal is to get it so I can upload my sisters data and have it searchable so she can build the tree with it. This is all coming from print and excel is what she knows so she's transcribing into that. Hundreds of books all out of copyright. She types pretty fast I guess but hopefully she won't blow up the database all by herself for a while and that will give me time.
Back to the top
 
Posted
Rating:
#107224
Avatar

Learning SQL and PHP could well be a good idea :). You can then get full control, with all the benefits of ocPortal's various framework features and other features.


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#107274
Avatar

Fan in action

Had a look at how the catalogue entries are stored in the db and I have a better understanding now. Was not what I expected. I thought the catalogue system would actually create tables. I can see why I'll need to do something else.
Back to the top
 
Posted
Rating:
#107276
Avatar

Yeah, hand-designing is always better for a system that has more than basic requirements. If we actually did table management, we'd come across unexpected things that would trip people up instead, like column or indexing limits on different database backends, or the site potentially locking down for a few minutes whenever a new field was added (while MySQL would rewrite all the record structures). Automated/abstracted systems always have some degree of compromise.


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
There are too many online users to list.
Control functions:

Quick reply   Contract

Your name:
Your message: