HTML Logo by World Wide Web Consortium (www.w3.org). Click to learn more about our commitment to accessibility and standards.

Moving forward with Composr

ocPortal has been relaunched as Composr CMS, which is now in beta. ocPortal 9 will be superseded by Composr 10.

Head over to compo.sr for our new site, and to our migration roadmap. Existing ocPortal member accounts have been mirrored.


Speed Issue in catalogues with many entries

Login / Search

 [ Join | More ]
 Add topic 
Posted
Rating:
#57446 (In Topic #12492)
Avatar

Well-settled

Hello,

I'm currently importing several thousands of catalogue entries which is album reviews.

Unfortunately things seem to slow down extremely in the catalogue list display, whether it's tabular (which displayes only 4 one line fields (i.e album title, artist, rating and reviewer) or normal list (first two fields).

Right now I'm at 2000 entries and it takes around 2-3 minutes to load up the list. So I'm afraid it will be impossible to browse with 4000 entries. I think that the delay is because it fetches all data (including the review's main body) instead of just the 4 fields I've checked that should show up in tabular lists.

Is there any way to speed up this process or at least filter out the un-needed content from the list view?
Back to the top
 
Posted
Rating:
#57452
Avatar

Fan in action

Hi there,

I've had the same problem with a catalogue once it started going over about 500 entries. Maybe Chris could clarify a little on this but it seemed to me in catalogue.php that it was indeed looping through everything and retrieving all the information.

I ended up putting together a hacky custom solution for myself where there is basically a switch in there that checks if it's a certain catalogue and then only retrieves the necessary info but I've no idea how portable it is. If you like I can post it here and you could have a bash at it, it worked in my instance anyway.

I'd be curious to know if this is a problem other people are having. From what I saw going through my files when the problem came up it looked like something that would impact anybody once their catalogues got over a certain size. If that's the case maybe it should be noted as an actual bug.

Any thoughts Chris?

Palanthas, let me know if you want me to post the workaround I came up with.

Cheerio,

Andrew
Back to the top
 
Posted
Rating:
#57453
Avatar

Hi,

I am working on a fix now, which will be included in 4.3 RC3, that does sorting inside the database. We did not do it originally as the nature of catalogues makes it a bit tricky but this is what we are now going to do as it's the only way when large datasets are involved.


Become a fan of ocPortal on Facebook or add me as a friend. Add me on on Twitter.
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about ocPortal whenever you see the opportunity.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying ocPortal on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#57455
Avatar

Fan in action

Hooray! Good to know :) Looking forward to seeing it…the temp solution I have going works for now on the 3 catalogues I need it to but I can see it getting messy down the road so having something in there in the core would make me mighty relieved. Thanks for clarifying.

a
Back to the top
 
Posted
Rating:
#57456
Avatar

Well-settled

That's great to know Chris! Yet another reason to look forward for 4.3 :)

Andrew, I'd really love to see your implementation, would be great if you could post your fix, maybe it'll do the work till the new version is released. Thanks a lot in advance.  :thumbs:
Back to the top
 
Posted
Rating:
#57457
Avatar

Fan in action

Hi Palanthas,

I've copied in my catalogues.php in below. I'm in the middle of upgrading my own site to 4.2.2 and am lost in various details so to be honest with you I can't precisely remember which bits of this are working. I'd also not recommend implementing this without really going through it as I can't vouch at all for how this will work on somebody else's setup…wish I could be more helpful but up to my eyes with other stuff right now…

The crude fix is basically the function check_for_writing_category() at the bottom. Anywhere you see that being called in the code, something is happening to account for a big set of stuff. As far as I remember it's just isolating the entries and not requesting the additional info on them when it's a certain catalogue/s but you'd really need to mess around with it.

Sorry can't be a bit more specific…hope it helps you get to bottom of things. If I have a chance next week, I'll follow up on this with a bit of detail.

Cheerio,

a

Attachment
» Download: cata.php (47 Kb, 92 downloads so far)
Back to the top
 
Posted
Rating:
#57460
Avatar

Well-settled

Thanks a lot Andrew, I'll give it a look. Really appreciate it! :thumbs:
Back to the top
 
1 guests and 0 members have just viewed this: None
Control functions:

Quick reply   Contract

Your name:
Your message: