HTML Logo by World Wide Web Consortium (www.w3.org). Click to learn more about our commitment to accessibility and standards.

Moving forward with Composr

ocPortal has been relaunched as Composr CMS. ocPortal 9 is superseded by Composr 10.

Head over to compo.sr for our new site, and to our migration roadmap. Existing ocPortal member accounts have been mirrored.


Fatal error: Allowed memory size exhausted

Login / Search

 [ Join | More ]
 Add topic 
Posted
Rating:
#111970 (In Topic #22533)
TQ
Avatar

Honoured member

Hi,

I have just received the following error whilst adding a download using URL> Copy to Server> where the source and destination are the same machine and the file size is 367MB:

Code

Fatal error: Allowed memory size of 67108864 bytes exhausted
(tried to allocate 31461376 bytes) in
\sources\files2.php on line 799
Whilst this is annoying, it could also be a blessing as it may have exposed the cause of an ongoing problem the site has been suffering for months namely, files over 300-400MB fail to upload directly citing nothing more than 'HTTP error' (the servers limit is 2GB). I've had to provide a separate site for members to upload large files to the server having failed to resolve this problem within ocP.

My research suggests that this is a cURL issue and I should increase php's memory allocation.

This in turn prompts me to ask how ocP is overriding my current php.ini memory_limit=128M with a value of 64M. I have, long ago, tried to do this as I have plenty of memory headroom. I have no .ini overrides in the site folders but have failed miserably to make any headway.

This specific Fatal error problem hasn't occurred before but I did notice that the incoming bandwidth of the server was high at the time suggesting that someone else was in the process of uploading a large file to the site (presumably consuming php memory).

I would be eternally grateful to anyone that could help resolve the upload limitation or shed some light on the current issue.

TQ
Back to the top
 
Posted
Rating:
#111971
Avatar

Ergh. Ultimately this problem is because I never expected huge files to get downloaded over https, due to the overhead of that.

But that's changed now for obvious reasons.

Upping the memory limit would not be a good approach. We intentionally keep it at a low level for the vast majority of cases to stop over-runs. Raising it here would introduce the ability to crash a server by copying a huge file.

There are a couple of solutions.

1)

We could disable the cURL downloader for when files are getting streamed to disk and let it use the PHP streams downloader instead. That's not a perfect solution as some servers have PHP stream downloading disabled. It'd probably work for you though.

Change:

Code

$use_curl=(($url_parts['scheme']!='http') && (function_exists('curl_version'))) || ((function_exists('get_value')) && (get_value('prefer_curl')==='1'));
to:

Code

$use_curl=((($url_parts['scheme']!='http') && (function_exists('curl_version'))) || ((function_exists('get_value')) && (get_value('prefer_curl')==='1'))) && ($write_to_file===NULL);
in sources/files2.php.

2)

I'll have to significantly rewrite the cURL downloading code to separately stream headers and content, sending content direct into the output file.
I'll do that but it'll take me a while.


Become a fan of Composr on Facebook or add me as a friend. Add me on on Twitter. Support me on Patreon
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about Composr whenever you see the opportunity or support me on Patreon.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying Composr on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#111974
TQ
Avatar

Honoured member

Thank you Chris,

I have just made the change and are uploading a 500MB file (it will take some time from this location) and will report back.

Chris said

I'll have to significantly rewrite the cURL downloading code
... and I will gladly put some hard cash towards your efforts. This is a significant problem to me and a solution would make my day, in fact, it would make my year!

Talking of cash, there are a couple of equally significant (to my site) changes I would like to have made to the 'Join' process. How do I go about outlining these changes and getting a price?

TQ
Back to the top
 
Posted
Rating:
#111975
Avatar

Hi,

Thanks for the offer but I would never want to solicit money for bug reports, I consider that basic service to all our users :).

Fortunately the fix wasn't as hard as I thought. My experience with debugging cURL has been a nightmare, but that's usually due to SSL stuff. In this case I was able to rework how the data transfer code works relatively easily. Previously it all came in through one string (hence memory issue), now it will stream into the file being downloaded to for the kind of case in this issue.
The only annoying thing is I have to change stuff 3 times – one for v9, one for v10, and one for v11, as each has divergent code at this point.

Attachment
sources/files2.php
» Download: files2.php (47 Kb, 50 downloads so far)


Regarding your custom development - you can open up a support ticket, either on here or on the compo.sr site. I know the v10 upgrade is a stresser for you, maybe you can consider asking us to do the upgrade. As long as you don't have dozens of overrides it probably can be done in around half a days work.

Regards,
Chris


Become a fan of Composr on Facebook or add me as a friend. Add me on on Twitter. Support me on Patreon
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about Composr whenever you see the opportunity or support me on Patreon.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying Composr on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#111976
Avatar

I would appreciate if you could test this particular code ASAP, as I am going to release the v10.0.1 release soon and I want to have a second opinion on the changes. I did write new unit tests which passed before the change and after, but I'd like a little extra confidence on this.


Become a fan of Composr on Facebook or add me as a friend. Add me on on Twitter. Support me on Patreon
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about Composr whenever you see the opportunity or support me on Patreon.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying Composr on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#111977
TQ
Avatar

Honoured member

I've just added the new code and will test it now.

Will reply to the rest shortly.

TQ
Back to the top
 
Posted
Rating:
#111978
TQ
Avatar

Honoured member

Let me start by saying that this may be nothing to do with ocP but something to do with my server setup.

Both the code you first provided and the new files2.php have not resolved my upload problem.

The original change allowed about 20% of the file to be uploaded before the progress bar reset and started climbing again. It went to approximately the same position before announcing that the upload was "Completed" but when I clicked the "Add a new Download" button the page just scrolled up to the File: Upload position. I left it there for some time but it was game-over.

The exact same thing is happening with the new files2.php except it reaches about 25% of the 500MB upload.

[The reason I mentioned it may be my (servers) fault is that the second time around I checked the clock and it looks like the upload resets after 15 minutes. My upload speed is abysmal (I'm on a land microwave link) at about 200kB/s]

After going through the cycle of resetting and then stating 'Completed', if I change the File:Upload to a different/smaller file nothing happens. No progress bar and no chance of clicking the "Add a new Download" button which is now greyed out.

I next refreshed the page and tried a 150MB file, it ran for about 14 minutes then reset the progress bar.

I again refreshed the page and tried with an 85MB file which worked like a charm having completed in 12 minutes.

I know that members with a fast internet connection can upload at least 300MB files (and 1.5GB files to FileRun on the same IIS server) without issue. If time is the issue, I don't know if it's ocP or the server. I have trawled the internet over the past few months and read every reference to up/download fails and made every IIS adjustment documented, to try and overcome this problem.

There is nothing in the PHP or IIS log to give me any pointers.

So, to answer your question, the truth is, I can't. I am no longer in a position to remote to another machine with a fast connection to see if time is the factor. If you are interested to have a go yourself, be my guest. You have an account and there is an Admins download section below downloads. If you don't validate the DL's, nobody will be any the wiser.

What I can say is that the new files2.php file doesn't appear to have caused any new issues for members. The stats for the last hour suggest that it's business as usual :)

With regard to you migrating my v9 site to v10, that sounds like a good plan. As to the overrides, a goodly amount are not customer facing but I'll have to live with it going pear shaped for a while (if you can). It will be a dogs dinner but it's a friendly dog :lol:

TQ
 
Back to the top
 
Posted
Rating:
#111980
Avatar

I'll run some tests.


Become a fan of Composr on Facebook or add me as a friend. Add me on on Twitter. Support me on Patreon
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about Composr whenever you see the opportunity or support me on Patreon.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying Composr on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#111981
Avatar

Hi,

The lack of progress bar on 2nd upload attempt was a bug. It was actually uploading, just with no progress bar. I've fixed it now.

I was able to upload a 453.9MB file to your site. My first attempt failed at around 756 seconds with a 500 error from IIS slightly before it had finished; my second succeeded as it completed slightly quicker.

This may be your problem:
https://forum.owncloud.org/viewtopic.php?t=18404
(just a guess based on my own quick Googling)

I don't think it would be a PHP or ocPortal problem, PHP is only called once the request has come into IIS.


Become a fan of Composr on Facebook or add me as a friend. Add me on on Twitter. Support me on Patreon
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about Composr whenever you see the opportunity or support me on Patreon.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying Composr on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#111982
TQ
Avatar

Honoured member

Hi Chris,

Chris said

This may be your problem:
https://forum.owncloud.org/viewtopic.php?t=18404
What a great find! It was only yesterday that I noticed it was a 15 minute time lapse because I previously depended on members reports which suggested it was 30 minutes but it seems to have done the trick (for me at least). Thank you so much for that.

Thank you also for all the other work you have done towards this problem. If you'd like to point me in the direction of a feature that needs some sponsorship, I'd gladly oblige.

In case it would help anyone else, or the above URL is not available in the future, this is a detailed version:

IIS Manage> Host Computer> FastCGI Settings> php-cgi.exe> Edit> Activity Timeout> 3600 (Default 600)
IIS Manage> Host Computer> FastCGI Settings> php-cgi.exe> Edit> Request Timeout> 3600 (Default 600)
IIS Manage> Host Computer> FastCGI Settings> php-cgi.exe> Edit> Idle Timeout> 600 (Default 300)

Restart IIS (iisreset.exe)

TQ
Back to the top
 
Posted
Rating:
#111983
Avatar

Great, I'm glad it's fixed. So actually it was 3 issues - IIS config, a plupload addon progressbar issue, and an HTTPS memory issue with our cURL downloader.

I'm going to add a note in the Composr FAQ about those IIS settings.

I'd definitely save your cash for getting upgraded to v10 TBH, but I do have a list of things I think are ripe for sponsoring for anyone reading…


These are things not on my personal list pf priorities but I think would really make people happy.


Become a fan of Composr on Facebook or add me as a friend. Add me on on Twitter. Support me on Patreon
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about Composr whenever you see the opportunity or support me on Patreon.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying Composr on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
Posted
Rating:
#111984
TQ
Avatar

Honoured member

There is one I'd like to throw a few coins at, how do I go about this?

TQ
Back to the top
 
Posted
Rating:
#111985
Avatar

Hi,

Here's the process :):
https://compo.sr/contact/sponsor.htm


Become a fan of Composr on Facebook or add me as a friend. Add me on on Twitter. Support me on Patreon
Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about Composr whenever you see the opportunity or support me on Patreon.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying Composr on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Back to the top
 
There are too many online users to list.
Control functions:

Quick reply   Contract

Your name:
Your message: