openCMA for Linux

[codestation]

Come on people, don't convert this into another distro flamewar.

I fond a big problem with opencma. It leaks memory like crazy when doing a full restore. I am currently trying to restore a 3.2GiB backup onto my new 8GB card and the opencma process is over 2GiB of memory usage (and growing).

Advertising

[yifanlu]

Come on people, don't convert this into another distro flamewar.

I fond a big problem with opencma. It leaks memory like crazy when doing a full restore. I am currently trying to restore a 3.2GiB backup onto my new 8GB card and the opencma process is over 2GiB of memory usage (and growing).

Heh, I'll look into that. I never did a memory benchmark which I should have done. Can you also run it with tools to see where it's leaking?

Advertising

[SMOKE]

It leaks memory like crazy when doing a full restore.

Yea, I noticed that too.
I was making a 600 MB backup earlier, and after 2 hours, it wasnt even half way done, so I gave up

[yifanlu]

It leaks memory like crazy when doing a full restore.

Yea, I noticed that too.
I was making a 600 MB backup earlier, and after 2 hours, it wasnt even half way done, so I gave up

Are you running it with log enabled, because if you read the README, it says that running with log enabled makes it REALLY slow.

Come on people, don't convert this into another distro flamewar.

I fond a big problem with opencma. It leaks memory like crazy when doing a full restore. I am currently trying to restore a 3.2GiB backup onto my new 8GB card and the opencma process is over 2GiB of memory usage (and growing).

Are you sure it's a leak? By (***) design, I read the entire file into memory before sending it (for some file sends, others are broken up into parts). I didn't have restores in mind. Obviously, I should use mmap, which is what I'm doing right now. https://github.com/yifanlu/VitaMTP/issues/10

[SMOKE]

Are you running it with log enabled, because if you read the README, it says that running with log enabled makes it REALLY slow.

That was the problem, thanks

[wth]

yeah it aways read it all in mem but no pb I have 4G ram lol
mmap seems cool though indeed

[yifanlu]

Okay, I got a quick request for anyone who's willing to help. I don't have the time to make a backup and restore it on my Vita right now, so if someone would do the following:

1) Run opencma with -l 4 logging options
2) Proceed to restore a backup
3) Cancel the restore a few seconds after the progress bar shows up

and send me the log or post it here. I can write code better optimized for backup/restoring. (If the Vita doesn't allow canceling the restore, don't send me all 4GB worth of log data, I only need enough to see how the process works. I think 1000 lines of log would be enough)

[codestation]

Are you sure it's a leak? By (***) design, I read the entire file into memory before sending it (for some file sends, others are broken up into parts). I didn't have restores in mind. Obviously, I should use mmap, which is what I'm doing right now. https://github.com/yifanlu/VitaMTP/issues/10

Yeah, that should be. I just saw that comment:

Code: Select all

// TODO: Use mmap when sending extra large objects like videos

and the big memory usage made sense. I will post a log in some minutes (i can't right now since i am at work and shouldn't be slacking that much ).

Offtopic: At 95% i got a OOM error and the OS killed the opencma process. If the restore got interrupted i can assume that the vita contents are corrupted too and i should restore it to factory settings?


Edit: Log file with -l 4 during a restore (i stopped it at ~40MiB) https://dl.dropboxusercontent.com/u/289 ... ta.log.zip

[yifanlu]

Are you sure it's a leak? By (***) design, I read the entire file into memory before sending it (for some file sends, others are broken up into parts). I didn't have restores in mind. Obviously, I should use mmap, which is what I'm doing right now. https://github.com/yifanlu/VitaMTP/issues/10

Yeah, that should be. I just saw that comment:

Code: Select all

// TODO: Use mmap when sending extra large objects like videos

and the big memory usage made sense. I will post a log in some minutes (i can't right now since i am at work and shouldn't be slacking that much ).

Offtopic: At 95% i got a OOM error and the OS killed the opencma process. If the restore got interrupted i can assume that the vita contents are corrupted too and i should restore it to factory settings?

Hahaha, it actually was a memory leak. Two in fact. One in reading of PSP saves (and media) and another in sending of Vita/backup files. I indirectly allocated memory and forgot to free. should be fixed in next release. (No mmap needed because the only time it sends huge files are when you're sending huge PSP saves like the kingdom hearts data install file).

EDIT: Push committed. I'll wait another couple of days for other bug reports before releasing an update.

[codestation]

Now my Vita is asking CMA for a update, should i have the psp2-updatelist.xml file in my cma dir? Looks like the embedded xml is invalid since i get a C3-14978-9 error and it doesn't let me use the content manager.

Log file: https://dl.dropboxusercontent.com/u/289 ... g2.log.zip