[mira_talk] Re: Lots of contigs, then segmentation fault
- From: Egon Ozer <e-ozer@xxxxxxxxxxxxxxxxxxx>
- To: mira_talk@xxxxxxxxxxxxx
- Date: Thu, 2 Jun 2011 10:37:17 -0500
Hey Bastien. I know you've got a lot on your plate, but I'm still trying to
find out why mira won't do hybrid assembly for me on OSX. If you'll remember,
when I run a bare bones (i.e. basic command line --project=EO
-job=denovo,genome,accurate,454,solexa -GE:not=8) hybrid 454 / solexa assembly
on my Mac, I wind up with 76814 contigs, none larger than 1745 bases. Running
the exact same command line on my university's server using the linux binaries
of the same version (3.2.1.17) I run out of disk space after the first pass due
to my being on a test account, but at that point I have 18498 total contigs, 88
of which are larger than 500 bp. The university server is busy and the queues
are long, so even if I can get a larger disk space allocation, that's not
really a preferable option since I have a perfectly good computer with plenty
of RAM and disk space and no queue sitting right on my desk that I think should
be capable of doing this.
Here's what else I've tried so far:
- Installed Ubuntu on a virtualbox, but the memory limit of virtualbox is 16G
of RAM and it keeps killing mira runs while trying to load the blocks after
skim, so that's no help.
- Built 3.2.1.17 from source on OSX instead of using the pre-built binaries,
just in case. Output from the hybrid run was nearly identical to the pre-built
binaries (big surprise, I know)
If it's any help, I was scanning through the log files from my last OSX run and
the partial server run. What I found was they start off exactly the same. The
first several pool statistics, deduced thresholds, and repeat ratio histograms
are identical.
There are a few minor differences after the skim:
OSX:
...
truncating EO_assembly/EO_d_tmp/EO_int_posmatchf_pass.1.lst from 5685491832 to
2526204024
truncating EO_assembly/EO_d_tmp/EO_int_posmatchc_pass.1.lst from 5601074328 to
2506863336
Skim summary:
accepted: 478708281
possible: 608170028
permbans: 0
Hits chosen: 470273590
...
Linux:
...
truncating EO_assembly/EO_d_tmp/EO_int_posmatchf_pass.1.lst from 5685493824 to
2526205752
truncating EO_assembly/EO_d_tmp/EO_int_posmatchc_pass.1.lst from 5601074688 to
2506864824
Skim summary:
accepted: 478708281
possible: 608170028
permbans: 0
Hits chosen: 470273688
...
There's further divergence in the output after this:
OSX output:
...
Total megahubs: 0
tcmalloc: large alloc 4294967296 bytes == 0x473800000 @
Can load up to 134217702 skim edges at once.
Partitioning into 4 blocks.
We have 419422280 skims in file.
...
Linux output:
...
Total megahubs: 0
System memory: 50620112896
Mem2keepfree: 7593016934
Used by MIRA: 14746963968
Mem avail: 28280131994
rsh increased memtouse to: 28280131994
tcmalloc: large alloc 13421522944 bytes == 0x38c9cc000 @
Can load up to 419422548 skim edges at once.
Partitioning into 1 blocks.
We have 419422548 skims in file.
...
There are some more differences in the block loading outputs after that, a lot
of which I'm assuming is due to mira not being able to use memory management on
OSX.
After that both log files go through "Filtering forward skims" and "Filtering
complement skims." After the next step, "Aligning possible forward matches,"
there are really divergent "Alignment stats":
OSX:
...
Potential: 2971963
Calculated: 40864
Evaded (PB): 0
Rejected (checkfun): 0
Trans 100 saved: 2931099
Banned overlap pairs: 2753 in 1673 sets.
...
Linux:
...
Potential: 18008746
Calculated: 2326435
Evaded (PB): 3
Rejected (checkfun): 0
Trans 100 saved: 15682308
Banned overlap pairs: 23229 in 15217 sets.
...
Similarly the output from "Aligning possible complement matches" is widely
divergent between the two. OSX soon goes on to kill an enormous number of
clusters (50,198 lines of "Killing cluster") versus Linux version producing 985
"Killing cluster" lines, ultimately culminating with OSX version moving 209187
reads to debris, Linux version moving 2188 reads to debris. The Linux version
then goes on to build a few long contigs, OSX a huge number of teeny-tiny ones.
Does any of this help you to figure out why OSX is going off the rails?
Any thoughts would be appreciated. Thanks.
- E
On May 15, 2011, at 3:03 PM, Bastien Chevreux wrote:
> On Friday 06 May 2011 15:27:29 you wrote:
> > ??????????????
> > Didn't work again!
>
> Got my DSL back, haven't forgotten you.
>
> Would you still like to run a couple of tests to see whether we can pinpoint
> the cause of the problem on OSX?
>
> Best,
> Bastien
>
>
>
Other related posts:
