cassandra - crash in compaction using lucid ppa package

Bug #725470 reported by Robert Collins
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cassandra-packages
Won't Fix
Critical
Clint Byrum
apache (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

ERROR [CompactionExecutor:1] 2011-02-26 21:07:35,264 AbstractCassandraDaemon.java (line 91) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.NoClassDefFoundError: org/apache/commons/collections/PredicateUtils
        at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:305)
        at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
        at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)

Related branches

Revision history for this message
Robert Collins (lifeless) wrote :

(I've filed this here because Clint is preparing to upload this to Ubuntu itself and needs to know its already got bugs:)

Revision history for this message
Robert Collins (lifeless) wrote :

possibly related:

 INFO [FlushWriter:1] 2011-02-26 22:26:59,432 Memtable.java (line 155) Writing Memtable-Schema@431432310(2346 bytes, 5 operations)
ERROR [FlushWriter:1] 2011-02-26 22:26:59,433 AbstractCassandraDaemon.java (line 91) Fatal exception in thread Thread[FlushWriter:1,5,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /var/lib/cassandra/data/system/Schema-tmp-e-61-Index.db (Too many open files)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException: /var/lib/cassandra/data/system/Schema-tmp-e-61-Index.db (Too many open files)

affects: ubuntu → apache (Ubuntu)
Changed in apache (Ubuntu):
status: New → Incomplete
status: Incomplete → Invalid
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

This is due to depending on libcommons-collections-java instead of libcommons-collections3-java

Fixing and pushing now.

Changed in cassandra-packages:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Clint Byrum (clint-fewbar)
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Robert can you try the version in the PPA now? Thanks!

Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 725470] Re: cassandra - crash in compaction using lucid ppa package

This specific error is gone, and a new one is here. This may be data
driven (I haven't deleted my cluster - in a rush this morning, will
fiddle this arvo for you).

 INFO [CompactionExecutor:1] 2011-03-02 05:28:28,582
CompactionManager.java (line 458) Compacted to
/var/lib/cassandra/data/system/Schema-tmp-f-244-Data.db. 600,198 to
599,871 (~99% of original) bytes for 220 keys. Time: 225ms.
ERROR [MigrationStage:1] 2011-03-02 05:28:28,589
AbstractCassandraDaemon.java (line 114) Fatal exception in thread
Thread[MigrationStage:1,5,main]
org.apache.cassandra.config.ConfigurationException: Previous version
mismatch. cannot apply.
        at org.apache.cassandra.db.migration.Migration.apply(Migration.java:115)
        at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:633)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
 INFO [MigrationStage:1] 2011-03-02 05:28:28,595
ColumnFamilyStore.java (line 665) switching in a fresh Memtable for
OOPS at CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1299004019135.log',
position=634361)

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

On Tue, 2011-03-01 at 18:27 +0000, Robert Collins wrote:
> This specific error is gone, and a new one is here. This may be data
> driven (I haven't deleted my cluster - in a rush this morning, will
> fiddle this arvo for you).
>
> INFO [CompactionExecutor:1] 2011-03-02 05:28:28,582
> CompactionManager.java (line 458) Compacted to
> /var/lib/cassandra/data/system/Schema-tmp-f-244-Data.db. 600,198 to
> 599,871 (~99% of original) bytes for 220 keys. Time: 225ms.
> ERROR [MigrationStage:1] 2011-03-02 05:28:28,589
> AbstractCassandraDaemon.java (line 114) Fatal exception in thread
> Thread[MigrationStage:1,5,main]
> org.apache.cassandra.config.ConfigurationException: Previous version
> mismatch. cannot apply.
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:115)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:633)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> INFO [MigrationStage:1] 2011-03-02 05:28:28,595
> ColumnFamilyStore.java (line 665) switching in a fresh Memtable for
> OOPS at CommitLogContext(file='/var/lib/cassandra/commitlog/CommitLog-1299004019135.log',
> position=634361)

There was something wrong with 0.7.0 thru 0.7.3 .. I've uploaded 0.7.3
just now to the stable PPA. Once it is available, can you try installing
it and running

nodetool scrub

Which should fix on-disk problems created by those versions. The bug
report mentions IOExceptions, which I don't see above, but it
specifically references compaction as the root of this problem, so it
would appear related.

https://issues.apache.org/jira/browse/CASSANDRA-2217

Changed in cassandra-packages:
status: In Progress → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.