eucalyptus-cloud: cloud fails to start on separate install

Sigh. Somehow I opened the bug with the incorrect apport file. Here is the /var/log/eucalyptus/*.

Reproduced.
Cloud-output shows:

17:27:39 ERROR 33 Exceptions | Fatal error occured: com.eucalyptus.util.Exceptions.uncatchable:29
17:27:39 ERROR 36 Exceptions | java.lang.Error: Failed to lookup required component: walrus

So it looks like in eucalyptus 2.0 the Walrus must be started before (or together with) the CLC. The Lucid autoregistration discussions made it clear the CLC must be up before the Walrus can be registered, so everything is designed to have the CLC installed before (the Walrus downloads its seed from the CLC). That's borken now, so I expect topology2 to always fail.

Marking Triaged/High. Thierry confirmed it. We now have to find out if this is a new (undocumented) feature of 2.0, or simply a regression.

Sending this upstream.

Upstream Eucalyptus: Can you confirm this new behavior? If so, we (Ubuntu) need to rework our install/startup procedure...

Assigning to Dave Walker. This bug is going to need serious attention.

Heh. I just installed CLC+Walrus, and got "Do not ever do whatever it is you did: Failed to lookup required component: storage". This is quite broken right now.

Resetting Importance to Critical -- the only topology that works, right now, is the all-in-one (CLC+Walrus+CC+SC).

Mail sent to upstream Eucalyptus for comment if this is a regression or intended design change.

@Dustin, Please don't assign me bugs without discussing them with me first. Thanks.

Still present on 2.0~bzr1218-0ubuntu1.

@chris grzegorczyk, Thanks for looking into this - do you have any progress to report?

Thanks.

Working on setting up regression test and merging the fix now.

Just installed eucalyptus-cloud on a fully-separated topology:

15:36:40 ERROR [ServiceDispatchBootstrapper:main] Ignoring the error that occured: com.eucalyptus.ws.ServiceDispatchBootstrapper.load:105
15:36:40 ERROR [Exceptions:main] java.util.NoSuchElementException: Missing entry for component 'walrus' info type: Component (com.eucalyptus.component.Component
java.util.NoSuchElementException: Missing entry for component 'walrus' info type: Component (com.eucalyptus.component.Component
        at com.eucalyptus.component.Components.lookup(Components.java:98)
        at com.eucalyptus.component.Components.lookup(Components.java:110)
        at com.eucalyptus.ws.ServiceDispatchBootstrapper.load(ServiceDispatchBootstrapper.java:103)
        at com.eucalyptus.bootstrap.Bootstrap$Stage.load(Bootstrap.java:93)
        at com.eucalyptus.bootstrap.SystemBootstrapper.load(SystemBootstrapper.java:129)
15:36:40 ERROR [ServiceDispatchBootstrapper:main] Ignoring the error that occured: com.eucalyptus.ws.ServiceDispatchBootstrapper.load:105
15:36:40 ERROR [Exceptions:main] java.util.NoSuchElementException: Missing entry for component 'storage' info type: Component (com.eucalyptus.component.Component
java.util.NoSuchElementException: Missing entry for component 'storage' info type: Component (com.eucalyptus.component.Component
        at com.eucalyptus.component.Components.lookup(Components.java:98)
        at com.eucalyptus.component.Components.lookup(Components.java:110)
        at com.eucalyptus.ws.ServiceDispatchBootstrapper.load(ServiceDispatchBootstrapper.java:103)
        at com.eucalyptus.bootstrap.Bootstrap$Stage.load(Bootstrap.java:93)
        at com.eucalyptus.bootstrap.SystemBootstrapper.load(SystemBootstrapper.java:129)
15:36:46 ERROR [SystemConfiguration:1727693667@qtp-941762215-8] com.eucalyptus.util.EucalyptusCloudException: Error locating information for [SystemConfiguration id=-1 cloudHost=null defaultKernel=null defaultRamdisk=null registrationId=null maxUserPublicAddresses=null doDynamicPublicAddresses=null systemReservedPublicAddresses=null dnsDomain=null nameserver=null nameserverAddress=null]
com.eucalyptus.util.EucalyptusCloudException: Error locating information for [SystemConfiguration id=-1 cloudHost=null defaultKernel=null defaultRamdisk=null registrationId=null maxUserPublicAddresses=null doDynamicPublicAddresses=null systemReservedPublicAddresses=null dnsDomain=null nameserver=null nameserverAddress=null]
        at com.eucalyptus.entities.EntityWrapper.getUnique(EntityWrapper.java:138)
        at edu.ucsb.eucalyptus.cloud.entities.SystemConfiguration.getSystemConfiguration(SystemConfiguration.java:257)
        at edu.ucsb.eucalyptus.admin.server.Registration.getRegistrationId(Registration.java:142)
        at edu.ucsb.eucalyptus.admin.server.Registration.doGet(Registration.java:148)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
        at org.mortbay.jetty.security.Se...

I installed a walrus (10.55.55.5) and a Cluster (10.55.55.8). The CLC stays running, the CC stays running (and seem to be recognised by the CLC), but the Walrus dies on startup:

17:06:29 ERROR [Transition$anonymously:main] :1281474389221:anonymously:TRANSITION_FAILED:Transition:from:PRIMORDIAL:to:INITIALIZED:com.eucalyptus.component.Component:Bootstrap.initialize.252
17:06:29 ERROR [Transition$anonymously:main] :1281474389221:anonymously:TRANSITION_ROLLBACK:Transition:from:PRIMORDIAL:to:INITIALIZED:com.eucalyptus.component.Component:Bootstrap.initialize.252
17:06:29 FATAL [SystemBootstrapper:main] java.util.NoSuchElementException: Missing entry for component 'eucalyptus' info type: Component (com.eucalyptus.component.Component
java.util.NoSuchElementException: Missing entry for component 'eucalyptus' info type: Component (com.eucalyptus.component.Component
        at com.eucalyptus.component.Components.lookup(Components.java:98)
        at com.eucalyptus.component.Components.lookup(Components.java:110)
        at com.eucalyptus.bootstrap.Component.isLocal(Component.java:114)
        at com.eucalyptus.bootstrap.Bootstrap$1.commit(Bootstrap.java:255)
        at com.eucalyptus.bootstrap.Bootstrap$1.commit(Bootstrap.java:252)
        at com.eucalyptus.component.Lifecycles$State$1.commit(Lifecycles.java:35)
        at com.eucalyptus.util.Transition$anonymously.commit(Transition.java:74)
        at com.eucalyptus.util.Transition.doTransition(Transition.java:133)
        at com.eucalyptus.util.Transition.transition(Transition.java:145)
        at com.eucalyptus.bootstrap.Bootstrap.initialize(Bootstrap.java:252)
        at com.eucalyptus.bootstrap.SystemBootstrapper.init(SystemBootstrapper.java:109)

So, better, but still no cigar.

This appears to be resolved on our end in the following rev. Please see the merge log for details.
------------------------------------------------------------
revno: 1220 [merge]
committer: decker <decker@personal-army>
branch nick: devel
timestamp: Tue 2010-08-10 04:22:52 -0700
message:
  merge: fixes to registration and remote bootstrapping of components & forward port of addressing fixes
------------------------------------------------------------

@carlos: sorry, i missed your message.

Can you try installing all of the components on each machine but only running Walrus, SC? This looks like it might be a matter of how things are packaged on our end.

I will try to reproduce on our side and get back ASAP.

Right now I am running an install for (CLC+Walrus), (CC+SC), (NC)*. So far, the (CLC+Walrus) installed OK, and kept running. When I finish this I will try installing all on each.

The (CC+SC) also fail to initialise:

18:15:02 FATAL [SystemBootstrapper:main] java.util.NoSuchElementException: Missing entry for component 'eucalyptus' info type: Component (com.eucalyptus.component.Component
java.util.NoSuchElementException: Missing entry for component 'eucalyptus' info type: Component (com.eucalyptus.component.Component
        at com.eucalyptus.component.Components.lookup(Components.java:98)
        at com.eucalyptus.component.Components.lookup(Components.java:110)
        at com.eucalyptus.bootstrap.Component.isLocal(Component.java:114)
        at com.eucalyptus.bootstrap.Bootstrap$1.commit(Bootstrap.java:255)
        at com.eucalyptus.bootstrap.Bootstrap$1.commit(Bootstrap.java:252)
        at com.eucalyptus.component.Lifecycles$State$1.commit(Lifecycles.java:35)
        at com.eucalyptus.util.Transition$anonymously.commit(Transition.java:74)
        at com.eucalyptus.util.Transition.doTransition(Transition.java:133)
        at com.eucalyptus.util.Transition.transition(Transition.java:145)
        at com.eucalyptus.bootstrap.Bootstrap.initialize(Bootstrap.java:252)
        at com.eucalyptus.bootstrap.SystemBootstrapper.init(SystemBootstrapper.java:109)

OK. I installed eucalyptus-cloud on the (CC+SC) machine, and restarted the SC and CC. They *are* up, and the CLC recognised them. I will now install the NCs, and see what happens.

So, yes, it seems something in packaging (requiring the cloud .jar?).

(CLC+Walrus), (CC+SC+CLC-down), (NC)+ is up. This CC is running with two NCs. I will run some tests now.

Hmmm. I will get back to you with a fix soon. Hopefully I can manage
to resolve the issue w/o any code changes. More shortly.

On Tue, Aug 10, 2010 at 3:25 PM, C de-Avillez
<email address hidden> wrote:
> OK. I installed eucalyptus-cloud on the (CC+SC) machine, and restarted
> the SC and CC. They *are* up, and the CLC recognised them. I will now
> install the NCs, and see what happens.
>
> So, yes, it seems something in packaging (requiring the cloud .jar?).
>
> --
> eucalyptus-cloud: cloud fails to start on separate install
> https://bugs.launchpad.net/bugs/613033
> You received this bug notification because you are a bug assignee.
>
> Status in Eucalyptus: Fix Committed
> Status in “eucalyptus” package in Ubuntu: Triaged
> Status in “eucalyptus” source package in Maverick: Triaged
>
> Bug description:
> Binary package hint: eucalyptus-cloud
>
> Installing a fully-separate topology (CLC, Walrus, CC, SC, and NCs each on their own machine). After installing the CLC Eucalyptus fails to initialise with:
>
> "Do not ever do whatever it is you did: Failed to lookup required component: walrus".
>
> ii  eucalyptus-cloud                         2.0~bzr1211-0ubuntu1
> ii  eucalyptus-common                        2.0~bzr1211-0ubuntu1
> ii  eucalyptus-java-common                   2.0~bzr1211-0ubuntu1
>
> ProblemType: BugDistroRelease: Ubuntu 10.10
> Package: debian-installer 20100211ubuntu17
> ProcVersionSignature: Ubuntu 2.6.35-14.19-generic 2.6.35
> Uname: Linux 2.6.35-14-generic x86_64
> Architecture: amd64
> CheckboxSubmission: 48216eabc5cef2006b8478621d301a28
> CheckboxSystem: d00f84de8a555815fa1c4660280da308
> Date: Thu Jul  1 15:15:09 2010
> Dependencies:
>
> EcryptfsInUse: YesInstallationMedia: Ubuntu-Server 10.04 "Lucid Lynx" - Beta amd64 (20100406.1)
> MachineType: Dell Inc. Inspiron 1721
> MemoryUsage:
>  total       used       free     shared    buffers     cached
>  Mem:       3796184    3741964      54220          0      66092    2333688
>  -/+ buffers/cache:    1342184    2454000
>  Swap:      3903484      30172    3873312
> ProcCmdLine: BOOT_IMAGE=/vmlinuz-2.6.35-14-generic root=/dev/mapper/hostname-root ro crashkernel=384M-2G:64M,2G-:128M
> ProcEnviron:
>  PATH=(custom, no user)
>  LANG=en_US.UTF-8
>  SHELL=/bin/bashSourcePackage: debian-installer
>
>
>
>

--
Chris Grzegorczyk
Co-founder, Eucalyptus Systems, Inc.
www.eucalyptus.com | 130 Castilian St. | Goleta, CA | 93117 | Mobile:
805-807-8237
________________________________________

This bug was fixed in the package eucalyptus - 2.0~bzr1224-0ubuntu1

---------------
eucalyptus (2.0~bzr1224-0ubuntu1) maverick; urgency=low

  [ Dave Walker (Daviey) ]
  * New upstream merge, r1224, should fix LP: #610259, #609112, #613033
  * debian/patches/10-disable-iscsi.patch: Removed, we should now be
    supporting iscsi.
  * debian/control: Promoted tgt from Recommends to Depends for
    eucalyptus-sc. (LP: #614503)

  [ Scott Moser ]
  * work around kvm virtio bug with -kernel by using boot floppy LP: #615529
  * remove the listing of dependency on grub-pc for workaround. LP: #613463
 -- Dave Walker (Daviey) <email address hidden> Wed, 11 Aug 2010 13:46:50 +0100

Reopening, does not seem fixed

@carlos: is that with r1225?

@chris: yes, it was. I am just now finishing a reinstall, and will be able to confirm in about 20 min.

Yes, not yet fixed. I am adding a tarball with the logs here

This was a (CLC+Walrus), (CC+SC) install. (CLC+Walrus) are in cempedak, (CC+SC) in mabolo. I did not even install the NCs.

This bug was fixed in the package eucalyptus - 2.0~bzr1230-0ubuntu1

---------------
eucalyptus (2.0~bzr1230-0ubuntu1) maverick; urgency=low

  * New upstream merge, r1230 - bug fixes:
    - (LP: #617127, #611769, #617032, #613033, #613033, #584497)
  * debian/patches/18-priv_security.patch: Added *_iscsitarget.pl scripts to
    the wrappers.conf whitelist. (LP: #615646)
  * debian/patches/15-fix-default-ramdisk.patch: Dropped, fixed applied upstream.
 -- Dave Walker (Daviey) <email address hidden> Fri, 13 Aug 2010 11:25:25 +0100

I have eucalytus at 2.0~bzr1241-0ubuntu4,1

I receive the following when sourcing eucarc

WARN: Walrus URL is not configured.

In reviewing the cloud-error.log It shows an initial failure with

13:32:55 ERROR [SystemConfiguration:529263462@qtp-248808824-7] com.eucalyptus.util.EucalyptusCloudException: Error locating information for [S
ystemConfiguration id=-1 cloudHost=null defaultKernel=null defaultRamdisk=null registrationId=null maxUserPublicAddresses=null doDynamicPublic
Addresses=null systemReservedPublicAddresses=null dnsDomain=null nameserver=null nameserverAddress=null]
com.eucalyptus.util.EucalyptusCloudException: Error locating information for [SystemConfiguration id=-1 cloudHost=null defaultKernel=null defa
ultRamdisk=null registrationId=null maxUserPublicAddresses=null doDynamicPublicAddresses=null systemReservedPublicAddresses=null dnsDomain=nul
l nameserver=null nameserverAddress=null]
        at com.eucalyptus.entities.EntityWrapper.getUnique(EntityWrapper.java:138)
        at edu.ucsb.eucalyptus.cloud.entities.SystemConfiguration.getSystemConfiguration(SystemConfiguration.java:257)
        at edu.ucsb.eucalyptus.admin.server.Registration.getRegistrationId(Registration.java:142)
        at edu.ucsb.eucalyptus.admin.server.Registration.doGet(Registration.java:148)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:926)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
13:32:57 FATAL [Bootstrap$Stage:main] Fatal error occured during bootstrap: com.eucalyptus.bootstrap.Bootstrap$Stage.start:118
java.lang.IllegalStateExcepti...

waltc, did you install eucalyptus-walrus and have you registered Walrus? You will get that warning ("Walrus URL is not configured") if you download credentials before you have Walrus registered.

Marking this bug fix released. Walt if you still have issue, please open a new bug.