Discussion:
lcrash and debian
Micah Anderson
2004-09-30 20:25:52 UTC
Permalink
In order to make lcrash available as a debian package, I need to sort
out a couple things, I am hoping you folks can help me.

First off, what is lkcdutils? Is it a combination of utilities from other
locations, modified specifically to work with lkcd?

Included in lkcdutils there is "crash". From what I can tell, crash is
an older version of something that comes from
http://people.redhat.com/anderson/ or
http://oss.missioncriticallinux.com/projects/crash (I am not sure
which). Is the crash included in lkcdutils something different, if so
what?

Second, how is lcrash related to crash?

And finally, what should I make available to debian users to be able
to work with things generated from the lkcd kernel patch?

I am working with very little time to resolve these issues, if I
cannot get a set of utilities to go with the lkcd kernel patch, then
it is likely that patch will be removed entirely from this version of
Debian. Instead, I would like to get it resolved so we can include it!

Thanks,
micah





-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Tom Morano
2004-09-30 21:00:07 UTC
Permalink
----- Original Message -----
From: "Micah Anderson" <***@riseup.net>
To: "Tom Morano" <***@sgi.com>
Cc: "Hariprasad Nellitheertha" <***@in.ibm.com>;
<lkcd-***@lists.sourceforge.net>
Sent: Thursday, September 30, 2004 1:25 PM
Subject: lcrash and debian
Post by Micah Anderson
In order to make lcrash available as a debian package, I need to sort
out a couple things, I am hoping you folks can help me.
First off, what is lkcdutils? Is it a combination of utilities from other
locations, modified specifically to work with lkcd?
lkcdutils is the application/utility side of the LKCD project. It contains
utilities for configuring the system to generate crash dumps, save dumps
after a system crash/reboot, and analyze the resulting dumps. lkcdutils also
contains various libraries for accessing kernel data in a dump image (or
live system memory).
Post by Micah Anderson
Included in lkcdutils there is "crash". From what I can tell, crash is
an older version of something that comes from
http://people.redhat.com/anderson/ or
http://oss.missioncriticallinux.com/projects/crash (I am not sure
which). Is the crash included in lkcdutils something different, if so
what?
You are correct in your first point. What is in lkcdutils is an older
version of the crash utility used by RedHat. It was placed in the lkcdutils
source as sort of a place holder when Mission Critical Linux dropped work on
it. Currently, the lcrash utility uses some of the gdb libraries contained
in the crash source.
Post by Micah Anderson
Second, how is lcrash related to crash?
These two utilities are related only in the similarity of their
functionality. They essentially both attempt to do the same things when
analyzing a system dump image (list running tasks, generate backtraces, dump
system memory, etc.).

As we go through the process of celaning up LKCD, we need to see if it makes
sense to leave the crash code in lkcdutils (most likley it does not).
Post by Micah Anderson
And finally, what should I make available to debian users to be able
to work with things generated from the lkcd kernel patch?
You will need the lkcdutils infrastructure for configuring the system and
saving the dumps. You will also need lcrash to analyze the dumps. I'm not
that familiar with the debian environment with regard to such things as
system startup scripts, etc. There may need to be some issues there (or
not).
Post by Micah Anderson
I am working with very little time to resolve these issues, if I
cannot get a set of utilities to go with the lkcd kernel patch, then
it is likely that patch will be removed entirely from this version of
Debian. Instead, I would like to get it resolved so we can include it!
Sure, I would like to see that also. I'll help in any way I can. BTW, what
time frame are we talking about?

Thanks,

Tom
Post by Micah Anderson
Thanks,
micah
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Micah Anderson
2004-09-30 21:20:04 UTC
Permalink
Post by Tom Morano
----- Original Message -----
Sent: Thursday, September 30, 2004 1:25 PM
Subject: lcrash and debian
Post by Micah Anderson
In order to make lcrash available as a debian package, I need to sort
out a couple things, I am hoping you folks can help me.
First off, what is lkcdutils? Is it a combination of utilities from other
locations, modified specifically to work with lkcd?
lkcdutils is the application/utility side of the LKCD project. It contains
utilities for configuring the system to generate crash dumps, save dumps
after a system crash/reboot, and analyze the resulting dumps. lkcdutils also
contains various libraries for accessing kernel data in a dump image (or
live system memory).
This is how I understood lkcdutils as well, but when I begain putting
it together for debian I got confused as to all the different elements
involved.

What I would like to do for debian is take the user-land binaries of
the utilities for creating dumps and analysis and put them in a
package called lkcdutils. Then I would like to put all the libraries
and other useful development pieces in a separate lkcdutils-dev
package. This is typically how it is done in Debian. Once I get my
hands on lkcdutils I will probably need your help sorting out the
pieces.

Is there a "release" of lkcdutils that corresponds with the newest
patch-sets? This would be the best for me to get, however I can also
pull from CVS the appropriate things and create for myself a tarball
to build the package from. Please let me know how I should do this so
I can get started.
Post by Tom Morano
Post by Micah Anderson
Included in lkcdutils there is "crash". From what I can tell, crash is
an older version of something that comes from
http://people.redhat.com/anderson/ or
http://oss.missioncriticallinux.com/projects/crash (I am not sure
which). Is the crash included in lkcdutils something different, if so
what?
You are correct in your first point. What is in lkcdutils is an older
version of the crash utility used by RedHat. It was placed in the lkcdutils
source as sort of a place holder when Mission Critical Linux dropped work on
Ok, so it seems like I should keep "crash" from my package (as the
redhat version is currently being packaged by someone else).
Post by Tom Morano
it. Currently, the lcrash utility uses some of the gdb libraries contained
in the crash source.
However, this makes me think that I cannot remove crash completely...
Are there plans to move the necessary gdb libraries from crash into
lcrash?
Post by Tom Morano
Post by Micah Anderson
Second, how is lcrash related to crash?
These two utilities are related only in the similarity of their
functionality. They essentially both attempt to do the same things when
analyzing a system dump image (list running tasks, generate backtraces, dump
system memory, etc.).
As we go through the process of celaning up LKCD, we need to see if it makes
sense to leave the crash code in lkcdutils (most likley it does not).
This makes sense, it may be that I will need to include it in the
package until that happens. Most likely it would be put in the
lkcdutils-dev package as it is only needed for building.
Post by Tom Morano
Post by Micah Anderson
And finally, what should I make available to debian users to be able
to work with things generated from the lkcd kernel patch?
You will need the lkcdutils infrastructure for configuring the system and
saving the dumps. You will also need lcrash to analyze the dumps. I'm not
that familiar with the debian environment with regard to such things as
system startup scripts, etc. There may need to be some issues there (or
not).
I'll tackle those as I get to them.
Post by Tom Morano
Post by Micah Anderson
I am working with very little time to resolve these issues, if I
cannot get a set of utilities to go with the lkcd kernel patch, then
it is likely that patch will be removed entirely from this version of
Debian. Instead, I would like to get it resolved so we can include it!
Sure, I would like to see that also. I'll help in any way I can. BTW, what
time frame are we talking about?
Good question... This is hard to pin down when new packages will be
frozen, the threat is that it should've happened already, but it
hasn't yet... so I anticipate it within the next week. I will try to
find out more details.

Micah


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Michael Holzheu
2004-10-01 08:01:30 UTC
Permalink
Hi Micah, Hi Tom,

As far as I know, lcrash does not use gdb libraries. Since crash has not
been ported to s390 yet, we currently do not include it into our
s390-lkcdutils package. Having crash/gdb is not necessary to build or run
lcrash.

Best Regards

Michael

------------------------------------------------------------------------
Linux for E-Server Development
Phone: +49-7031-16-2360, Bld 71032-03-U09
Post by Micah Anderson
Post by Tom Morano
----- Original Message -----
Sent: Thursday, September 30, 2004 1:25 PM
Subject: lcrash and debian
Post by Micah Anderson
In order to make lcrash available as a debian package, I need to sort
out a couple things, I am hoping you folks can help me.
First off, what is lkcdutils? Is it a combination of utilities from other
locations, modified specifically to work with lkcd?
lkcdutils is the application/utility side of the LKCD project. It contains
utilities for configuring the system to generate crash dumps, save dumps
after a system crash/reboot, and analyze the resulting dumps. lkcdutils also
contains various libraries for accessing kernel data in a dump image (or
live system memory).
This is how I understood lkcdutils as well, but when I begain putting
it together for debian I got confused as to all the different elements
involved.
What I would like to do for debian is take the user-land binaries of
the utilities for creating dumps and analysis and put them in a
package called lkcdutils. Then I would like to put all the libraries
and other useful development pieces in a separate lkcdutils-dev
package. This is typically how it is done in Debian. Once I get my
hands on lkcdutils I will probably need your help sorting out the
pieces.
Is there a "release" of lkcdutils that corresponds with the newest
patch-sets? This would be the best for me to get, however I can also
pull from CVS the appropriate things and create for myself a tarball
to build the package from. Please let me know how I should do this so
I can get started.
Post by Tom Morano
Post by Micah Anderson
Included in lkcdutils there is "crash". From what I can tell, crash is
an older version of something that comes from
http://people.redhat.com/anderson/ or
http://oss.missioncriticallinux.com/projects/crash (I am not sure
which). Is the crash included in lkcdutils something different, if so
what?
You are correct in your first point. What is in lkcdutils is an older
version of the crash utility used by RedHat. It was placed in the lkcdutils
source as sort of a place holder when Mission Critical Linux dropped work on
Ok, so it seems like I should keep "crash" from my package (as the
redhat version is currently being packaged by someone else).
Post by Tom Morano
it. Currently, the lcrash utility uses some of the gdb libraries contained
in the crash source.
However, this makes me think that I cannot remove crash completely...
Are there plans to move the necessary gdb libraries from crash into
lcrash?
Post by Tom Morano
Post by Micah Anderson
Second, how is lcrash related to crash?
These two utilities are related only in the similarity of their
functionality. They essentially both attempt to do the same things when
analyzing a system dump image (list running tasks, generate backtraces, dump
system memory, etc.).
As we go through the process of celaning up LKCD, we need to see if it makes
sense to leave the crash code in lkcdutils (most likley it does not).
This makes sense, it may be that I will need to include it in the
package until that happens. Most likely it would be put in the
lkcdutils-dev package as it is only needed for building.
Post by Tom Morano
Post by Micah Anderson
And finally, what should I make available to debian users to be able
to work with things generated from the lkcd kernel patch?
You will need the lkcdutils infrastructure for configuring the system and
saving the dumps. You will also need lcrash to analyze the dumps. I'm not
that familiar with the debian environment with regard to such things as
system startup scripts, etc. There may need to be some issues there (or
not).
I'll tackle those as I get to them.
Post by Tom Morano
Post by Micah Anderson
I am working with very little time to resolve these issues, if I
cannot get a set of utilities to go with the lkcd kernel patch, then
it is likely that patch will be removed entirely from this version of
Debian. Instead, I would like to get it resolved so we can include it!
Sure, I would like to see that also. I'll help in any way I can. BTW, what
time frame are we talking about?
Good question... This is hard to pin down when new packages will be
frozen, the threat is that it should've happened already, but it
hasn't yet... so I anticipate it within the next week. I will try to
find out more details.
Micah
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Lkcd-general mailing list
https://lists.sourceforge.net/lists/listinfo/lkcd-general
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Corey Minyard
2004-10-01 13:04:04 UTC
Permalink
crash has code for s390 and s390x. I don't know if they work because I
don't have one of these beasts, but there is at least code there.

-Corey
Post by Michael Holzheu
Hi Micah, Hi Tom,
As far as I know, lcrash does not use gdb libraries. Since crash has not
been ported to s390 yet, we currently do not include it into our
s390-lkcdutils package. Having crash/gdb is not necessary to build or run
lcrash.
Best Regards
Michael
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Tom Morano
2004-10-01 16:41:22 UTC
Permalink
----- Original Message -----
From: "Corey Minyard" <***@acm.org>
To: "Michael Holzheu" <***@de.ibm.com>
Cc: "Micah Anderson" <***@riseup.net>; "Hariprasad Nellitheertha"
<***@in.ibm.com>; "lkcd-devel" <lkcd-***@lists.sourceforge.net>;
<lkcd-***@lists.sourceforge.net>;
<lkcd-general-***@lists.sourceforge.net>; "Tom Morano" <***@sgi.com>
Sent: Friday, October 01, 2004 6:04 AM
Subject: Re: [lkcd-devel] Re: [lkcd-general] Re: lcrash and debian
Post by Corey Minyard
crash has code for s390 and s390x. I don't know if they work because I
don't have one of these beasts, but there is at least code there.
The version of crash that is in lkcdutils is very old. It was put there
(years ago) as a place holder. At that time, there was some direct use of
the gdb libraries it contained (we're talking about the version of lcrash
before the arch independent version). I don't know if it makes sense to even
keep this version of the source code around any more. It's quite large
(because of the gdb source) and really bogs down the downloading of project
source. Also, If someone is interested in the crash source, they should be
able to download a much more current version from RedHat...

Tom
Post by Corey Minyard
-Corey
Post by Michael Holzheu
Hi Micah, Hi Tom,
As far as I know, lcrash does not use gdb libraries. Since crash has not
been ported to s390 yet, we currently do not include it into our
s390-lkcdutils package. Having crash/gdb is not necessary to build or run
lcrash.
Best Regards
Michael
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Tom Morano
2004-10-02 15:42:27 UTC
Permalink
----- Original Message -----
From: "Corey Minyard" <***@acm.org>
To: "Tom Morano" <***@sgi.com>
Cc: "Michael Holzheu" <***@de.ibm.com>; "Micah Anderson"
<***@riseup.net>; "Hariprasad Nellitheertha" <***@in.ibm.com>;
"lkcd-devel" <lkcd-***@lists.sourceforge.net>;
<lkcd-***@lists.sourceforge.net>;
<lkcd-general-***@lists.sourceforge.net>
Sent: Friday, October 01, 2004 7:47 PM
Subject: Re: [lkcd-devel] Re: [lkcd-general] Re: lcrash and debian
Post by Tom Morano
Post by Corey Minyard
crash has code for s390 and s390x. I don't know if they work because I
don't have one of these beasts, but there is at least code there.
The version of crash that is in lkcdutils is very old. It was put there
(years ago) as a place holder. At that time, there was some direct use of
the gdb libraries it contained (we're talking about the version of lcrash
before the arch independent version). I don't know if it makes sense to even
keep this version of the source code around any more. It's quite large
(because of the gdb source) and really bogs down the downloading of project
source. Also, If someone is interested in the crash source, they should be
able to download a much more current version from RedHat...
Tom
Very true. I am using the ones from Redhat with massive patches to make
then work cross.
I would be willing to maintain the version in your CVS, if you like. I
already maintain a CVS repository, it wouldn't be a big deal to move it.
I'd also be willing to add my lkcdtools.
Corey,

I'm not sure the LKCD project is the best place for crash (it was only
pushed into the LKCD tree because it wasn't clear where it was going to end
up after MCL stopped supporting it). I may be off base on this as I don't
use crash and am totally focused on lcrash. At the very least, I don't think
it belongs inside of the lkcdutils tree. Ideally, I would like to see the
two projects combine their efforts somewhat. I'm not sure if that's possible
however. I would be interested in hearing input from others on this list
regarding their preference.

Regarding your lkcdtools, this sounds like something that would make sense
to include in the LKCD project. Right now, we are trying to get the tree
synced up with what is shipping on SuSE SLES9 and get LKCD working with
Debian. Beyond that, I'm focused on getting this working on SGI Altix (IPF)
systems. The basic layout of the source and release mechanism is open for
discussion. I would assume the same is true for the base level utilities
themselves. This project is definitely in need of some more energy. Feel
free to jump in... :]

Thanks,

Tom



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Tom Morano
2004-10-04 16:31:38 UTC
Permalink
----- Original Message -----
From: "Corey Minyard" <***@acm.org>
To: "Tom Morano" <***@sgi.com>
Cc: "Michael Holzheu" <***@de.ibm.com>; "Micah Anderson"
<***@riseup.net>; "Hariprasad Nellitheertha" <***@in.ibm.com>;
"lkcd-devel" <lkcd-***@lists.sourceforge.net>;
<lkcd-***@lists.sourceforge.net>;
<lkcd-general-***@lists.sourceforge.net>
Sent: Monday, October 04, 2004 6:04 AM
Subject: Re: [lkcd-devel] Re: [lkcd-general] Re: lcrash and debian

.
.
.
Post by Tom Morano
Corey,
I'm not sure the LKCD project is the best place for crash (it was only
pushed into the LKCD tree because it wasn't clear where it was going to end
up after MCL stopped supporting it). I may be off base on this as I don't
use crash and am totally focused on lcrash. At the very least, I don't think
it belongs inside of the lkcdutils tree. Ideally, I would like to see the
two projects combine their efforts somewhat. I'm not sure if that's possible
however. I would be interested in hearing input from others on this list
regarding their preference.
I really don't like lcrash because it is so tied to the kernel version.
With crash, I can use the same tool to work on different versions with
the same tool. In an environment where I have to analyze crash dumps
from lots of customers running on different kernel versions, different
versions of LKCD (not to mention some using mcore), crash is much
nicer. I like the crash user interface better, too, especially the
ability to do gdb operations with it. The ability to nicely print
arbitrary data structures is quite handy.
I guess I don't quite see where lcrash is so tied to the kernel version. The
only thing that would cause it problems is a functional change in a kernel
data structure (remove/change a key struct member, etc.). Of course that may
not be the case with the version currently in the CVS tree (I've noticed
that there are some problems with the treatment of local copies of dump
headers, for example). Being able to do gdb operations is a nice feature,
however lcrash can nicely print arbitrary data structures too, so I'm not
quite sure what you mean by your last statement.
Plus, I have cross support for crash, which is also an absolute
necessity for us since we support host development on different
platforms than the target environment.
I guess I'm missing something here. The current version of lcrash appears to
have cross architecutre support (if you build it that way). I haven't used
it much that way, so perhaps there are some issues there that I don't know
about.
If I was to support this, I would maintain the one that does
cross-development support, BTW. I think there are enough people
interested in this to make it worthwhile.
Post by Tom Morano
Regarding your lkcdtools, this sounds like something that would make sense
to include in the LKCD project. Right now, we are trying to get the tree
synced up with what is shipping on SuSE SLES9 and get LKCD working with
Debian. Beyond that, I'm focused on getting this working on SGI Altix (IPF)
systems. The basic layout of the source and release mechanism is open for
discussion. I would assume the same is true for the base level utilities
themselves. This project is definitely in need of some more energy. Feel
free to jump in... :]
Probably the ugliest think about lkcdutils is building it. It really
needs to use autoconf/automake. (So does crash, BTW). But I don't have
a lot of reason to fix that because I'm not using it any more. I might
work on fixing crash, but I have a lot of other things on my plate right
now.
IMHO, lkcdutils doesn't need to exist any more. Let Dave handle the
crash dump support (with a little support from you) and use some simple
tool for crash dump extraction like the one HP developed or lkcdtools.
That way you can focus your energy on other things.
Perhaps in the future...right now that is not an option (for us). There are
various reasons why lkcdutils (lcrash) is our crash dump toolset of choice.
A lot of these tie directly to support of our (SGI) systems and the timing
of product releases. We will have to see how this entire space evolves over
time. Right now, I'm just glad to see a higher level of activity and energy
in this project.

Tom



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Tom Morano
2004-10-04 18:22:22 UTC
Permalink
----- Original Message -----
From: "Corey Minyard" <***@acm.org>
To: "Tom Morano" <***@SGI.COM>
Cc: "Michael Holzheu" <***@de.ibm.com>; "Micah Anderson"
<***@riseup.net>; "Hariprasad Nellitheertha" <***@in.ibm.com>;
"lkcd-devel" <lkcd-***@lists.sourceforge.net>;
<lkcd-***@lists.sourceforge.net>;
<lkcd-general-***@lists.sourceforge.net>
Sent: Monday, October 04, 2004 10:23 AM
Subject: Re: [lkcd-devel] Re: [lkcd-general] Re: lcrash and debian
Post by Tom Morano
I guess I don't quite see where lcrash is so tied to the kernel version. The
only thing that would cause it problems is a functional change in a kernel
data structure (remove/change a key struct member, etc.). Of course that may
not be the case with the version currently in the CVS tree (I've noticed
that there are some problems with the treatment of local copies of dump
headers, for example). Being able to do gdb operations is a nice feature,
however lcrash can nicely print arbitrary data structures too, so I'm not
quite sure what you mean by your last statement.\
This is exactly what I'm talking about. If a user turns on/off things
like preemption and SMP, major kernel data structures change. Plus,
lcrash will only work with one version of the LKCD header (we currently
support two). I don't want to have to compile a different version of
lcrash for each possibility. And our users will complain bitterly if
they have to do that.
This is not the sort of change I'm referring to. I'm talking about more
functional changes between kernel versions, like when the next and prev
pointers in the task_struct were changed to be list_head structs. This sort
of change requires a similar change to be made in the crash dump analysis
tool (crash or lcrash). If a particular kernel is built with certain ifdefs
turned on or off, that should be something the tools can handle. lcrash
should be able to handle this. The version we are using at SGI can. The
trick is to use the struct definition in the type information to determine
the proper member offset, size, etc. If the current lcrash does not handle
this sort of thing, I would consider that a bug that needs to be fixed.
p *(ipmi_interfaces[0])
and get a nicely formatted output, just like gdb?
lcrash does not require a full kernel (with debug info intact) to run. All
it needs is the Kerntypes file to get kernel data type definitions. I
understand that crash does need a full kernel to work properly (I might not
be right abuot this). Your above example assumes that the crash analysis
tool knows what data type impi_interfaces is. If you were to issue the same
command in lcrash, but instead of having the variable name, have a typecast
reference to the variable address (we do know that address to use for this
symbol -- from System.map), then it would print out a formatted struct (not
quite like gdb, but more like it is declared in a C header file). lcrash
should be able to do the same thing as crash if it has a full kernel to work
with. I don't think that it currently can do this. Nor is it able to get
line number information for task backtraces. Again, a full kernel would be
required for this. I think a bit of work is required to get it to work, but
not a huge amout (all the interfaces to the debug info are already there).
Post by Tom Morano
Probably the ugliest think about lkcdutils is building it. It really
needs to use autoconf/automake. (So does crash, BTW). But I don't have
a lot of reason to fix that because I'm not using it any more. I might
work on fixing crash, but I have a lot of other things on my plate right
now.
IMHO, lkcdutils doesn't need to exist any more. Let Dave handle the
crash dump support (with a little support from you) and use some simple
tool for crash dump extraction like the one HP developed or lkcdtools.
That way you can focus your energy on other things.
Perhaps in the future...right now that is not an option (for us). There are
various reasons why lkcdutils (lcrash) is our crash dump toolset of choice.
A lot of these tie directly to support of our (SGI) systems and the timing
of product releases. We will have to see how this entire space evolves over
time. Right now, I'm just glad to see a higher level of activity and energy
in this project.
Fair enough, I understand how that goes :).
:]

Thanks for the input...

Tom



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Corey Minyard
2004-10-04 17:23:09 UTC
Permalink
Post by Tom Morano
I guess I don't quite see where lcrash is so tied to the kernel version. The
only thing that would cause it problems is a functional change in a kernel
data structure (remove/change a key struct member, etc.). Of course that may
not be the case with the version currently in the CVS tree (I've noticed
that there are some problems with the treatment of local copies of dump
headers, for example). Being able to do gdb operations is a nice feature,
however lcrash can nicely print arbitrary data structures too, so I'm not
quite sure what you mean by your last statement.\
This is exactly what I'm talking about. If a user turns on/off things
like preemption and SMP, major kernel data structures change. Plus,
lcrash will only work with one version of the LKCD header (we currently
support two). I don't want to have to compile a different version of
lcrash for each possibility. And our users will complain bitterly if
they have to do that.

On the second part, you mean I can type something like:

p *(ipmi_interfaces[0])

and get a nicely formatted output, just like gdb?
Post by Tom Morano
Probably the ugliest think about lkcdutils is building it. It really
needs to use autoconf/automake. (So does crash, BTW). But I don't have
a lot of reason to fix that because I'm not using it any more. I might
work on fixing crash, but I have a lot of other things on my plate right
now.
IMHO, lkcdutils doesn't need to exist any more. Let Dave handle the
crash dump support (with a little support from you) and use some simple
tool for crash dump extraction like the one HP developed or lkcdtools.
That way you can focus your energy on other things.
Perhaps in the future...right now that is not an option (for us). There are
various reasons why lkcdutils (lcrash) is our crash dump toolset of choice.
A lot of these tie directly to support of our (SGI) systems and the timing
of product releases. We will have to see how this entire space evolves over
time. Right now, I'm just glad to see a higher level of activity and energy
in this project.
Fair enough, I understand how that goes :).

-Corey



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Corey Minyard
2004-10-04 13:04:16 UTC
Permalink
Post by Tom Morano
----- Original Message -----
Sent: Friday, October 01, 2004 7:47 PM
Subject: Re: [lkcd-devel] Re: [lkcd-general] Re: lcrash and debian
Post by Tom Morano
Post by Corey Minyard
crash has code for s390 and s390x. I don't know if they work because I
don't have one of these beasts, but there is at least code there.
The version of crash that is in lkcdutils is very old. It was put there
(years ago) as a place holder. At that time, there was some direct use of
the gdb libraries it contained (we're talking about the version of lcrash
before the arch independent version). I don't know if it makes sense to
even
Post by Tom Morano
keep this version of the source code around any more. It's quite large
(because of the gdb source) and really bogs down the downloading of
project
Post by Tom Morano
source. Also, If someone is interested in the crash source, they should
be
Post by Tom Morano
able to download a much more current version from RedHat...
Tom
Very true. I am using the ones from Redhat with massive patches to make
then work cross.
I would be willing to maintain the version in your CVS, if you like. I
already maintain a CVS repository, it wouldn't be a big deal to move it.
I'd also be willing to add my lkcdtools.
Corey,
I'm not sure the LKCD project is the best place for crash (it was only
pushed into the LKCD tree because it wasn't clear where it was going to end
up after MCL stopped supporting it). I may be off base on this as I don't
use crash and am totally focused on lcrash. At the very least, I don't think
it belongs inside of the lkcdutils tree. Ideally, I would like to see the
two projects combine their efforts somewhat. I'm not sure if that's possible
however. I would be interested in hearing input from others on this list
regarding their preference.
I really don't like lcrash because it is so tied to the kernel version.
With crash, I can use the same tool to work on different versions with
the same tool. In an environment where I have to analyze crash dumps
from lots of customers running on different kernel versions, different
versions of LKCD (not to mention some using mcore), crash is much
nicer. I like the crash user interface better, too, especially the
ability to do gdb operations with it. The ability to nicely print
arbitrary data structures is quite handy.

Plus, I have cross support for crash, which is also an absolute
necessity for us since we support host development on different
platforms than the target environment.

If I was to support this, I would maintain the one that does
cross-development support, BTW. I think there are enough people
interested in this to make it worthwhile.
Post by Tom Morano
Regarding your lkcdtools, this sounds like something that would make sense
to include in the LKCD project. Right now, we are trying to get the tree
synced up with what is shipping on SuSE SLES9 and get LKCD working with
Debian. Beyond that, I'm focused on getting this working on SGI Altix (IPF)
systems. The basic layout of the source and release mechanism is open for
discussion. I would assume the same is true for the base level utilities
themselves. This project is definitely in need of some more energy. Feel
free to jump in... :]
Probably the ugliest think about lkcdutils is building it. It really
needs to use autoconf/automake. (So does crash, BTW). But I don't have
a lot of reason to fix that because I'm not using it any more. I might
work on fixing crash, but I have a lot of other things on my plate right
now.

IMHO, lkcdutils doesn't need to exist any more. Let Dave handle the
crash dump support (with a little support from you) and use some simple
tool for crash dump extraction like the one HP developed or lkcdtools.
That way you can focus your energy on other things.

-Corey


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Rainer Bawidamann
2004-10-07 09:55:33 UTC
Permalink
Hi Corey!
Plus, I have cross support for crash, which is also an absolute
necessity for us since we support host development on different
platforms than the target environment.
If I was to support this, I would maintain the one that does
cross-development support, BTW. I think there are enough people
interested in this to make it worthwhile.
I'm very interested in this. But I haven't found this in Dave Anderson's
crash. Is it included there? If not, is you cross-development support
available somewhere?

Is you version able to work cross-endian? I.e. debug a dump from a
big-endian system on a little-endian system?

Rainer


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Corey Minyard
2004-10-07 12:59:47 UTC
Permalink
Post by Rainer Bawidamann
Hi Corey!
Plus, I have cross support for crash, which is also an absolute
necessity for us since we support host development on different
platforms than the target environment.
If I was to support this, I would maintain the one that does
cross-development support, BTW. I think there are enough people
interested in this to make it worthwhile.
I'm very interested in this. But I haven't found this in Dave Anderson's
crash. Is it included there? If not, is you cross-development support
available somewhere?
No, it is not included there. Since the patch is so huge, he has
decided not to take it (for now). I suspect he will change his mind
when Redhat's Wind River friends want this function :).

I can post the current version on my web site, if you like. It is
against a somewhat older version of crash.
Post by Rainer Bawidamann
Is you version able to work cross-endian? I.e. debug a dump from a
big-endian system on a little-endian system?
Yes, you can debug cross-endian and between targets with different type
sizes.

-Corey


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Rainer Bawidamann
2004-10-08 07:22:23 UTC
Permalink
Corey Minyard <***@acm.org> wrote on 07.10.2004 14:59:47:

[cross-crash]
Post by Corey Minyard
I can post the current version on my web site, if you like. It is
against a somewhat older version of crash.
Yes I would like to see your code. Is it a patched tarball or do you have
patches? I had some problems making patches because of the build process
of crash ...

Just curious: The cross-crash from MontaVista Jeff Haran talks about: is
this yours too?


Rainer


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Corey Minyard
2004-10-08 13:32:38 UTC
Permalink
Post by Rainer Bawidamann
[cross-crash]
Post by Corey Minyard
I can post the current version on my web site, if you like. It is
against a somewhat older version of crash.
Yes I would like to see your code. Is it a patched tarball or do you have
patches? I had some problems making patches because of the build process
of crash ...
Ok, it is at http://home.comcast.net/~minyard
Post by Rainer Bawidamann
Just curious: The cross-crash from MontaVista Jeff Haran talks about: is
this yours too?
Yes, although the one he has is quite a bit older. I work for
MontaVista and this is one of the many things I work on.

-Corey


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Corey Minyard
2004-10-02 02:47:09 UTC
Permalink
Post by Tom Morano
Post by Corey Minyard
crash has code for s390 and s390x. I don't know if they work because I
don't have one of these beasts, but there is at least code there.
The version of crash that is in lkcdutils is very old. It was put there
(years ago) as a place holder. At that time, there was some direct use of
the gdb libraries it contained (we're talking about the version of lcrash
before the arch independent version). I don't know if it makes sense to even
keep this version of the source code around any more. It's quite large
(because of the gdb source) and really bogs down the downloading of project
source. Also, If someone is interested in the crash source, they should be
able to download a much more current version from RedHat...
Tom
Very true. I am using the ones from Redhat with massive patches to make
then work cross.

I would be willing to maintain the version in your CVS, if you like. I
already maintain a CVS repository, it wouldn't be a big deal to move it.

I'd also be willing to add my lkcdtools.

-Corey


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Tom Morano
2004-10-01 17:02:39 UTC
Permalink
----- Original Message -----
From: "Micah Anderson" <***@riseup.net>
To: "Tom Morano" <***@sgi.com>
Cc: "Hariprasad Nellitheertha" <***@in.ibm.com>;
<lkcd-***@lists.sourceforge.net>; "lkcd-devel"
<lkcd-***@lists.sourceforge.net>
Sent: Thursday, September 30, 2004 2:20 PM
Subject: Re: lcrash and debian
Post by Micah Anderson
Post by Tom Morano
----- Original Message -----
Sent: Thursday, September 30, 2004 1:25 PM
Subject: lcrash and debian
Post by Micah Anderson
In order to make lcrash available as a debian package, I need to sort
out a couple things, I am hoping you folks can help me.
First off, what is lkcdutils? Is it a combination of utilities from other
locations, modified specifically to work with lkcd?
lkcdutils is the application/utility side of the LKCD project. It contains
utilities for configuring the system to generate crash dumps, save dumps
after a system crash/reboot, and analyze the resulting dumps. lkcdutils also
contains various libraries for accessing kernel data in a dump image (or
live system memory).
This is how I understood lkcdutils as well, but when I begain putting
it together for debian I got confused as to all the different elements
involved.
What I would like to do for debian is take the user-land binaries of
the utilities for creating dumps and analysis and put them in a
package called lkcdutils. Then I would like to put all the libraries
and other useful development pieces in a separate lkcdutils-dev
package. This is typically how it is done in Debian. Once I get my
hands on lkcdutils I will probably need your help sorting out the
pieces.
It should be set up already to do this. Three rpms get built (with the
current spec file). lkcdutils, lkcdutils-devel, and a source rpm.
Post by Micah Anderson
Is there a "release" of lkcdutils that corresponds with the newest
patch-sets? This would be the best for me to get, however I can also
pull from CVS the appropriate things and create for myself a tarball
to build the package from. Please let me know how I should do this so
I can get started.
I posted this (patch set) last week and will post a new, updated copy later
today. Since this is what I am planning on pushing into the CVS tree, I
would appreciate if you could do you testing (against Bob's kernel patches)
using the current CVS soruce plus my lkcdutils patch set.

Thanks,

Tom



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Micah Anderson
2004-10-01 17:25:49 UTC
Permalink
Post by Tom Morano
Post by Micah Anderson
What I would like to do for debian is take the user-land binaries of
the utilities for creating dumps and analysis and put them in a
package called lkcdutils. Then I would like to put all the libraries
and other useful development pieces in a separate lkcdutils-dev
package. This is typically how it is done in Debian. Once I get my
hands on lkcdutils I will probably need your help sorting out the
pieces.
It should be set up already to do this. Three rpms get built (with the
current spec file). lkcdutils, lkcdutils-devel, and a source rpm.
Are these RPMs available somewhere?
Post by Tom Morano
Post by Micah Anderson
Is there a "release" of lkcdutils that corresponds with the newest
patch-sets? This would be the best for me to get, however I can also
pull from CVS the appropriate things and create for myself a tarball
to build the package from. Please let me know how I should do this so
I can get started.
I posted this (patch set) last week and will post a new, updated copy later
today. Since this is what I am planning on pushing into the CVS tree, I
would appreciate if you could do you testing (against Bob's kernel patches)
using the current CVS soruce plus my lkcdutils patch set.
Am I correct in reading that there is a set of patches, in addition to
the kernel patches, that are applied to the kernel to be able to use
lkcdutils?

Micah


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Tom Morano
2004-10-01 18:15:37 UTC
Permalink
----- Original Message -----
From: "Micah Anderson" <***@riseup.net>
To: "Tom Morano" <***@sgi.com>
Cc: "Hariprasad Nellitheertha" <***@in.ibm.com>;
<lkcd-***@lists.sourceforge.net>; "lkcd-devel"
<lkcd-***@lists.sourceforge.net>
Sent: Friday, October 01, 2004 10:25 AM
Subject: [lkcd-general] Re: lcrash and debian
Post by Micah Anderson
Post by Tom Morano
Post by Micah Anderson
What I would like to do for debian is take the user-land binaries of
the utilities for creating dumps and analysis and put them in a
package called lkcdutils. Then I would like to put all the libraries
and other useful development pieces in a separate lkcdutils-dev
package. This is typically how it is done in Debian. Once I get my
hands on lkcdutils I will probably need your help sorting out the
pieces.
It should be set up already to do this. Three rpms get built (with the
current spec file). lkcdutils, lkcdutils-devel, and a source rpm.
Are these RPMs available somewhere?
No, you would need to build them. At some point, we should have rpms
available to downlaod from the project web page. We are a bit away from that
now...
Post by Micah Anderson
Post by Tom Morano
Post by Micah Anderson
Is there a "release" of lkcdutils that corresponds with the newest
patch-sets? This would be the best for me to get, however I can also
pull from CVS the appropriate things and create for myself a tarball
to build the package from. Please let me know how I should do this so
I can get started.
I posted this (patch set) last week and will post a new, updated copy later
today. Since this is what I am planning on pushing into the CVS tree, I
would appreciate if you could do you testing (against Bob's kernel patches)
using the current CVS soruce plus my lkcdutils patch set.
Am I correct in reading that there is a set of patches, in addition to
the kernel patches, that are applied to the kernel to be able to use
lkcdutils?
You are correct. I will send out an updated copy of these patches later
today.

Tom



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Micah Anderson
2004-09-30 21:45:34 UTC
Permalink
Post by Tom Morano
Sure, I would like to see that also. I'll help in any way I can. BTW, what
time frame are we talking about?
Actually, I think I was a little quick on the trigger on my answer to
this in my prior email. The freeze has happened for some certain types
of packages in Debian, but the LKCD stuff is not within that
structure.

The freeze for the section where LKCD is will happen two weeks after
security infrastructure drops into place. Some people seem to think
that another month or two is probable, but it also could happen in two
weeks. It is uncertain, it is a complex set of diffused points that
make this decision. :)

Micah


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Tom Morano
2004-10-01 16:43:37 UTC
Permalink
It sounds like there is a bit of time yet to work out the LKCD kinks so that
it can make it into the next Debian release. We are working with other
deadlines that are much tighter than that... :]

Thanks,

Tom

----- Original Message -----
From: "Micah Anderson" <***@riseup.net>
To: "Tom Morano" <***@sgi.com>
Cc: "Hariprasad Nellitheertha" <***@in.ibm.com>;
<lkcd-***@lists.sourceforge.net>; "lkcd-devel"
<lkcd-***@lists.sourceforge.net>
Sent: Thursday, September 30, 2004 2:45 PM
Subject: Re: lcrash and debian
Post by Micah Anderson
Post by Tom Morano
Sure, I would like to see that also. I'll help in any way I can. BTW, what
time frame are we talking about?
Actually, I think I was a little quick on the trigger on my answer to
this in my prior email. The freeze has happened for some certain types
of packages in Debian, but the LKCD stuff is not within that
structure.
The freeze for the section where LKCD is will happen two weeks after
security infrastructure drops into place. Some people seem to think
that another month or two is probable, but it also could happen in two
weeks. It is uncertain, it is a complex set of diffused points that
make this decision. :)
Micah
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Jeff Haran
2004-10-07 17:23:09 UTC
Permalink
-----Original Message-----
Minyard
Sent: Thursday, October 07, 2004 6:00 AM
To: Rainer Bawidamann
Subject: [lkcd-general] Re: [lkcd-devel] Re: Re: lcrash and debian
Post by Rainer Bawidamann
Hi Corey!
Plus, I have cross support for crash, which is also an absolute
necessity for us since we support host development on different
platforms than the target environment.
If I was to support this, I would maintain the one that does
cross-development support, BTW. I think there are enough people
interested in this to make it worthwhile.
I'm very interested in this. But I haven't found this in
Dave Anderson's
Post by Rainer Bawidamann
crash. Is it included there? If not, is you cross-development support
available somewhere?
No, it is not included there. Since the patch is so huge, he has
decided not to take it (for now). I suspect he will change his mind
when Redhat's Wind River friends want this function :).
I can post the current version on my web site, if you like. It is
against a somewhat older version of crash.
Here at Brocade we have a cross version of crash that executes on linux x86 platforms (e.g. PCs) and which analyzes dumps obtained from PowerPC platforms (e.g our FibreChannel switches). The code came to us from MontaVista, who was then where we got our kernels from, but I had to make several fixes to it in order to get it to deal properly with loadable modules. This code deals with the big-endian/little-endian issue, PC's being little and PowerPCs being big.

One of the other complexities is dealing with the varying ways that page tables are organized on different PowerPC processors. On some PowerPC variants, PTEs are 32 bits. On others, they are 64 bits due to the 64 bit physical addresses that those processors implement (my understanding is that the same issue exists among x86 variants, those with and without PAE). crash has to deal with these differences if it is to make sense of the page tables contained in the dump files. Changing the code to deal with these differences at run time was something that I couldn't figure out how to do (and I really didn't have the time to do), so I opted to create two versions, one for each PTE size.

Having spent a non-trivial amount of time looking at this code, I think creating a cross version of crash that will properly analyze dumps from any other processor and any kernel configuration would be a very difficult problem. There are just too many variables to deal with at run-time.

Jeff Haran
Brocade Communications Systems
Post by Rainer Bawidamann
Is you version able to work cross-endian? I.e. debug a dump from a
big-endian system on a little-endian system?
Yes, you can debug cross-endian and between targets with
different type
sizes.
-Corey
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on
ITManagersJournal
Use IT products in your business? Tell us what you think of
them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to
find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Lkcd-general mailing list
https://lists.sourceforge.net/lists/listinfo/lkcd-general
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Corey Minyard
2004-10-07 18:46:44 UTC
Permalink
Post by Jeff Haran
Here at Brocade we have a cross version of crash that executes on linux x86 platforms (e.g. PCs) and which analyzes dumps obtained from PowerPC platforms (e.g our FibreChannel switches). The code came to us from MontaVista, who was then where we got our kernels from, but I had to make several fixes to it in order to get it to deal properly with loadable modules. This code deals with the big-endian/little-endian issue, PC's being little and PowerPCs being big.
One of the other complexities is dealing with the varying ways that page tables are organized on different PowerPC processors. On some PowerPC variants, PTEs are 32 bits. On others, they are 64 bits due to the 64 bit physical addresses that those processors implement (my understanding is that the same issue exists among x86 variants, those with and without PAE). crash has to deal with these differences if it is to make sense of the page tables contained in the dump files. Changing the code to deal with these differences at run time was something that I couldn't figure out how to do (and I really didn't have the time to do), so I opted to create two versions, one for each PTE size.
Having spent a non-trivial amount of time looking at this code, I think creating a cross version of crash that will properly analyze dumps from any other processor and any kernel configuration would be a very difficult problem. There are just too many variables to deal with at run-time.
Jeff Haran
Brocade Communications Systems
Jeff,

I have recently spent quite a bit of time with this code, and I have
most of the PowerPC variants working. Of course, there are always going
to be problems with variants of processors (ARM and PPC are notoriously
bad). But what I have works pretty well, and it's a ton better than not
having anything.

-Corey


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
Loading...