Tripoli
Disclaimer: Tripoli is a prototype. It was easy to put in
the features that it has; it will be much harder to add/change anything
from here. It also has some bugs. Use at your own risk!
About Tripoli
Tripoli is a prototype of a tool to help programmers quickly narrow
down where a bug is by looking at which code was executed in one run
but not another.
More jargonly, it is a differential code coverage tool implemented
as an
Eclipse plug-in and based on
EclEmma, which in turn is based on
Emma.
For example, if you wanted to figure out which code handles changing
the colour of a square, you would do two runs. Tripoli would log
which methods were executed in both runs, and tell you which were
executed in the first run but not the second.
- In the first run (the "fat" run), you would create a square,
open the colour picker, select a colour, press okay, then quit the
application.
- In the second run (the "skinny" run), you would create a square,
open the colour picker, select a colour, press cancel, then quit
the application.
Both runs would run initialization code, code for creating a square,
code for creating the colour picker dialog, and code to handle colour
selections. However, only the first ("fat") run would run the code to
actually colour the square. Tripoli would then tell you about only
that square-colouring code.
Tripoli has two ways of teling you. First, it colours the code in
your Eclipse source editors. Executable code that was executed only
in the first ("fat") run is highlighted in green, while all other
code is highlighted in red, as seen in this picture:
It also shows a view listing all the methods that were executed in
the first run but not the second, as seen in this picture:
How to install Tripoli
If you already have EclEmma installed in your Eclipse, you will need
to remove it, and also clean out any
emma.jars that are
hiding in various corners of your system. They conflict with Tripoli.
- Uninstall EclEmma in Eclipse.
- Search for emma.jar in your file system.
Sometimes Eclipse puts a copy in
./configuration/org.eclipse.osgi/bundles/number/number/.cp/emma.jar.
- Remove copies of emma.jar that are in subdirectories of
any Eclipse(s) that you are going to use.
- Look in your workspace for com.mountainminds.*.
- Remove copies of emma.jar that are in your workspace.
Once you have no EclEmma in your system:
How to run Tripoli
To do a diff of two runs, you need to get code coverage information
twice and "subtract" the second from the first. This means that your first
run needs to be your "fat" run (the run with more code, e.g. ABCDEF),
and the second needs to be your "skinny" run (the run with less code, e.g.
ABCD).
To do a coverage run, find the
Run Coverage button
(
) in the
toolbar, to the left of the regular
Run (
) and
Run as Debug buttons.
Press the
Run Coverage button, and do your "fat" run.
Quit, and you should see the
Coverage View appear:
Press the
Run Coverage button again.
Do your "skinny" run and quit the application.
Now that you have your "fat" and "skinny" runs done, you need to diff them.
Find the
Coverage View, and find the "Diff Sessions" button
(
) in the
Coverage View toolbar.
Press it.
Congratulations, you now have a Tripoli diff! Lines of code that
were in the "fat" run but not the "skinny" run will be green, and all
other executable lines will be (well, should be) red.
The coverage information from a run is called a "session".
The
Coverage View lets you look at information from one session.
The
Coverage View lists all the classes and methods that were
executed in the region of interest. (For non-diff sessions, that
of course means the all the lines that executed in the run, not just
the diff.) You can change which session you look at via the
Select Session toolbar button (
).
Don't worry too much about what is in the code coverage percentage column
for diff sessions. Those are probably
wrong -- I haven't worried about making them be sensible or meaningful.
Only worry about whether a method is in the Coverage View list or not.
If you want more information, you can read up on the
EclEmma documentation.
(Just remember that everywhere where the EclEmma documentation says
"merge", think "diff".)
Warnings!
The order of your runs matters!
Tripoli effectively subtracts the second coverage
run from the first coverage run, so the "fat" run always has to be
first and the "skinny" run second. EclEmma is not commutative.
Red is ambiguous!
When looking at a Tripoli diff run, red is ambiguous. Lines that are
red include:
- lines that weren't executed in either the fat or the skinny run
- lines that were executed in both the fat or the skinny run
- lines that were only executed in the skinny run
If you really need to know why a line is red (i.e., which of those three
conditions it satisfied), you can look at the fat and skinny runs
individually by changing the coverage session via the Select
Session button ()
in the Coverage View toolbar.
Don't try to diff more than two sessions!
I am not completely sure what the results mean, but I suspect
it is not likely to be what you want.
How to run Triopoli effectively
Experience helps in figuring out what to do in your "skinny" run. (The
"fat" run is usually easy -- usually it's just whatever you do to
reproduce the problem.) Here is what I have learned so far:
Do as much as possible in the skinny run.
Because of how we mentally group actions, it is easy to forget a step
that should go in the skinny run. If you want to see what code colours
a square, you might be tempted in your "skinny" run to only create the
square, then quit. To you, the colour picker dialog is part of colouring
the square, but there is a lot of code associated with displaying the
colour picker dialog. Unless you display the colour picker dialog, all
the colour picker dialog methods will show up in your diff.
In general:
- If you want to see what the menu action Foo does, in your skinny,
highlight the menu option Foo and then slide off of it (without
selecting it). Menu options sometimes are created lazily, so if
you don't highlight the menu option, the menu creation methods might
show up in your diff.
- If you want to see what code handles resizing a window, make sure
in your "skinny" run to hover the cursor over the corner long enough
for the cursor to change.
- If you want to see what code is executed after you press OK
on a dialog, bring up the dialog and do everything that you would do,
but press Cancel instead of OK.
Don't be afraid of doing extra things in your skinny run.
Things that are in the skinny run but not in the fat run won't show
up in the diff. While you have to be careful that you don't accidentally
do anything that is related to the problem area, you can do unrelated
things all day long. For example, suppose there was a problem with
changing the width of a line via a menu action. In your skinny, for your
first try, you might select the line, highlight the menu action for
changing the width, slide off the menu item, and then quit. If you then
see a bunch of things related to "Undo" in the diff, you might guess that
those are generic to all actions, not just to changing the width.
In that case, you might want to do a skinny run where you do everything
as before, except that before you quit, you change
the colour
of the line via a menu action. You will probably get a more precise
diff that way.
Don't be afraid to try a few different diffs. It is frequently
faster to redo a diff than to examine a lot methods, so if you get a lot
of methods, try doing something a little different on your skinny run.
You might get a more precise diff. And if you don't, you can always go
back to the previous diff.
Instead of omitting a step in your skinny run, you can try
doing something different. For example, instead of omitting the
step where you change the width of a line, try changing the colour
instead.
FAQ
Why doesn't Tripoli work for me?
I have done really minimal testing. I run on Linux, and have tried
it on Eclipse 3.2 and 3.3.1. I've also done limited testing on
Windows Eclipse 3.2. I have no idea if it will work on anything else.
If it is so raw and untested, why are you releasing it now?
Because I'm planning on blogging about the research I just did
using Tripoli, and don't want to answer a bunch of email messages
that all ask, "Can I try it?" Also, it would be a lot of work -- far,
far more than I have alrady put in --
to take it to the next level.
Why do you say it would be a lot of work to improve Tripoli?
First, Tripoli is built on top of Emma, which is built on top of
EMMA, and EMMA is not supported. Any changes I want to make to EMMA,
I'll have to do myself. Second, EMMA 1.2 is designed somewhat
monolithically -- you start it and you get back a session. There's no
starting and stopping logging during execution, and I'd like to
be able to set "start logging" and "stop logging breakpoints".
Third, EclEmma and EMMA really are structured around single sessions,
which have one bit of information about lines (hit/not hit).
I would need to do a lot of heavy thinking to figure out
how to best handle paired sessions.
Fourth, I repurposed EclEmma/EMMA's merge function for the diff.
It's easy to rewrite EMMA's merge function to allow either merging
or diffing based on a boolean that's passed in, but the place in
EclEmma that knows if it is a merge or diff has poor communication
to the place that does the merge. (NB: I'm sure at least one of the
things I said here was wrong (from being uninformed), but I don't know
which one, and I bet only one is wrong.)
What did your research show?
With only a slight bit of hand-waving, that Tripoli was really
useful. See my
technical report for the gory details.
Why is it called Tripoli?
It is an enhanced EclEmma, so EEE, or "triple-E",
which, in Canadian English, sounds like "Tripoli".
Why do I have to remove emma.jar from everywhere? What
is that file?
EclEmma is built on top of EMMA, which is an IDE-agnostic code
coverage tool. I had to make a few changes to EMMA, which are reflected
in emma.jar. If you aren't careful about cleaning out the old
emma.jar files, Eclipse can use an older
copy of emma.jar, which will not give you the results you want.
What changes did you make to EMMA?
I changed the "merge" function to do a "diff" instead. I changed
about three lines of code in CoverageData.jar.
I want to do both merging and diffing! Why can't I do both?
Sorry, by the way that EclEmma is written, it was easy to do
one or the other but not both. I might fix that someday, but
not for quite a while, if at all.
I don't like the the red and the green source line colouring!
Can I change the colours?
Yes, you can. See the
EclEmma
documentation (under Highlighting Preferences).
Why aren't the definition lines of methods or comments ever coloured?
EclEmma does its counting during execution. The definition lines
of methods (which look like public int foo(int bar)) don't
make it into the bytecode as distinct entities, so they don't get
counted. Comments don't make into the bytecode at all, so they don't
get coloured either.
Sometimes none of the lines in a file are coloured! Why not?
Sometimes, none of the lines of code in a file will be coloured.
This will happen if you modified the code in between the fat and
skinny runs. (If you really care about why, email me. Otherwise,
just accept it -- explaining it seems to confuse people.)
It also happens randomly sometimes. Sometimes there is just a second or
two delay before it gets coloured; sometimes it does't ever colour.
This is a known bug, but I have no idea what the problem is, and
am not sure if it is a Tripoli problem or an EclEmma problem. Sorry.
(Remember, this is a prototype!)
What does it mean if a line is coloured yellow?
It means that it is partially executed. See the fine
EclEmma
documentation on line colouring.
Kaitlin Duck Sherwood
