Hi Christian,
On 11/20/2017 10:00 PM, Christian Hechtl wrote:
Hi Ralf,I'm not sure if I understood your question correctly.
i have one quick question regarding the new ML analysis. You find it
inline.
Am 07.11.2017 um 20:16 schrieb Ralf Ramsauer:
Hi,So the reason for this and the next step ist to reduce the evaluation
I'm about to release a new version of PaStA in a couple of days. A short
summary of new features:
- Mails are entirely treated like commits: patchstack analysis and
upstream analysis are now completely merged. In this way, we can
also map patches on the ML that are not linked to upstream patches.
Beware: command line interface slightly changed
- new fancy log/output interface. Pretty nice to see how long an
analysis actually runs.
- Filenames of patchstack / mailbox analyses changed! (see git log)
- PaStA's webfrontend can be found in the webfrontend branch
- several minor bugfixes
Please checkout next, everything is queued there. Things on next might
still change.
This is the new workflow of the mbox analysis:
1. ./pasta mbox_prepare list-name filename
2. ./pasta cache -create mbox
3. ./pasta analyse -mbox init
4. ./pasta analyse -mbox rep
5. ./pasta rate
6. ./pasta analyse -mbox upstream
7. ./pasta analyse rate
The result will be stored in
PaStA-resources/[project]/resources/similar-mailbox
Explanation of those steps:
1. This step unfolds the huge ML dump file and creates a directory
structure that is suitable for further processing. Takes a while.
2. The creation of the mailbox cache does not only create a python
pickle file that contains the whole cache, it also filters out and
invalidates email that do not contain patches. BTW, these emails might
still contain patches that we just can not parse yet. Could be object
for further investigations... On our 16GiB Linux LKML dump, we currently
have 1684320 mails with no patches, and 537406 mails where we detected
patches. IOW, ~25% of all mails on the LKML dump contain parseable
patches.
3. The init step creates an equivalence class in the result file for
EACH email. This is the precondition for any further analysis: The
initial assumption is, that every patch is dissimilar in the beginning,
and by successive analyses, we get coarser equiv' classes.
4. The analyse rep ('analyse representatives') compares representatives
of all equivalence classes against each other. As just mentioned,
initially every message-id is in its own equivalence class, so we have
537406 equivalence classes in the beginning, which actually leads to
537406^2 = 2.8*10^11 comparisons. Huh, quite some effort...
The problem set is reduced by PaStA's fancy preevaluation step:
We have a set of n commits that we want to compare against another set
of m commits. In case of the representative analysis those sets are
the same, so m = n.
For each commit, determine a list of files that are affected by
this commit. Iteratively, Create a dictionary that maps filenames to
commits. This allows us for determining a set of commits given by a
filename (with reasonable low effort).
Make use of this assumption:
Two commits can only be similar, if they touch at least one common
file.
Bingo, this preevaluation step reduces the problem set by a significant
factor. (I saw reductions up to a factor of ~400!)
Successive runs of analyse rep can be used to shrink the number of
equivalence classes.
steps of step 6? Or is there another reason for this?
5. Do the eventual mapping by applying thresholds.
6. Determine a complete representative system of all messages, and
compare them against upstream.
Basically same as step 4, with the difference that
m = complete set of representatives (IOW, a message ID per class)
n = all upstream hashes of interest
The representative of an equivalence class is the mail in a class with
the youngest date. Rationale: if a patch is accepted, its upstream
commit is probably very close to the youngest submission on the ML.
7. Do the eventual mapping by applying thresholds.
Cheers,
Ralf
Best regards,
Christian
