Programming Assignment I

Select one of the assignments below. Assignments marked with can be extended as a continution in Programming Assgnment II. Projects marked with can be (with some additional work and depending on the results) published as scientific papers.

Main goal is to use the tobii-pytracker software and MVTec dataset to design and conduct pilot study on how XAI methods and humans “analyze” images in the task of anomaly detection. Note: you can use eyetracking device emulaiton that is implemented in the tobii-pytracker software, but also try a real hardware if you wqant.

In particular this assumes:

1. Obtain an MVTec datset and plug it in to the tobii-pytracker
2. Train anomaly-detecot using patchcore algorithm: see how to do this – its a one-liner.
3. Get human-eyetracking results and compare with heatmaps that can be obtained from anomalib. Calculate similarities.

Additional extensions:

1. Use more models than just patchore and XAI methods like SHAP which are model-agnostic
2. Implement customModel for tobii-pytracker, that will create bounding boxes, cor convex hulls for the detected anomalies at runtime (See CustomModel section).

The goal is to take the ACFX software and test it both from the perspective of online app: Streamplit app and SDK.

Any issue, bug or inconsistency with documentaiton should be reported to the issue tracker.

Possible extensions to a paper/master thesis: how to use thes emethod to other than tabular cases/models, for instance: Graph neural networks, or images?

Testing WinCLIP model for explainable anomaly detection. See if the model can be used to obtain textual explanations of a detrected anomalies.

Find ready to use frameworks that allwos to measure predictive multiplicity – how much two or more models that provide the same accuracy in predicion are able to provide conflicting predictions in areas not covered in trainig set.

You may start by lookint this paper and its references: RashomonGB: Analyzing the Rashomon Effect and Mitigating Predictive Multiplicity in Gradient Boosting

How to use LNN to obtaion explanations of their predictions? For instance how to extract the activated predicates to outrput the rule that resulted in final decission?

ProtoTSNet is a inherently interpretable DNN for time series classificaiton. Inspect its variant which goal is to guide training procedure with manually created prortypes.

What could be done to improve its predictive power?

Try to implement TSProto variant that will work for images too (use segmentation to detect segments), cluster segments to detect prototypes, build a decision tree that explains a decision using this visual prototypes.

Try to run tutorials of TSProto in Colab (it will require some code tweeking/requirements adjustment to work with numpy 2.0.0. Once done, create a pull request to include the changes in the current implementation of TSProto