85 lines
5.5 KiB
TeX
85 lines
5.5 KiB
TeX
\chapter{Evaluation}
|
|
\label{eval}
|
|
|
|
We tried to asses the quality of ICFS by the following metrics:
|
|
|
|
\begin{itemize}
|
|
\item Security : Does ICFS effectively mitigate unauthorized access by untrusted processes?
|
|
\item Usability : Does the interactive model reduce configuration burden while maintaining user control?
|
|
\item Performance : What is the overhead introduced by ICFS compared to native filesystem operations?
|
|
\item Compatibility : How well does ICFS integrate with existing workflows and software (e.g., CLI tools, legacy applications)?
|
|
\end{itemize}
|
|
|
|
\section{Test environment}
|
|
|
|
For performance and usability tests, we used an HP Pavilion Laptop 15-cc563st with an Intel® Core™ i7-7500U processor, a Western Digital WDS250G2B0B WD Blue 3D NAND internal M.2 SATA SSD and 12 GB of DDR4 RAM, running Fedora Linux 42 (Workstation Edition) with Linux 6.14.5-300.fc42.x86\_64 kernel, GNOME 48 desktop environment with Wayland window session.
|
|
|
|
For additional compatibility tests, a KVM virtual machine running on the same laptop was used. The OS on the virtual machine was Debian GNU/Linux 12 (bookworm) x86\_64 with 6.1.0-27-amd64 kernel, GNOME 43.9 with X11 windowing system. The virtual machine was given 2 CPU cores and 2 GB of RAM.
|
|
|
|
To assess the qualities of ICFS during regular usage, we have mounted it to a directory pre-filled with files, and attempted to access the files with different software. While trying to simulate common use-case scenarios, we tested the following programs' compatibility and usability with ICFS:
|
|
|
|
\begin{itemize}
|
|
\item {\TeX}studio.
|
|
\item Programming IDE/text editor (Neovim).
|
|
\item A web browser (Mozilla Firefox).
|
|
\item File manager (GNOME Files, also known as Nautilus).
|
|
\item File synchronisation software (Syncthing).
|
|
\item Standard command line core utilities (e.g. \verb|ls|, \verb|cd|, \verb|grep|, ...).
|
|
\end{itemize}
|
|
|
|
We chose these programs not only because of their commonality, but also since they exhibit fairly diverse patterns of filesystem usage that are relevant in practice.
|
|
|
|
\iffalse
|
|
|
|
For performance testing, special scripts were written, that test the speed of `open` operations. We did not test other operations, since
|
|
|
|
\fi
|
|
|
|
\todo[inline, caption={Complete Evaluation}, author={\textbf{Draft note}}]{The rest of this chapter is \emph{very} incomplete, and only contains a brief and informal talk about the issues I am facing right now. This is not the actual thesis-worthy text. All issues discussed were relevant as of 13.05.2025}
|
|
|
|
\section{Security}
|
|
|
|
\todo[inline, author={\textbf{Draft note}}]{What to even say here?}
|
|
|
|
\section{Usability}
|
|
|
|
While it is difficult to put an objective score on the ease of use of any system, we measure the usability of ICFS by the amount of dialogues the user has to consciously react to while using the system.
|
|
|
|
In our testing we found that most programs' filesystem usage was quite manageable with ICFS. {\TeX}studio and Neovim only required the user to interact with a single dialogue when opening existing projects, and none when creating new files.
|
|
|
|
The file manager Nautilus generally did not require any access permissions to function, but generated more dialogues than initially expected. When started, Nautilus opened files to scan them to display mini-previews of their contents.
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
\includegraphics[width=0.6\linewidth]{./images/picture-preview-nautilus.png}
|
|
\caption{Example of a picture mini-preview: image on the left is the scaled-down version of the actual image stored in the file.}
|
|
\end{figure}
|
|
|
|
So, for every image to be scanned, a permission was required. It might seem like a simple problem to solve at first: after all if the user gives even a temporary permission to the entire folder, all images should open seamlessly. The issue is, that Nautilus uses another program (gdk-pixbuf-thumbnailer) to scan images on its behalf, one image at a time. Hence, every temporary permission given to that program, will only last until the next image is opened. One practical solution to this problem is to give gdk-pixbuf-thumbnailer a permanent permission to access the folder in question.
|
|
|
|
The web browser used the filesystem only when a file was downloaded and consequently needed saving. To select the path for the file, web browsers use file selection dialogues provided by the system file manager.
|
|
|
|
\begin{figure}[H]
|
|
\centering
|
|
|
|
\includegraphics[width=\linewidth]{./images/file-selection-dialogue.png}
|
|
|
|
\caption{}
|
|
\end{figure}
|
|
|
|
This is mostly solved in my experience. It wasn't as annoying to use the terminal programs as I initially expected.
|
|
|
|
As for terminal programs, I see these possible \sout{solutions} improvements:
|
|
\begin{itemize}
|
|
\item Use SID and TTY to identify a shell session (like \verb|sudo| does).
|
|
\end{itemize}
|
|
|
|
\section{Performance}
|
|
|
|
Performance of ICFS is terrible. \sout{Unfortunately, I was unable to make perf work with it for some reason, so I don't really know what is slowing operations down. So those are my speculations for what \emph{may} be the bottleneck. A lot of it is caused by it's design. For example, ICFS needs to look through procfs to get process creation time, and there is no way of going around this it seems.}
|
|
|
|
I managed to get the \verb|perf| to work: it showed that almost all time was consumed by libfuse, not my program. My code used something like 0.0001\% of runtime on a pretty heavy test. I am not quite sure what to do, since libfuse is already optimised to smithereens. I guess I will just write how it is and not touch the performance ever again.
|
|
|
|
\section{Limitations}
|
|
\label{eval:limit}
|