Statement of Purpose Essay - ETH Zurich

Focus. My research goals are oriented towards efficiency, explainability (including adversarial conditions) and robustness of models for knowledge-intensive tasks, machine translation systems or ML/DL in general. I am attracted to NLP because it is a very vibrant and interdisciplinary field that offers fascinating problems requiring multiple perspectives. I am less inclined towards pushing model performances up a few points every conference and would rather focus on methods that help in the aforementioned areas. Coming from a computer science background strongly focused on mathematics, I always strived to get a more thorough understanding of algorithms and models and try to build them as robustly as possible. Neither CPU/GPU, memory nor training data are unlimited and making informed design choices regarding the model or the implementation can help reduce these resource requirements. I believe that critical progress can be made not only in the areas of analyzing and improving modern NLP models but also in tools for machine learning tools as well. My interest in model robustness and reliability is also projected on the tools I would like to use more: instead of just the omnipresent Python, the programming languages in which I wish to become more proficient during my research are typed, compiled and more low-level, such as Rust. I also want to start contributing more to public projects to become more connected with other researchers. For these reasons, I find dependable and efficient NLP attractive and wish to learn more about the mathematical aspects of ML/DL in detail. I am greatly interested in knowledge-intensive NLP tasks which include question answering, fact-checking or working with knowledge-bases in general. I find the fusion of non-parametric and parametric knowledge very interesting and would like to explore it further. I would also like to continue the development of using KBs for other tasks which are not usually associated with KBs, such as language modelling. This and the exploration of fusion has the benefit of making the models more robust and explainable but also more efficient by predicting which inputs require KB access. Because retrieval is resource demanding, work on improving its efficiency and also tackling the growing KB size requirements is also crucial and would make it more accessible for more research departments and other applications. Motivation. I first came into contact with academia during my bachelor when I joined the Bergamot project with my experimental MT user interface. While working as a student assistant with Ondrej Bojar, I further developed the prototype and designed user experiments, eventually extending the project into my bachelor’s thesis and several publications (including ACL-level venues). During this time, I got well-acquainted with MT and had the chance to publish more papers centred around user interaction with MT, also with colleagues from other European universities, one of which was a collaboration with a company providing commercial MT services. This experience sparked my interest in NLP, and during my last year of bachelor studies I took on multiple master-level NLP courses from which I received the best grades. The research experience at Charles University was very formative, and I learned a lot of skills from my colleagues and supervisors regarding collaboration, academic writing, presenting at conferences and research in general. For my masters, I was given the opportunity to explore new teaching and research environments at two new universities. During the first semester at Saarland University, I tried to broaden my study curriculum by taking classes such as ML in Cybersecurity and Explainability Methods of Neural Networks. In both cases, I genuinely enjoyed writing the semester papers and extended both of them to full articles which got later published. This was my first experience in designing, developing and publishing projects on my own, without supervision. These smaller projects were related to MT model distillation and neural MT-induced word alignment. At the new university, I became a student research assistant under the supervision of Dietrich Klakow. In this job, I am working on topics related to information retrieval and knowledge-intensive NLP. My first project is concerned with reducing the document index dimensionality to make using it more accessible even with lower computational and memory resources. The other project is a survey of how different NLP systems which rely on knowledge base access share the same abstract architecture. This would make it possible to e.g. use advances in question answering to improve language modelling with KB. The experience has vastly broadened my horizons and convinced me that there is a lot of exciting research in NLP that can be done at the intersection of various subfields. For my diploma thesis, I am planning to follow up on the dimensionality reduction research to find out what is the best way to split articles to make an index in a textual database (e.g. Wikipedia) and how much can be filtered to save on space and retrieval time and storage requirements. During the second semester at Saarland University, I also tutored an advanced masters course Statistical Natural Language Processing. I very much enjoyed tutoring and explaining concepts to other students, and dedicated a lot of time to preparing the class material and assignments. Next semester I am going to tutor a class Neural Networks Implementation and Applications and in the future, I will try to look for more teaching opportunities. Conclusion. My areas of interest are in line with the publications of Mrinmaya’s Lab, e.g. data efficiency (learning pipelines with limited data), and robustness (robust pre-trained language models). His single-author work on knowledge-graph compression is closely related to my current research project (knowledge-base index compression). From interdisciplinary perspective, this can be complemented by focus on fairness and accountability for which supervision of an expert in sociology would be beneficial. For parts of research here I could also utilize my experience with working with human subjects in the context of NLP tasks. Through the experience both in the industry and research departments, I grew to deeply appreciate the whole research process (reading, prototyping, discussions, running experiments, writing, presentation) and wish to participate in more projects. I am determined to pursue a PhD for both active research and teaching and hope to eventually seek a tenured position. I believe, that the combination of supervisors from different fields of expertise could allow me to join the two foci in my interests and do impactful research. Because of these reasons, I would like to spend the following years at ETH Zurich.