View Proposal

---

Proposer

Hans Wolfgang Loidl

Title

Parallel Big-data Computation on Hadoop

Goal

Implement and assess the performance of a typical big-data application on the Hadoop software infrastructure for parallel pattern computation

Description

Big Data computing poses challenges on several fronts. It requires the processing of enormous amounts of data, which is beyond the computational capabilities of commodity server hardware. Therefore, parallel programming technologies need to be applied to perform the computations in time. The goal of this project is to use the Hadoop [1] software infrastructure (or a related infrastructure such as Apache Spark) in order to implement a typical, data-intensive application. Potential application domains are high-performance scientific computation or bio-informatics. The application can be implemented either in Java, using the low-level Hadoop API, or in one of the emerging scripting languages supported by the Hadoop framework, such as Pig or Hive. This project should summarise the effort involved in prototyping, transforming and implementing the initial application, fundamental problems encountered in this project, which might be problematic in automatising this process, and assess the overall performance and scalability of the final, parallel version.

Resources

Hadoop (or similar) installation

Background

Good general programming skills; some background on parallel programming (e.g. F21DP)

Url

External Link

Difficulty Level

Moderate

Ethical Approval

None

Number Of Students

3

Supervisor

Hans Wolfgang Loidl

Keywords

parallel computing, big-data

Degrees

Bachelor of Science in Computer Science
Bachelor of Science in Computer Systems
Master of Engineering in Software Engineering
Master of Science in Artificial Intelligence
Master of Science in Artificial Intelligence with SMI
Master of Science in Computer Systems Management
Master of Science in Computing (2 Years)
Master of Science in Data Science
Master of Science in Network Security
Master of Science in Robotics
Master of Science in Software Engineering
Bachelor of Science in Computer Science (Cyber Security)
Bachelor of Science in Statistical Data Science
BSc Data Sciences
MSc Applied Cyber Security