Big Data

How to Analyze Big Data

Data Science 4.3(88)7h 7m50 lectures5 sections

What you'll learn

Understand the sources of big data
Deal with unstructured data and integrate data
Know the characteristics of big data
Build a big data strategy
Use the Hadoop system and the Mapreduce model
Install Cloudera and work with the distributed file system
Understand how machine learning relates to big data
Use Knime and Jupyter
Explore data and improve its quality
Handle missing data

About this course

This course teaches Big Data, meaning how to collect large amounts of data, analyze it and use it to extract useful information. You start with applications that give data its value, then the sources of big data from machines, people and organizations, and dealing with unstructured data and integrating it. Next you get to know the characteristics of big data, which are volume, variety, velocity, veracity and value, as well as data science, building a big data strategy and the processing steps. Then you move on to distributed computing, the Hadoop system, the distributed file system and the Mapreduce model, with installing Cloudera and a hands-on application. After that you learn how machine learning relates to big data, installing Knime and working with Jupyter, exploring data, preparing it, improving its quality and handling missing data. The course suits beginners in data analysis and anyone who works with data or wants to enter this field without previous experience. By the end, you will understand how big data works and be able to explore, prepare and analyze it with the tools mentioned.

Expected outcomes

Master big data collection and analysis using modern techniques.
Understand how to use Hadoop for distributed data management.
Apply machine learning techniques to analyze big data.
Master data exploration tools like Knime and Jupyter.
Develop practical projects based on big data analysis.

Learner reviews

These reviews are from learners who bought the content and completed at least 50% of it — there are no reviews from anyone who hasn't gone through the material.

4.3

★★★★☆

88 reviews

5★

4★

3★

2★

1★

★★★★★Seba·منذ 3 شهرشاهد 50٪

تم عرض كل التقييمات (1)

Course content

Big Data Sources

8 lectures

IntroductionDefines Big Data as a continuous, fast-growing flow of data, and links it to the supercomputer idea and cloud computing, which make it possible to store, process and analyze that data.7:04
Applications that Add Value to DataShows applications where Big Data gains value, such as analyzing customer opinions and sentiment to improve products, and human genome analysis in medicine with its huge storage needs.13:25
Big Data SourcesIdentifies the three sources of Big Data: machines, people and organizations, with examples of each, and notes that data can be structured, semi-structured or unstructured.2:42
Machines as Data SourcesExplains machines as the largest source of Big Data, such as sensors, airplanes, cameras and GPS, smart devices and the Internet of Things, and the benefit of analyzing their data, like detecting illness early.7:01
People as Data SourcesExplains people as a source of Big Data, mostly unstructured (posts, photos, video, emails), and the challenges of its volume, velocity, organization and cost of analysis.5:46
Working with Unstructured DataExplains handling unstructured data produced by people using Hadoop, Storm, Spark and NoSQL databases like Neo4j and Cassandra, and the steps to extract value from it.5:38
Organizations as Data SourcesExplains organizations as a Big Data source, where data is structured, stored in relational databases and queried with SQL, often siloed between departments, with the cloud solving the isolation problem.7:08
Data IntegrationExplains combining data in Tableau in two ways: joining related tables with join types such as inner and left, and union to stack similar data from different sheets or sources.4:46

Characteristics of Big Data

7 lectures

Introduction to Big Data CharacteristicsIntroduces the criteria that characterize Big Data: volume, variety, velocity, veracity and valence, plus value and how individuals, companies and society benefit from it.2:59
VolumeExplains volume as the first Big Data criterion, the units of measurement from megabytes to zettabytes, and the challenges of large volume in storage, retrieval, transfer, networking, cost and performance.5:20
VarietyExplains variety as a Big Data criterion and its link to complexity, with its four dimensions: structural, media, semantic and availability, using an email as an example of a data object in many forms.5:14
VelocityExplains velocity as a Big Data criterion in creating, storing and analyzing data, the difference between real-time and batch processing, and how speed affects cost.5:42
VeracityExplains veracity as a Big Data criterion for data quality and trustworthiness, how accuracy drops as volume and sources grow, and why measuring it matters, as in the Google mistake.7:52
EquityExplains valence as a Big Data criterion expressing how connected data items are, the chemistry origin of the term, and how greater connectedness adds complexity to modeling and analysis.4:34
ValueExplains value as the last Big Data characteristic and applies all six characteristics to the Catch the Pink Flamingo game example, from volume, variety and velocity to veracity and valence.7:15

Data Analysis Steps

4 lectures

Data ScienceExplains Data Science and how it turns Big Data into insights and decisions, which fields it grew out of, and the technical, business and soft skills of a data scientist.10:19
Building a Big Data StrategyExplains building a Big Data strategy: its four parts (goal, policy, plan, action), short and long term goals, organizational buy-in, implementation stages, and policies on data lifetime and quality.7:55
The Five Components of Data ScienceExplains the five components of data science: people, purpose, process, platforms and programmability, and the role of each in getting value from Big Data.5:43
Data Processing StepsExplains the steps of the data science process: acquiring data, preparing it (exploration and preprocessing), analyzing it, then presenting results with reports and visualization tools.8:33

Working with Hadoop

12 lectures

Distributed Operating SystemCovers the foundations of distributed systems before Hadoop: how data is spread across nodes and racks, how replication makes the system more fault tolerant, and how the difficulty of updating data is handled.10:26
Distributed Computing ProcessExplains computing on a single node versus parallel nodes, commodity clusters, distributed computing over the internet, data parallelism, and how parallel connection tolerates node and rack failures.8:02
Big Data Programming ModelsExplains programming models for Big Data and their requirements, then MapReduce as a model that distributes processing and combines results, supporting large volumes, error handling and scale-out.7:36
Why Hadoop?Explains why Hadoop is used and its goals: scalability, handling failures, supporting different data types, running concurrent jobs and delivering value, with a preview of its upcoming components.4:33
Explaining the Hadoop SystemExplains the Hadoop ecosystem and its history from Google's MapReduce, its layers from HDFS at the bottom to Hive and Pig, tools like ZooKeeper and NoSQL databases, and distributions like Cloudera.13:39
Distributed File System in HadoopExplains Hadoop's distributed file system HDFS: splitting large files into blocks, replication to handle node failures, and its two components NameNode and DataNode in a master-worker relationship.7:42
Resource ManagementExplains YARN as Hadoop's resource manager working with HDFS to run multiple applications like MapReduce and Spark, its design with Resource Manager and Node Manager, and what it added over version one.7:29
Simplest MapReduce Programming ModelExplains MapReduce as the simplest Big Data programming model: Map produces key-value pairs on each file, then shuffle and sort, then Reduce combines the values, using a word count example.10:56
Computational ComputingExplains cloud computing and how it enabled working with Big Data, by renting resources over the internet as needed instead of buying hardware, giving scalability and resource management.9:28
Installing ClouderaWalks through installing Cloudera on a virtual machine using Oracle VM VirtualBox, from downloading and importing the file to running it, then shows the Hadoop tools inside such as Spark, NameNode and YARN.10:35
Working with Distributed File System in ClouderaExplains working with HDFS from the terminal in Cloudera using Hadoop commands to copy a file from local to HDFS, list files, delete them, and copy back to local.11:33
Practical ApplicationA hands-on Hadoop exercise running the Word Count MapReduce example: setting input and output files, then browsing the output folder and final result file, and keeping the work saved in Cloudera and VirtualBox.16:07

Machine Learning and Big Data

19 lectures

A Quick Look at Machine LearningDefines Machine Learning as building systems that learn from data without explicit programming, with everyday examples like reading handwritten digits, and how it differs from Data Mining and Data Science.11:43
Machine Learning and Big DataExplains the best-known Machine Learning techniques: classification, regression, cluster analysis and association analysis, and the difference between supervised and unsupervised learning based on whether the target is defined.14:02
Operations Implemented Using Machine LearningReviews the steps of the Machine Learning process: acquiring, exploring and preprocessing data, then building and testing the model on a random sample, repeating steps if the model misses the goal.8:41
Data ExtractionExplains the CRISP-DM methodology for data mining and its phases: business understanding, data understanding, data preparation, modeling and evaluation, returning to business understanding if results are unsatisfactory.9:26
Overview of Programs and ToolsPresents tools for Machine Learning with Big Data: KNIME, a graphical platform using drag and drop and connected nodes in a workflow, and Spark MLlib, which relies on writing code.9:29
Installing KnimeWalks through downloading and installing KNIME, then using the Node Repository and dragging nodes like Excel Reader into the workflow and configuring them, with an example of splitting data for training and testing.13:22
Working with JupyterExplains downloading Spark and Anaconda, launching Jupyter Notebook and Jupyter Lab, working with and uploading project files, and writing Python code in the browser to handle data.9:34
Features and DataExplains basic data terms in Machine Learning: feature and sample and their alternative names, and the difference between numeric and categorical variables, with table examples.8:24
Data ExplorationExplains data exploration and its goals, its techniques of summary statistics and visualization, and what it reveals: correlations, trends, value ranges and outliers.5:38
Descriptive StatisticsExplains summary statistics and their categories: measures of center like mean and median, measures of spread like standard deviation, variance and range, plus shape and dependence between variables.11:53
Data ValidationExplains using a contingency table to describe categorical variables, plus quick data validation questions: checking row and column counts, date validity, and the number of missing values before processing.8:30
Exploratory VisualizationsExplains data visualization in the exploration stage using different plots: Histogram for the center and distribution of data, Scatter Plot for the relationship between two variables and outliers, and Box Plot for comparing distributions.13:56
Using Visualization with KnimeA hands-on KNIME exercise on data exploration: creating a Workflow, reading the Daily Weather.csv file, then drawing a Histogram and Scatter Plot of air temperature and relative humidity, showing a negative correlation between them.8:59
Data PreparationExplains why data preparation matters before analysis and its goals, data quality problems such as missing, duplicate, noisy and outlier values, and Data Wrangling with its two categories: feature selection and feature transformation.5:04
Data QualityPresents the main data quality problems: missing values, duplicate data, invalid or inconsistent data, noise and outliers, and their negative effect on analysis results, which is why they must be handled first.4:40
Data Quality Processing TechniquesExplains data quality techniques: removing samples with missing values or imputing reasonable values, merging duplicate data, correcting invalid data from another source, and handling outliers.6:51
Feature IdentificationExplains Feature Selection within data preparation: removing features with many missing values or no use such as ID, combining features, deriving new ones, and recoding, such as grouping age into categories.10:05
Feature TransformationExplains Feature Transformation and its purpose, including Scaling by changing the range of values, reducing noise with filters on audio and images, and Aggregation of values, to simplify data before analysis.7:03
Practical Application on Missing DataA hands-on KNIME exercise handling missing values with three solutions: removing samples with missing values in a given column, replacing them with the column mean, or removing all incomplete samples, comparing the Histogram before and after.17:01