- 60
- 315 927
Data Engineering For Everyone
Приєднався 12 чер 2021
Welcome to our Data Engineering For everyone channel! Here, we aim to make the complex world of data engineering accessible to everyone. From beginners to advanced users, we provide in-depth tutorials, tips, and best practices for designing, building, and maintaining data systems. From data warehousing and ETL, to data modeling and big data technologies, we cover it all. Join us as we demystify the field of data engineering and help you become a pro in no time.
Here, we will explore the exciting world of data engineering and help make it accessible to everyone. From the basics of data storage and processing to advanced techniques for big data analysis, we will cover it all. Join us as we dive into the tools and technologies used by data engineers to build and maintain the systems that power today's data-driven organizations. Whether you're a beginner or a seasoned professional, this channel is for you
Contact me @ www.linkedin.com/in/nilendrasingh
Here, we will explore the exciting world of data engineering and help make it accessible to everyone. From the basics of data storage and processing to advanced techniques for big data analysis, we will cover it all. Join us as we dive into the tools and technologies used by data engineers to build and maintain the systems that power today's data-driven organizations. Whether you're a beginner or a seasoned professional, this channel is for you
Contact me @ www.linkedin.com/in/nilendrasingh
MS Fabric End to End Project on Fraud Analytics
Introduction : 00:00
Data Sources : 03:09
Workspace creation : 04:15
ADLS Gen2 Shortcut : 05:29
AWS S3 Shortcut : 08:20
Onelake explorer : 11:35
Dataflow Gen2 : 13:40
Load to Delta tables : 17:33
Notebook for Machine Learning : 19:10
MLFlow : 23:45
SQL Visual Query : 32:46
Semantic Data Modelling : 34:26
MLFlow Result : 36:12
Dashboarding : 37:19
Recap : 38:50
Data Sources : 03:09
Workspace creation : 04:15
ADLS Gen2 Shortcut : 05:29
AWS S3 Shortcut : 08:20
Onelake explorer : 11:35
Dataflow Gen2 : 13:40
Load to Delta tables : 17:33
Notebook for Machine Learning : 19:10
MLFlow : 23:45
SQL Visual Query : 32:46
Semantic Data Modelling : 34:26
MLFlow Result : 36:12
Dashboarding : 37:19
Recap : 38:50
Переглядів: 287
Відео
Data Engineering for Everyone: Top Choice in Bard AI's YouTube Plugin!
Переглядів 2347 місяців тому
🚀 Welcome to Data Engineering for Everyone, the go-to educational resource for aspiring and experienced data engineers alike! Our channel has recently been recognized as a top result in Bard AI's UA-cam plugin, and we're thrilled to share our knowledge and passion with an even wider audience.
Demystifying Microsoft Fabric: A Collaborative Dive with Microsoft GBB
Переглядів 2039 місяців тому
In this enlightening session, Microsoft GBB Ian Clarke takes us through the captivating world of Microsoft Fabric. This video is tailored to break down the technical jargons and unfurl the essence of MS Fabric in the simplest language possible. Whether you're a Data Engineer or a Data Scientist, this session has something in store for you. Ian Clarke elaborates on how MS Fabric fosters seamless...
Exploring the Power of Generative AI: An Introduction to Cutting-Edge Technology
Переглядів 349Рік тому
In this video, we will be exploring the exciting field of generative AI and its potential applications. We will cover the basics of how generative AI works and its underlying technology, including neural networks and deep learning. While we will touch upon the basics of GPT (Generative Pretrained Transformer) language models and their capabilities, we will be diving deeper into the details of G...
From Data Ingestion to Model Deployment: A Comprehensive Azure ML and Databricks End to End Project
Переглядів 6 тис.Рік тому
From Data Ingestion to Model Deployment: A Comprehensive Azure ML and Databricks End to End Project
Expert Insights on Spark Memory Management: Answering the Top Interview Questions
Переглядів 915Рік тому
Are you preparing for a Spark-related job interview and want to showcase your knowledge of memory management in Spark? Look no further! In this video, I will walk you through the most commonly asked Spark memory management interview questions and provide in-depth answers to help you succeed. From understanding the role of executor memory and storage memory, to managing memory spills, this video...
Top Spark Performance Tuning Interview Questions and Answers
Переглядів 1,9 тис.Рік тому
Get ready for your Spark performance tuning interview with this comprehensive video. We cover the most commonly asked interview questions on optimizing Spark performance and provide clear, concise answers to help you ace the interview. Whether you're an experienced Spark developer or just starting out, this video is a must-watch for anyone looking to improve their Spark performance tuning skills
Master Spark Partitioning and Bucketing: Top Interview Questions Answered
Переглядів 1,8 тис.Рік тому
#crackSparkInterviews #AllAboutSpark In this video, we will discuss the most frequently asked interview questions on Spark partitioning and bucketing. Spark is a popular big data processing framework, and a deep understanding of partitioning and bucketing is crucial for optimizing the performance of Spark jobs. We will cover topics such as the difference between partitioning and bucketing, the ...
Most Asked interview question in Apache Spark ‘Joins’
Переглядів 1,9 тис.Рік тому
#sparkinterviews Learn the ins and outs of Apache Spark Join operations in this comprehensive interview-style tutorial . Discover the different types of Spark joins, including inner join, outer join, left join, right join and more. Get hands-on experience with real-life examples of joining large datasets using Spark. Whether you're a data engineer or data scientist, this video is a must-watch f...
chatgpt generates Spark code in minutes
Переглядів 1,3 тис.Рік тому
We will be exploring chatgpt and how it can be used to generate spark code
Azure IOT - End to End Project
Переглядів 16 тис.Рік тому
This video is covering end to end project, from IOT data generation to real time alerting, data analytics and visualisation using Azure IOT Hub, device provisioning service, Azure Stream Analytics, Azure storage, Azure Databricks , Azure Event Hub , Azure functions and Azure Logic Apps. It is full hands on Project which will give you enough confidence on how to implement end to end data pipelin...
Azure IOT End to End Project : Understanding Customer Situation
Переглядів 1,5 тис.2 роки тому
In this video we will understand the customer problem statement for our Azure IOT end to end project.
Azure IOT End to End Project (Part 1)
Переглядів 3,8 тис.2 роки тому
Azure IOT End to End Project (Part 1)
Fraud Analytics using Azure Synapse and Power BI: End to End Project
Переглядів 15 тис.2 роки тому
Fraud Analytics using Azure Synapse and Power BI: End to End Project
Snowflake : Time Travel and Fail Safe
Переглядів 2,2 тис.2 роки тому
Snowflake : Time Travel and Fail Safe
Apache ORC :Master Class (Everything you need to know about ORC)
Переглядів 5 тис.2 роки тому
Apache ORC :Master Class (Everything you need to know about ORC)
Azure : Data Factory and DataBricks End to End Project
Переглядів 146 тис.2 роки тому
Azure : Data Factory and DataBricks End to End Project
Avro file format : Schema Evolution Support , Read and Write Avro files using Spark.
Переглядів 5 тис.3 роки тому
Avro file format : Schema Evolution Support , Read and Write Avro files using Spark.
Git Master Class : Git internals and commands explained in most simplified way
Переглядів 9273 роки тому
Git Master Class : Git internals and commands explained in most simplified way
Spark End to End Project : Sentiment analysis Twitter : Kafka and Spark Structured Streaming
Переглядів 26 тис.3 роки тому
Spark End to End Project : Sentiment analysis Twitter : Kafka and Spark Structured Streaming
Apache Kafka : Leader and follower partitions and ISR
Переглядів 1,2 тис.3 роки тому
Apache Kafka : Leader and follower partitions and ISR
Spark Structured Streaming : Aggregations ,Watermark and Joins Simplified
Переглядів 3,8 тис.3 роки тому
Spark Structured Streaming : Aggregations ,Watermark and Joins Simplified
Spark Structured Streaming : Input sources and Triggers
Переглядів 1 тис.3 роки тому
Spark Structured Streaming : Input sources and Triggers
(27) Spark Structured Streaming : Master Class
Переглядів 1,2 тис.3 роки тому
(27) Spark Structured Streaming : Master Class
(26) Spark Streaming : Stateful operations Hands-on
Переглядів 6733 роки тому
(26) Spark Streaming : Stateful operations Hands-on
(25) Spark Streaming : Stateless Vs Stateful operations Explained
Переглядів 9363 роки тому
(25) Spark Streaming : Stateless Vs Stateful operations Explained
Also I encourage for the New Viewers! Don't Involve in this project! As there no proper information or Instances to complete it.
I Tried to do the project! But failed in Uploading the Machine Learning ONXX Files! The repository where you kept in description doesn't even have proper files! If anyone need Synapse SQL code i have updated in my repsoitory! Check that! github.com/MithunDataPro/Fraud-Analytics-using-Azure-Synapse-and-Power-BI-End-to-End-Project.git
wish I could see it....
Great
Can you please provide me your github link, where this code is uploaded. It's urgent
Hello sir, can't we use spark streaming directly to fetch the data from twitter? What are the benefits of using kafka over spark streaming here?
U don't have GitHub
Amazing class, thank you.
Big project and learned lot of things thank you
please expecting next video sir please make one
Hello
Wow, very detailed. Loved the explanation! Thank you!
Please provide us the notes
Worst Explaination ever... If you want to explain hurry burry Why to make videos..... Datafactory side not even explain anything properly ( get metadata schema parts ) ..... Prepare how to explain..before making video
where is first 4 videos?
where is the part-2 video uploaded sir?
Where is the full video, or implementation part?
The explanation is very good and clear but as you did not provide the databricks notebook it is not going to help viewers because we learn and understand better through practical.
Hi sir Hope you are doing well I am an enthusiastic fresher data engineer. I want to create a data engineering project by taking a one month free subscription on Azure Cloud and show that project on my resume. If my one month free subscription on Azure Cloud expires and the resources get exhausted, will my data engineering project disappear or I will not be able to see it? Can I still show my data engineering project on my resume and the company can see it even after my one month free subscription on Azure Cloud expires? Thank you so much
Hi A doubt in the lecture at or around 44:40, from hash table from small dataframe, there are records like 1) A 1-50, 2) B 51-100 3) C 101-500 , Here A,B,C are hash values ? in that case customer_id is unique ryt, then how come buckets are generated? each hash should have separate bucket ryt? and also one more question, once the probe table is joined to Hash table, to get the bucket id what happens after that? how the customer data gets joined? because hash table does not contain the customer information ryt? even in the case of broadcast hash join concept, you mentioned broadcast is happening for hash tables (52:10) . but when i check other resources they mentioned , the smaller table itself is broadcasted to all the worker nodes where the other dataset resides. please confirm this one too?
Plz remove bg music
bro can you please provide the onenote?
Nice video 😊
Great video brother!! Kudos!!
👎👎👎
Just informing you none of your viewers can work this project because the SQL ingestion script is missing. I tried typing all of it in but there's still some missing
Please share the documents
shame there are Audio issues and unable to grasp a considerable amount of the video.
Apologies Samantha. I might do a voiceover soon.
Happy new Year 2024
Hi , very detailed vedio . Any chance would you share this onenote book please?
Excellent details...thanks so much
aazing project but i was unable to do it. you provided the dataset, but i tried recreating the script you used to create the table. i had to quit because i couldnt see all of it.
Amazing sir!! please keep doing
Great work bro ! One thing that I missed is how to setup with multiple brokers
big project, man ! congratz !
Can you please share your slide decks or notes files?
Poorly explained. Dont go by views😊
great content ! 😊
Thanks for the great content. Can you please create a video on Spark Performance Tuning using Spark UI with Hands on, so that it would be easy to visualize things internally.
nice video full and clear explanation ,thank you
how ratings.csv looks like? Its not shown in the video
can u plz upload full video
Hey can u please share these notes
Hi, Can u share these notes?
Kindly share the gitHub link for the code.
Great Work Bro
this video is enough for data engineering coding part?
Great video ❤ Can you please share the notes as well as that would be so helpful
couple of things i have noticed : 1) you have filtered only COMPLETE orders only and trying to use wide transformation where there is no scope of shuffle 2) No of jobs depends on no of actions No of tasks depends on no of partitions of data