Data Engineering For Everyone
Data Engineering For Everyone
  • 60
  • 315 927
MS Fabric End to End Project on Fraud Analytics
Introduction : 00:00
Data Sources : 03:09
Workspace creation : 04:15
ADLS Gen2 Shortcut : 05:29
AWS S3 Shortcut : 08:20
Onelake explorer : 11:35
Dataflow Gen2 : 13:40
Load to Delta tables : 17:33
Notebook for Machine Learning : 19:10
MLFlow : 23:45
SQL Visual Query : 32:46
Semantic Data Modelling : 34:26
MLFlow Result : 36:12
Dashboarding : 37:19
Recap : 38:50
Переглядів: 287

Відео

Data Engineering for Everyone: Top Choice in Bard AI's YouTube Plugin!
Переглядів 2347 місяців тому
🚀 Welcome to Data Engineering for Everyone, the go-to educational resource for aspiring and experienced data engineers alike! Our channel has recently been recognized as a top result in Bard AI's UA-cam plugin, and we're thrilled to share our knowledge and passion with an even wider audience.
Demystifying Microsoft Fabric: A Collaborative Dive with Microsoft GBB
Переглядів 2039 місяців тому
In this enlightening session, Microsoft GBB Ian Clarke takes us through the captivating world of Microsoft Fabric. This video is tailored to break down the technical jargons and unfurl the essence of MS Fabric in the simplest language possible. Whether you're a Data Engineer or a Data Scientist, this session has something in store for you. Ian Clarke elaborates on how MS Fabric fosters seamless...
Exploring the Power of Generative AI: An Introduction to Cutting-Edge Technology
Переглядів 349Рік тому
In this video, we will be exploring the exciting field of generative AI and its potential applications. We will cover the basics of how generative AI works and its underlying technology, including neural networks and deep learning. While we will touch upon the basics of GPT (Generative Pretrained Transformer) language models and their capabilities, we will be diving deeper into the details of G...
From Data Ingestion to Model Deployment: A Comprehensive Azure ML and Databricks End to End Project
Переглядів 6 тис.Рік тому
From Data Ingestion to Model Deployment: A Comprehensive Azure ML and Databricks End to End Project
Expert Insights on Spark Memory Management: Answering the Top Interview Questions
Переглядів 915Рік тому
Are you preparing for a Spark-related job interview and want to showcase your knowledge of memory management in Spark? Look no further! In this video, I will walk you through the most commonly asked Spark memory management interview questions and provide in-depth answers to help you succeed. From understanding the role of executor memory and storage memory, to managing memory spills, this video...
Top Spark Performance Tuning Interview Questions and Answers
Переглядів 1,9 тис.Рік тому
Get ready for your Spark performance tuning interview with this comprehensive video. We cover the most commonly asked interview questions on optimizing Spark performance and provide clear, concise answers to help you ace the interview. Whether you're an experienced Spark developer or just starting out, this video is a must-watch for anyone looking to improve their Spark performance tuning skills
Master Spark Partitioning and Bucketing: Top Interview Questions Answered
Переглядів 1,8 тис.Рік тому
#crackSparkInterviews #AllAboutSpark In this video, we will discuss the most frequently asked interview questions on Spark partitioning and bucketing. Spark is a popular big data processing framework, and a deep understanding of partitioning and bucketing is crucial for optimizing the performance of Spark jobs. We will cover topics such as the difference between partitioning and bucketing, the ...
Most Asked interview question in Apache Spark ‘Joins’
Переглядів 1,9 тис.Рік тому
#sparkinterviews Learn the ins and outs of Apache Spark Join operations in this comprehensive interview-style tutorial . Discover the different types of Spark joins, including inner join, outer join, left join, right join and more. Get hands-on experience with real-life examples of joining large datasets using Spark. Whether you're a data engineer or data scientist, this video is a must-watch f...
chatgpt generates Spark code in minutes
Переглядів 1,3 тис.Рік тому
We will be exploring chatgpt and how it can be used to generate spark code
Azure IOT - End to End Project
Переглядів 16 тис.Рік тому
This video is covering end to end project, from IOT data generation to real time alerting, data analytics and visualisation using Azure IOT Hub, device provisioning service, Azure Stream Analytics, Azure storage, Azure Databricks , Azure Event Hub , Azure functions and Azure Logic Apps. It is full hands on Project which will give you enough confidence on how to implement end to end data pipelin...
Azure IOT End to End Project : Understanding Customer Situation
Переглядів 1,5 тис.2 роки тому
In this video we will understand the customer problem statement for our Azure IOT end to end project.
Azure IOT End to End Project (Part 1)
Переглядів 3,8 тис.2 роки тому
Azure IOT End to End Project (Part 1)
Fraud Analytics using Azure Synapse and Power BI: End to End Project
Переглядів 15 тис.2 роки тому
Fraud Analytics using Azure Synapse and Power BI: End to End Project
Snowflake : Type of tables
Переглядів 1,7 тис.2 роки тому
Snowflake : Type of tables
Snowflake : Time Travel and Fail Safe
Переглядів 2,2 тис.2 роки тому
Snowflake : Time Travel and Fail Safe
Apache ORC :Master Class (Everything you need to know about ORC)
Переглядів 5 тис.2 роки тому
Apache ORC :Master Class (Everything you need to know about ORC)
Snowflake : Master Class (Part 1)
Переглядів 6 тис.2 роки тому
Snowflake : Master Class (Part 1)
Azure : Data Factory and DataBricks End to End Project
Переглядів 146 тис.2 роки тому
Azure : Data Factory and DataBricks End to End Project
Avro file format : Schema Evolution Support , Read and Write Avro files using Spark.
Переглядів 5 тис.3 роки тому
Avro file format : Schema Evolution Support , Read and Write Avro files using Spark.
Git Master Class : Git internals and commands explained in most simplified way
Переглядів 9273 роки тому
Git Master Class : Git internals and commands explained in most simplified way
Spark End to End Project : Sentiment analysis Twitter : Kafka and Spark Structured Streaming
Переглядів 26 тис.3 роки тому
Spark End to End Project : Sentiment analysis Twitter : Kafka and Spark Structured Streaming
Apache Kafka : Leader and follower partitions and ISR
Переглядів 1,2 тис.3 роки тому
Apache Kafka : Leader and follower partitions and ISR
Apache Kafka : Master class 1/2
Переглядів 1,9 тис.3 роки тому
Apache Kafka : Master class 1/2
Spark Structured Streaming : Aggregations ,Watermark and Joins Simplified
Переглядів 3,8 тис.3 роки тому
Spark Structured Streaming : Aggregations ,Watermark and Joins Simplified
Spark Structured Streaming : Input sources and Triggers
Переглядів 1 тис.3 роки тому
Spark Structured Streaming : Input sources and Triggers
Path to become a data engineer
Переглядів 1,6 тис.3 роки тому
Path to become a data engineer
(27) Spark Structured Streaming : Master Class
Переглядів 1,2 тис.3 роки тому
(27) Spark Structured Streaming : Master Class
(26) Spark Streaming : Stateful operations Hands-on
Переглядів 6733 роки тому
(26) Spark Streaming : Stateful operations Hands-on
(25) Spark Streaming : Stateless Vs Stateful operations Explained
Переглядів 9363 роки тому
(25) Spark Streaming : Stateless Vs Stateful operations Explained

КОМЕНТАРІ

  • @user-om5ur3in5e
    @user-om5ur3in5e 12 днів тому

    Also I encourage for the New Viewers! Don't Involve in this project! As there no proper information or Instances to complete it.

  • @user-om5ur3in5e
    @user-om5ur3in5e 12 днів тому

    I Tried to do the project! But failed in Uploading the Machine Learning ONXX Files! The repository where you kept in description doesn't even have proper files! If anyone need Synapse SQL code i have updated in my repsoitory! Check that! github.com/MithunDataPro/Fraud-Analytics-using-Azure-Synapse-and-Power-BI-End-to-End-Project.git

  • @whosestone
    @whosestone 19 днів тому

    wish I could see it....

  • @sainulabid7854
    @sainulabid7854 23 дні тому

    Great

  • @bhanukhandelwal8033
    @bhanukhandelwal8033 Місяць тому

    Can you please provide me your github link, where this code is uploaded. It's urgent

  • @bhanukhandelwal8033
    @bhanukhandelwal8033 Місяць тому

    Hello sir, can't we use spark streaming directly to fetch the data from twitter? What are the benefits of using kafka over spark streaming here?

  • @susantsahoo6393
    @susantsahoo6393 Місяць тому

    U don't have GitHub

  • @sergioantonin4784
    @sergioantonin4784 Місяць тому

    Amazing class, thank you.

  • @bhushanthoke858
    @bhushanthoke858 2 місяці тому

    Big project and learned lot of things thank you

  • @Shashikumar_shanmuka
    @Shashikumar_shanmuka 2 місяці тому

    please expecting next video sir please make one

  • @rashidpatel1955
    @rashidpatel1955 2 місяці тому

    Hello

  • @shubhampawade2933
    @shubhampawade2933 3 місяці тому

    Wow, very detailed. Loved the explanation! Thank you!

  • @sachinkv-oy7ix
    @sachinkv-oy7ix 3 місяці тому

    Please provide us the notes

  • @ramswaroop1560
    @ramswaroop1560 3 місяці тому

    Worst Explaination ever... If you want to explain hurry burry Why to make videos..... Datafactory side not even explain anything properly ( get metadata schema parts ) ..... Prepare how to explain..before making video

  • @ChetanSharma-oy4ge
    @ChetanSharma-oy4ge 4 місяці тому

    where is first 4 videos?

  • @shriramravichandran2956
    @shriramravichandran2956 4 місяці тому

    where is the part-2 video uploaded sir?

  • @abusalem411
    @abusalem411 4 місяці тому

    Where is the full video, or implementation part?

  • @SanjayKumar-rw2gj
    @SanjayKumar-rw2gj 4 місяці тому

    The explanation is very good and clear but as you did not provide the databricks notebook it is not going to help viewers because we learn and understand better through practical.

  • @HemantKumar-su1qt
    @HemantKumar-su1qt 4 місяці тому

    Hi sir Hope you are doing well I am an enthusiastic fresher data engineer. I want to create a data engineering project by taking a one month free subscription on Azure Cloud and show that project on my resume. If my one month free subscription on Azure Cloud expires and the resources get exhausted, will my data engineering project disappear or I will not be able to see it? Can I still show my data engineering project on my resume and the company can see it even after my one month free subscription on Azure Cloud expires? Thank you so much

  • @naveena2226
    @naveena2226 5 місяців тому

    Hi A doubt in the lecture at or around 44:40, from hash table from small dataframe, there are records like 1) A 1-50, 2) B 51-100 3) C 101-500 , Here A,B,C are hash values ? in that case customer_id is unique ryt, then how come buckets are generated? each hash should have separate bucket ryt? and also one more question, once the probe table is joined to Hash table, to get the bucket id what happens after that? how the customer data gets joined? because hash table does not contain the customer information ryt? even in the case of broadcast hash join concept, you mentioned broadcast is happening for hash tables (52:10) . but when i check other resources they mentioned , the smaller table itself is broadcasted to all the worker nodes where the other dataset resides. please confirm this one too?

  • @sowmya6942
    @sowmya6942 5 місяців тому

    Plz remove bg music

  • @harsheetchordiya1773
    @harsheetchordiya1773 5 місяців тому

    bro can you please provide the onenote?

  • @AbhishekYadav-ms7ce
    @AbhishekYadav-ms7ce 6 місяців тому

    Nice video 😊

  • @nagrotte
    @nagrotte 6 місяців тому

    Great video brother!! Kudos!!

  • @VR-Astro-Health-Etc
    @VR-Astro-Health-Etc 6 місяців тому

    👎👎👎

  • @zaiydmala6970
    @zaiydmala6970 7 місяців тому

    Just informing you none of your viewers can work this project because the SQL ingestion script is missing. I tried typing all of it in but there's still some missing

  • @praneethjeevan2173
    @praneethjeevan2173 7 місяців тому

    Please share the documents

  • @samanthamccarthy9765
    @samanthamccarthy9765 7 місяців тому

    shame there are Audio issues and unable to grasp a considerable amount of the video.

  • @manishkumar73
    @manishkumar73 7 місяців тому

    Happy new Year 2024

  • @worldofvishruth9017
    @worldofvishruth9017 8 місяців тому

    Hi , very detailed vedio . Any chance would you share this onenote book please?

  • @prasadpg6189
    @prasadpg6189 8 місяців тому

    Excellent details...thanks so much

  • @matthewmark7224
    @matthewmark7224 8 місяців тому

    aazing project but i was unable to do it. you provided the dataset, but i tried recreating the script you used to create the table. i had to quit because i couldnt see all of it.

  • @PoojaM22
    @PoojaM22 8 місяців тому

    Amazing sir!! please keep doing

  • @raheelmasood8656
    @raheelmasood8656 9 місяців тому

    Great work bro ! One thing that I missed is how to setup with multiple brokers

  • @felipesantos1789
    @felipesantos1789 9 місяців тому

    big project, man ! congratz !

  • @komalilyas-gh9tu
    @komalilyas-gh9tu 9 місяців тому

    Can you please share your slide decks or notes files?

  • @avinashkumarrai8925
    @avinashkumarrai8925 9 місяців тому

    Poorly explained. Dont go by views😊

  • @abhishekmazumdar2980
    @abhishekmazumdar2980 9 місяців тому

    great content ! 😊

  • @user-vn7mu6ew9o
    @user-vn7mu6ew9o 9 місяців тому

    Thanks for the great content. Can you please create a video on Spark Performance Tuning using Spark UI with Hands on, so that it would be easy to visualize things internally.

  • @user-mx9kk8np6u
    @user-mx9kk8np6u 10 місяців тому

    nice video full and clear explanation ,thank you

  • @user-de6qh6uj3r
    @user-de6qh6uj3r 11 місяців тому

    how ratings.csv looks like? Its not shown in the video

  • @akshaydubey4901
    @akshaydubey4901 11 місяців тому

    can u plz upload full video

  • @AbhishekVerma-hx7rc
    @AbhishekVerma-hx7rc 11 місяців тому

    Hey can u please share these notes

  • @AbhishekVerma-hx7rc
    @AbhishekVerma-hx7rc 11 місяців тому

    Hi, Can u share these notes?

  • @ranjansrivastava9256
    @ranjansrivastava9256 11 місяців тому

    Kindly share the gitHub link for the code.

  • @mohamedjendoubi4450
    @mohamedjendoubi4450 11 місяців тому

    Great Work Bro

  • @avinash7003
    @avinash7003 Рік тому

    this video is enough for data engineering coding part?

  • @varunvijaywargi5497
    @varunvijaywargi5497 Рік тому

    Great video ❤ Can you please share the notes as well as that would be so helpful

  • @suneelsunkari4354
    @suneelsunkari4354 Рік тому

    couple of things i have noticed : 1) you have filtered only COMPLETE orders only and trying to use wide transformation where there is no scope of shuffle 2) No of jobs depends on no of actions No of tasks depends on no of partitions of data