System design basics: Real-time data processing

Please talk about getting job abroad

Does any message queue support ordering based on some key ?

Hi, I really love your vdos. I have a question. Can you make a vdo which explains about capacity management calculation for vms in a distributed env. Would be very helpful. I mean how do I calculate what should be the server cpu and ram of each server in a distributed env.

Thanks for your time and patience teaching it, all your videos are amazing, I hope that soon increase the number of likes or subscriptions

Great video. It explains concept of distributing systems without going into technical branding.

Do you have any blog expalining these problems and concepts?

excellent work narendra

Thanks, Narendra!
It would be good to hear from you with your excellent style of explanation about all most popular technologies (may be as brief overview) for distributed systems like apache.* stack (cassandra, kafka, etc.), hadoop.* etc., in one video in case to have complete picture of what we can use and what for.


Guys, support me by likes, please :-) Hope Narendra will do it soon)

Great video as usual :)
Can you please do a video on Google Map design

Hi @narendra, What is the logic by which a driver forwards a message to one of the many executors?
Thanks for the informative video as always.

Why do we need a layer before the executors, can't the executors (subscribe) get msg from stream and process it?

You forget to shuffle and sort part, writing directly from executor to db would cause concurrency issue. Spark do shuffle and sort, to accumulate the data in single piece like collect all 200 as count 200->500 [as single or time based which will be multiple] , then write

Sir Make video for IRCTC system

Thanks for explaining the topic in laymen terms..

Very nice explanation Naren Sir

I have a scenerio where i have to analyze data on 50-100gb data as of now. And i also have to apply filter on whole data and need a real time filtered result then which approach i have to use. And data also require some calculation in between.

Narendra, I dont miss yr videos. they are very informative. I have request. can you please make video for system Integration keeping in mind secure integration and data exchange. u may take some different scenarios for integration and talk about solution. I searched a lot but difficult to find good article about integration.

Thank you Narendra :D

@techDummies Can you recommend me courses on Kafka with pyspark?

can the job processers directly grab tasks from the queue?

There are rules to prohibit defining something in terms of itself.
That is, you can’t define stream processing as a stream that is processed by processing a stream, in which a stream is used which in turn is then processed.

PubSub + Cloud Dataflow + Big table + BigQuery

Best place to ace system design!!

You are awsome sir! I like the old school pen and whiteboard explanation

Isn't kafka a pull based system? Meaning , the subscribers need to pull the data rather than a driver pushing data to subscribers

Thanks a lot Narendra .. Love watching your videos .. Your hard work is our short work :P

Please do a video on backpressure and other type of failures in this architecture

Kafka is a PubSub and not a queue.

If I am correct, in Kafka producer need to decide on partition. How can consumer scale if producer is not in our control (we can't control number of partitions from producer)?

nice teaching

Which means this somewhat similar to concerrency

Thank you for the nice informative videos. i have a request to you, if you can make a system design video on Forex Trading platform

That is hilarious. "We cannot collect data on one machine" (why?), then immediately we need some queueing, then immediately let's say "kafka". That is so bad that it made me laugh.

Live use case example and a context surrounding messaging queue, logs etc would have made this video much better.. nice try

Good job but I think it might better if you don’t go with a solution in mind. In the next videos, try to evolve the solution over few minutes to provide reasons for scale. This way the students learn about the tradeoffs.