How Discord Stores TRILLIONS of Messages

If Discord would only create a good UX. Having may servers is a nightmare to go through..

How does BYTEBYTEGO create these diagram animation does anyone know ?

Can you make me a pro at syatem design ?? I just love this subject...

Hats off for the on call team during these processes 😅

Waouh!

Thank you for this informative content. I saw your post on LinkedIn and came here to know more about the Rust Dat Service. Can you link any article that this service is based off of? Id live to know its working as a Distributed systems engineer and connoisseur

Nice explainer. Can you also please link the OG blogpost in the description?

how to run scylla db in the cloud??

Thanks for this, very helpful.

Can't they use AWS dynamoDB?

I’m just curious, what tools are you using to make this beautiful diagram and fancy effects 😮❤

excellent conten.

wow, can i ask what after effect template is used for presentation? I would like to present my work in school.

Joy != rust

great content!!

👀👌

"discord found themselves in a bit of a pickle"....good one :)

That's just awesome sharing. Thanks!

Show!!!!

One thing missing here is how they setup the monitoring system to have such analysis from the running system, which is also an important part. Would like to hear more about this.

How is their solution different from RAID10? It is essentially raid1 array mirroring a raid0 set. Am i missing something?

Scylla is basically a C++ version of Cassandra. This story is one more proof of java's inferiority. Why are people still using Java ?? 🤷🏽‍♂️

Well it doesn't actually tell HOW it stores trillions of messages (like partitioning, cache organization, hot path issues), just a high level overview of the migration and dbms

Is it ok to move older messages to a separate database periodically? So the main database would not be carrying so much data all the time.

And have dedicated service to read messages from that database with historical messages.

someone please explain more on the RAID 0 setup? What is the safety net under that? Am I missing/not understanding something?

Rust is a superstar !!

Could you make a video on the Data services part which written in Rust ? That will be a quite interesting to many folks !!

I am confused the difference between "Request coalescing" and what your other videos call caching? Since "Request coalescing" is part of the success here I'd like to understand the difference better, thanks!

next month I'll migrating one of my database, it's so painful to planning and flow designing of migration, but this video help a lot.

What tools do you use to create these videos

QUICK QUESTION: I've seen this style of YT video somewhere else too, somewhere in a tech video I remember. Is there a tool you're using to make these kind of videos? Or did you (or someone you hired) edited the video yourself?

Friendly programming noob here: Can someone explain why Garbage Collection-free gives Scylla an advantage over Cassandra? Thanks in advanced.

What pipeline tool did they use for the data migration from Cassandra to Scylla?

my takeaways and a question:

ScyllaDB with C++ under the hood and no GC was a part of it.
Request coalescing is another part.
Selecting and deciding on optimal Google Cloud services i.e Persistent disk is another part.
The two layered RAID setup is another part.
Modifying the Linux kernel to do write on the Persistent disks and reads from the local SSDs was another part(But I'm wondering how they made the changes to the Linux kernel of the Google VMs? anyone help me out here, do they bake the changes to a VM image and deploy the image to the Google VMs?)
And finally for the migration using a data migrator written in memory safe and highly performant Rust was another great decision.

cool，I want to try in my work

I have literally never seen a company be happy with choosing Cassandra. I know of engineering orgs who have an entire pod of SREs dedicated to nothing but keeping the Cassandra alive.

Please make video in realtime application like figma, Google docs, or multiplayer game

What software do you use to create such presentation?

So much effort for messages that nobody can ever find again anyway 😂Seriously, Discord scrollback might as well be write-only it's so unusable. :(

Only 9 days what a huge achievement

The real beauty of ScyllDB lies in Seastar framework which bypasses OS kernel to achieve close to metal speed. Also, Cassandra 5.0 which will release in few months from now won't have this issues and will much higher throughput as it will be able to compile against Java 17 SDK and might also probably run Java 21 which has Generational ZGC garbage collector which gives you sub milliseconds latency. Cassandra also have the advantages of defining aggregate function in Java or JS and run in Casandra DB instance itself rather than fetching all data in application server which is super expensive.

"of course. Migration production data is no joke!". Thanks for the quality contents.

Really good content. Just one point - writes go to both the local SSDs and the persistent disks at the same time. Write-mostly in Linux md means that reads go to the other disks, so in this case the local SSDs.

Amazing topic!!!

thanks for content :)

Amazing. I have read about the features of Scylla but this a definitive proof that is an excellent database.

Thanks for the content. A little introduction was missing what kind of database this is and why it is better, except for C++ under the hood