Thats true, and I didn't actually want to suggest using the cheapest server available as a singular node either.
I just wanted to put the given numbers into the scale of the cheapest baremetal servers available on the market, to illustrate that this is not an amount of data you'd have to consider for scaling. Realistically speaking, you'd want to have more then 2 nodes so you don't have to worry about rebooting/updates etc
Well, you probably want at least 2 for redundancy, but active/passive setup on 2 nodes can cover great many use cases if you're not running some memory hogging Rails blob that needs hundreds of MB just to render a site.
Hey, I like this idea! Actually, I only have a single node running the control plane. And, to answer the original question, it felt like overkill to dedicate an entire NUC to running just the control plane. So, I decided to run two k8s nodes on each NUC. I guess there's potentially a bit of a performance hit due to the extra overhead of running multiple kubelets, but it hopefully won't matter too much in practice.
On the contrary, scaling and parallelism are much easier on a multicore machine (possibly with a NIC queue per socket) because most nontrivial workloads are ultimately communication bound. Single load balancers can handle most real sites' traffic and direct them to different cores, and for those concerned about (for example) DDOS attempts, services like Cloudflare will allow you to combine IP anycast with a reverse proxy to avoid this problem far from your machine.
For a large percentage of use cases, the only reason to have multiple machines at all is for fault tolerance, and for most of the others, it's just not being able to fit all your data in memory (increasingly rare). That said, modern networking has gotten good enough that in a really high-end datacenter you can sometimes still get increased performance out of the network (using RDMA, etc.), but you have to really work hard for it (usually bypassing the OS networking stack).
Recommended is 3 servers, 2 CPU cores, 2 GB of RAM.
However, single server configurations are perfectly acceptable for personal clusters.
I would strongly recommend taking a cluster backup when you get it up and running though, since single cluster configurations do not deal well with unexpected power failures.
I have no idea about the OP's system, but I can see some advantages to breaking components up across a network, at least for vertical scaling. First is that server prices do not scale linearly: four 10-core machines will generally be cheaper than a single 40-core machine. Second is that some programs can hog memory/CPU time; forcing programs to run on their own machine can ease some of that contention.
These gains can seem rather small, but can make a big difference at large scales. If you don't mind the drop in latency it's a perfectly acceptable architecture.
When you approach the limits of what your kernel can handle, then it may be time to split your workload across boxes or to carve smaller boxes out of your metal (and probably directly attaching NICs to the VMs to the host OS doesn't have to deal with them). Making your workload horizontally scalable is always a sound engineering choice.
But...
Splitting a horizontally scalable workload across a dozen virtual servers that are barely larger than the smallest laptop you can get from Best Buy, you are just creating self-inflicted pain. Chances are the smallest box you can get from Dell can comfortably host your whole application.
The fact remains the odds of you needing to support more than 10K simultaneous connections are vanishingly small.
If you're eventually going to hit the upper limit of a single machine, have you really gained that much? Eventually you'll have to figure out a way to coordinate between processes, because some will be running on other nodes.
Hmm, good to know. And to update another assumption-- my guess is that this would be a pretty expensive configuration, and you may be better off picking a lower TDP server that can still get the job done, yes?
Also, do you actually have room for 40 nodes in a rack? Betwee TOR switches, storage, and miscellaneous other, it seems unlikely.
Personally, I run 2 clusters of 5 nodes each, in geographically disparate zones.
There are some shared infrastructure dependencies that aren't redundant, but generally speaking the system can survive loss of multiple nodes and/or an entire site.
I'm not really sure what you mean. The number of nodes you need depends on the capacity you need and your replication factor.
Most C* deployments are in the 9-15 node range. You could safely deploy with 6 nodes and RF3 if 6 nodes gives you enough capacity in terms of both disk space and IOPS.
This is standard practice in some domains. Reserve 1 or 2 cores for admin and possibly for the interrupt daemon, and isolate/affinitize app processes to the remaining cores -- ideally giving more resources to your "hot" threads, and better resources (the core "closest" to your NIC) to your network thread, etc.
I think this is the right approach, and I really admire the work you do at ScyllaDB. For something truly critical, you really do want to have multiple nodes available (at least 2, and probably 3 is better). However, you really should want to have backup copies in multiple datacenters, not just the one.
Today, if I were running something that absolutely needed to be up 24/7, I would run a 2x2 or 2x3 configuration with async replication between primary and backup sites.
reply