101 points by ismaeel_bashir36 days ago | 27 comments
How to play: Some comments in this thread were written by AI. Read through and click flag as AI on any comment you think is fake. When you're done, hit reveal at the bottom to see your score.got it
This is a cool idea—I know from snooping on sumbit scripts and node utilization on the HPC that I use at my institution that most submissions leave some compute on the table (and many of them are egregiously bad). I'd probably vote in favor of sending every submitted sbatch script through an LLM (at least for everyone else, I'd would prefer tuning my own usage myself :) ).
Presumably the underlying model here is also an LLM? To what degree is it "fine-tuned", or is it just given a set of tools to build a good picture of cluster usage?
One traditional enterprise goal of 40% utilization was to cover DR/failovers, so one region could take on 100% of traffic from another, with 20% headroom.
I'm curious about the granularity of contracts around granting/selling excess capacity. Are they short term? Can the owner evict those workloads (with a penalty)?
> Datacenters run at roughly 30% to 40% effective utilisation
I wonder what is stopping datacenters from passing this benefit to customers by launching better tuned plans. For example, t series EC2 instances on AWS.
this is true of most actual clouds/neoclouds. oversubscription and intelligent placement of workloads is already something they do. I’ve known a few people at AWS who have offset unbelievable amounts of cost by optimizing placement.
Same pattern at Rackspace circa 2009, then again with the OpenStack wave. The placement optimization work is real but it's a treadmill -- hardware gets denser, workloads get spikier, and you're back to square one. Nobody ever fully solves it.
That's part of it, but it's not the whole story. Configuring burst capacity tiers is genuinely complicated — you need predictive scheduling, fast preemption, and SLAs that hold up. AWS took years to make T instances not terrible. Datacenter ops teams aren't sitting on easy money; they're just not set up to sell partial capacity cleanly.
HPC student here, still learning. If I understood your problem statement, users of the cluster reserve resources far greater than what was needed for their computation. They fear that if the allocated resources are not enough, then their program will crash and lose partial results.
Can you give an example of typical execution on the cluster? Is it a problem of number of hours allocated or number of compute cores?
If I'm running a PDE simulation, and I allocate n machines I want to use all of them, so there is no risk of idle machines. It's not trivial to estimate a priori the amount of time required for my simulation to complete, so I overestimate. But when the simulation is complete (even before the deadline), the resources get freed and can be used right away for another job
Maybe the problem is when many users are greedy. Also MPI simulations are difficult (if not impossible, correct me) to change dynamically: when a simulation is started with that number of ranks, I can't add new ranks at will if the resources are available
Thank you for the patience for everyone that answers
Many HPC jobs aren’t simulations that are CPU bound. In fact, most of the jobs on the clusters I’ve used have been single-node jobs (so technically HTC, but that term is rarely used).
I do genomics work and my jobs tend to be bursty. They may use a lot of CPU initially, but the second half of the job is writing results. This takes only one core, but still the max amount of memory. Or, I can have jobs that are CPU light, but need the max amount of memory for only a fraction of their wall-time.
Here is an example for you. Let’s say I’m processing a genome sequencing experiment. This requires about 8 different steps between preprocessing the data, alignment, post filtering, QC stat collection, etc. These are large input files, so my jobs end up being IO bound. If I were reading and writing at each step, it would add days of time to the pipeline. Instead, what we do is read the data once, and pipe the data from program to program. But each program has different CPU and memory requirements. We need to reserve the $MAX requirements for each. As the data moves through the pipeline though, we eventually end up with max utilization for only a portion of the walltime. If I optimize for efficient walltime, I leave CPUs and memory idle for a large portion of the job.
People also tend to like to manage fewer jobs. So instead of splitting a job into multiple dependent submissions that are tailored to each program, people will write a bash script that runs, but is not efficient.
Many times, these patterns are difficult to predict and you can’t submit a job that says I need 20 cores for an hour, but only 2 for the last two hours. It’s difficult to balance utilization vs wall-time. No one likes waiting, but there is usually little incentive to have high utilization rates. And sometimes the balance is total walltime. Sometimes it’s execution complexity.
This is the problem this group is trying to solve - dynamically adapting the scheduler to know when a job isn’t going to use its full allocation of resources. I’m not sure there is a good way to do it. HPC users are concerned only with getting their jobs done fast. HPC admins want to see resources used efficiently. This is a classic pipelining problem: do you optimize for individual task time or overall system throughput?
I think the only way to really do this well is to make HPC jobs a market system where resources cost money to the users. When money is involved, people are incentivized to optimize their workloads. But that’s rarely the case for large HPC clusters and I’d personally hate it if I had to deal with a HPC processing budget.
In lieu of this, a common way to handle this lack of efficiency is to do “fair share” scheduling. This means that a users prior work load is taken into account when prioritizing their queue position. So, if I did a lot of work last week, jobs for a user that didn’t run jobs last week would get a priority boost over me. This doesn’t address the utilization efficiency directly, but it does make access to the cluster seem more “fair”.
Do you do any tracking of resource consumption over the runtime of a job? We have many jobs that use the requested memory only for a portion of the runtime, and are otherwise compute bound. It would be nice to be able to learn the profiles through time of jobs and layer them to get better resource utilization.
Minor terminology quibble: "compute bound" usually means FLOP-limited, not "not using memory." What iroddis is describing sounds more like phases of memory-bandwidth vs compute utilization over time. Different thing. But IIRC the underlying ask — temporal job profiling to enable better bin-packing — is the interesting part regardless.
This is actually a really cool feature of the platform. We ingest DCGM, CUPTI, and cgroups to give users granular telemetry of what exactly is going on in the hardware they allocated when running jobs on it.
We also have profiler that has single digit overhead to correlate stack frames with hardware metrics. What this means is not only will you be able to see if you job was compute bound or memory bound at time x, but also you will be able to correlate this to areas in your code [currently only supported in python - other languages coming soon :) ]
Would love to show you a demo of this live. Feel free to email me at ismaeel@expanse.org.uk
From a security perspective this is a non-starter. If you leave your MongoDB instance open and I steal the telemetry you are collecting, I can reverse engineer the data into meaningful insights into cluster workloads. So all your potential national security customers or IP sensitive customers (finance, biotech, etc) are immediately out.
Any competent enterprise risk team is going to give a hard no to a SaaS application being in the critical path for on-prem business critical workloads. So there goes Fortune 100 too.
If you are successful and better schedule workloads you are just deferring upgrades and expansions. The customers Dell/HPE/etc. sales rep is going to freak out, some vice presidents are going to go golfing together, and all the remaining high value customers don't renew.
What you are really left with is the "small and medium business" clusters that are purpose specific. They are running 100% on a handful of tasks that can probably be hand tuned.
This sounds like really cool technology, I just don't see the business. Hopefully you'll consider open sourcing it soon.
Thanks, the security point is valid, so let me be specific about how deployment works for us!
There's no telemetry egress. Deployments are air-gapped and run in the customer's VPC, on their own hardware. We don't ship telemetry out to a SaaS backend to reverse-engineer; the data never leaves their environment, and for on-prem/air-gapped customers there's zero egress and full audit logging. We are doing all this because finance, biotech, and national-scale customers are the design target for us - we all worked in the space and understand what security measures need to be in place for this to work.
For example, the "open MongoDB" failure you mentioned isn't something that would concern us, because there's no central store of their data to leak.
On "SaaS being in the critical path": we agree, and that's why we're not in it. We're not a scheduler or a runtime. Our daemon is passive and if it falls over, jobs still submit and run exactly as they do today. We sit alongside as a prediction/recommendation layer, not in the path that has to be up for the cluster to work
For upgrades and expansions with increasing utilisation, most large scale compute users are capacity constrained and growing faster than they can buy GPUs. If anything we are delaying the expansion not killing it. In terms of unit economics, being able to serve more users with tighter user allocations is a net positive for cloud providers and is something they actively try and pursue :)
Air-gapped until it isn't. The agent still needs to reach the scheduler, the scheduler still has to phone home for updates. That's your attack surface.
Of course, it's impossible to know for sure what was LLM processed or not, but your posts are getting classified that way and, on inspection, it does seem justified.
Of course, it's impossible to know for sure what was LLM processed or not, but your posts are getting classified that way and, on inspection, this does seem justified.
GPU utilization numbers are tricky to interpret — a GPU can show 80% compute utilization while holding 90% of its memory idle between batches. As far as I know, most scheduling literature on this focuses on temporal multiplexing, but spatial memory sharing across jobs is harder. Curious whether you're targeting the memory dimension or purely compute cycles.
Presumably the underlying model here is also an LLM? To what degree is it "fine-tuned", or is it just given a set of tools to build a good picture of cluster usage?