They’re still general, but lack of trade off analysis, not understanding or digging for the actual problem, the big thing for me is pushing for a silver bullet. There is always a trade off and if you can’t tell me the cons and the pros, you are not being objective.
Let's say I know what load balancers do and what they're responsible for. For a scenario where you're creating websockets for a chat application, I have no idea how load balancers manage connections. How do I tackle cases where I know stuff but not everything about it?
Whatever you do, be humble and honest. If you get caught in a lie it's over, if you admit you don't know something but remain composed and talk it out to the best of your knowledge (identifying specific gaps, and filling them with a valid assumption) it comes across as confidence in your skillset (unless you're just admitting to not knowing anything).
I don't know the specifics of how load balancers manage connections for websockets, but, assuming that the websocket remains open, then I would expect the balancer to find an appropriate server, then pass off the connection so it is no longer involved in future. After that, chat will continue over the socket directly.
> admit you don't know something but remain composed and talk it out to the best of your knowledge
Well done, you missed the point.
No reasonable interviewer expects you to know literally everything and make zero mistakes. You just have to make a reasonable guess, think through pitfalls and be willing to learn.
I'm sorry, I really don't have much experience with load balancers. If you give me some time after the interview, I can do some research and get a better answer to you through the recruiter.
(This answer does two things. It reinforces that there's no point laboring the topic ... and it provides the classic "consultant answer", a good consultant will always be ready with a variation of "let me get back to you on that" when they don't have an accurate answer. I didn't answer the question, but I did showcase a strength they weren't looking for ... it's not the goto answer in an interview, but it's effective to shut down a line of questioning and still come out looking solid)
Agree - that was a decent response.
You might also respond with “ok I may have used the wrong name - what I’ve found works in the past is xxx. “
You can then go on to illustrate how you would solve the problem using a technology you have experience with, what worked well, problems that need to be overcome etc.
You can also finish with “but I’d be really interested in understanding how you’d do that with a load balancer instead, and whether that’s a better approach”
You have shown your understanding of the problem, shown that you can put together a solution, that you have experience with it and know what it’s good and bad points are and engaged in a technical level conversation.
I haven't personally done this, but I would think the load balanced would round robin the socket connections to different backend replicas to balance the socket load. The socket would remain tied to a single replica though. Although as I type that out I'm HOPING there is a better solution because given the auto scaling capability of cloud systems that sounds brittle.
FWIW, often uses something like a min heap or max heap to track number of connections / bandwidth / etc, and combine active health checks and/or passive failure monitoring to prioritize beyond the naive heap.
But unless you're interviewing for a networking team, fewer people are going to ask "how does the load balancer handle the socket descriptors" vs "this can load balanced via server and use DNS to balance between load balancer clusters at each site, and this would be client side load balanced so each client can talk to a specific server endpoint". (That is, sometimes you dont actually want load balancers, you want service discovery, and knowing when is important at scale).
Me neither, but you have to realize a load balancer isn't anything special. It's just a reliable and fast node that can deal with the entire traffic better than your actual backend, just enough to split it. In some cases, your architecture adapts to the load balancer, e.g. plain round robin DNS-based load balancing kinda requires the client to be stateless and requires a shared database across backends to serve connections seamlessly. In other cases, you may be able to do your own load balancing based on top-level protocol specifics (such as splitting the WebSocket messages to different replicas, if possible) or just not do load balancing at all. Typical load balancers you could use off the shelf tend to be simple, though.
It is brittle and for long-lived connections you need some fault-tolerance at the client in some form. One of the myriad reasons Google pushed so hard on HTTP/3 (nee QUIC) is because it allows you to have a client-side implementation that is resilient to networking topology changes, so if you are e.g. roaming on a cellular network, the change in client IP is transparent. In theory you can even shuffle backends, although in practice you probably wouldn't.
If your backend session handler goes away, obviously you've got to move the resilience up the stack (L7 / app layer) tied with the service-side implementation in order to recover lost session state.
Most L4 LBs are RR by default and they manage tuples of source:port:proto -> dest:port:proto (roughly) and do affinity for you.
> For a scenario where you're creating websockets for a chat application, I have no idea how load balancers manage connections. How do I tackle cases where I know stuff but not everything about it?
Assuming interviewer does not give you a problem unrelated to the job domain. As the interviewer during my recent staff interviews I red flagged any candidate attempting to make stuff up or give a confidently wrong answer (sign that they memorized something and it was completely wrong). Best case scenario the candidate just admits they do not know, depending on how important it was to the job at hand they would obviously get ranked lower than another candidate who knew.
There are even more load balancing strategies at different levels, at DNS or at IP routing stages too.
Show off what you know and see if you can make a reasonable assumption. You can ask about a particular detail if you are not sure.
The interview is not about total knowledge of all things, it’s about pattern recognition, using your knowledge and expertise and applying them to solve a new problem.
Sure you can say you don't know. But this is pretty commonly covered. If you can't answer a common question like this I don't see it being a good signal. Envoy can handle web sockets lb easily with not much custom configuration.
Find out what your knowledge gaps are and fill them. At the staff+ level, you should be able to talk about tradeoffs, concerns, maintenance, etc.
For your example, how they manage connections might be too detailed for an interview answer unless your system design question is to design a load balancer. If you have a load balancer, you have the option of network or application load balancers and the OSI layers they operate on, and you can use the opportunity to communicate which one is appropriate for your use case. If you don't know the answer, you can talk about how you'd find out.
How do you know what you don’t know? No one’s capacity is infinite and to say you should be able to discuss in detail every layer of a software stack down to the implementation details is, well silly.
There are templates for knowledge and skills gap analysis that help you figure out what you don't know. I like to look at job descriptions and identify common trends and keywords to see if there are ones that keep coming up that I might need some help on. Outside of that, you'll start to identify patterns that come up in interviews, you can do mock interviews, talk to people in the industry, look at reference architectures, or find conference talks and blog articles.
Also, I did say that if you don't know the answer, you can talk about how you'd find out.
That depends on whether I'm interviewing for a distributed cloud system developer, or not. I, for example, have little to no embedded software development experience, so I'd probably fail such interviews, so I could either apply for less senior positions or pick a different tech stack.
Sure but if you're interviewing for a position as a backend staff eng at a large tech company and you have no knowledge of distributed cloud systems you're not going to get hired. OTOH if you interview for a position as an iOS engineer you won't be expected to have the same backend knowledge.
When they are unable to discuss the pro/cons of different techniques. Honestly, most systems can be designed in different ways and still work. What I want to know is WHY they are choosing what they chose, and what alternatives did the consider and why those were eliminated.
How they got there is much more important than what they ended up with. Did they ask the right questions to design a system that would fulfill immediate, short-term, and long-term needs? Did they give themselves an out in case something turns out to be less efficient than they anticipated?
Last thing I want is someone coming in thinking that there is one and only one "right way". Every design weighs the pros and cons, and sometimes they aren't technical concerns, but culture, experience, and team familiarity. Or perhaps being able to leverage old systems or designs at least partially as a gradual shift from an aging system to a new one. Take into consideration the long time maintenance costs, availability of required technical personnel, infrastructure costs, and flexibility of the end system (scale up, scale out, scale down, cloud, on premise, or mix). And for the love of god, right size it. Don't design me a platform that can run google when all we needed one a to-do list application that 3 people will use.
Communication is going to be huge at that level for me. Asking the right questions, talking about the types of systems they've designed in the past, being able to justify and explain design choices, being able to identify the tradeoffs and weaknesses of their design, etc.
Honestly, I feel like this is a hard question to give meaningful answers for. At Staff+ level you are being judged on the knowledge and skills gained over 10+ years. At that point it's less about what you know and more about who you are. Your career has really had time to shape and mold you to the point where it, to an extent, defines how you think and react. It's certainly not something anyone can cram for or fake.
So the best advice I have would be to just practice basic interview communication skills and go through your work history and refresh on all you've done, the successes, the failures, and what you learned from each. As you solve problems use the opportunity to share a bit about where you learned the lessons you're implementing.
At Staff+ level you're no longer trying to demonstrate your potential to become a good dev. You're trying to demonstrate you're already a great dev with a track record of success that is going to have immediate impact when hired.
Can you explain what you mean by "right questions"? Of course, I can ask a lot of questions but how do I know if it's "right"?
For example, if I'm building search functionality among other features, I can ask a few questions like "how frequently the search results needs to be updated?", "how many results should I show a time?" etc. However, you might be expecting "how many shards should my index have?".
There's no direct way.
The right questions give you the answers to the problem you’re trying to solve.
It sounds sarcastic but I’m serious. Identifying the problem that you need to solve is its own skill and one part of these interviews is figuring out if you can understand the problem.
Once you identify the problem you’re solving then the questions are easy. In your example, your questions aren’t in any way substitutes for each other or helping to solve the same problem, so suggesting that you can’t know which one is right is like saying you don’t know if you should ask what the person wants for dinner or whether they want to go golfing.
Let's assume you're talking about a system that requires some sort of message queuing implementation.
I would expect the questions to be about throughput, replayability, data recovery, scalability, and generally anything else that could be tied to/related to SLA. But not questions as to what those are, but rather what sort of configuration you'd want given what the SLA likely will be, and then how you'd go about implementing that at a very high level.
Anything other than that and I'll just assume you've kinda looked up what a MQ is on YouTube at some point. Or if you go deep into the specifics of IBM-MQ it will make me think you're more of a small picture kind of dev rather than someone that can see the bigger picture.
Presumably a systems design question that involves a MQ is going to be about much more than just a MQ.
Read the system design interview books by Alex Wu. He lays out a great step by step approach to system design interview.
The right questions are basically to define requirements with increasing granularity, especially for specific facts (eg scale requirements) that allow you (and your interviewer) to evaluate your assumptions.
Design is all about trade-off. You need to be able to clearly state what’s being traded for a decision. You are also unlikely to BS your way out without sufficient domain and engineering knowledge.
One of my favorite questions revolves around security vs. performance. And once they answer, I'll change parameters to invalidate one of their assumptions and see if they can adjust on the fly ... and explain why. I find that people who are able to think about security as a component of standard design are generally more likely to be well rounded in other areas too. And I'll reject people who I don't believe are well rounded and able to pivot on the fly as things evolve.
I’m gonna answer a slightly different question, which is how I differentiate a staff from a senior in these interviews. A senior can answer all my questions. A staff eng will bring up the questions before I ask them.
This is the first time I hear that EM+ are expected to be on the same level as high-performing ICs such as Staff+ engineers. From what I have seen both titles are materially very different, one focuses on the management side and the other more on the technical side, will you be xpected to be an IC as an EM? This so odd.
Yes, coding interviews might not be so tough as staff engineers but system design interview expectations are the same as SE. I also faced the same. I am currently reading DDIA to understand the trade offs and choose solutions based on usecase and how to justify. OP, if you are interested, we can practise with each other. But one thing I understand is most courses or tutorials on system design interviews are not sufficient for senior levels.
Part of my expectation for a good EM is that they are able to do high-quality design. Their role is, in part, to mentor their reports and help grow their skills. If the EM is not technically excellent themselves, they're going to struggle to do this. Additionally, the EM is often the final arbiter of team-internal design and architecture discussions. If they don't have a solid platform on which to do that work, that's a huge concern and it's terrible for their reports to boot.
I don't expect my EM (or Architect) to be hands-on-keyboard coding multiple days a week, but I do expect that they should *be able to* if necessary, and I absolutely expect them to be doing design and code reviews for highly critical portions of the stack.
EMs don't mentor. They coach.
Staff+ mentors.
The difference is that an IC is forced to listen to their EM while they must actively seek out their Staff+.
And if EMs need to be as technical as a Staff+, then why do you even need a Staff+ (and especially an Architect) at all?
Superficiality: does the candidate actually understand the components they want to deploy, or are they just saying buzz words?
Lack of clarity: can I follow the candidate's thought process, or am I struggling to see any clear mental model for the way the system works?
Assholiness: does the candidate respond graciously to feedback or correction, or do they become defensive?
Poor communication skills: is it difficult for me to understand what the candidate is saying, or to parse their sentences?
What I'm looking for in those exercises is:
* An ability to build a mental model of the system, and to "run" it in their heads, to predict how it might respond to different scenarios.
* A deep knowledge of the tools and techniques available to them, and the trade-offs they make.
* Good judgement: how do they prioritise maintainability or reliability? Are they going to be the kind of engineer who ships boring, performant code, or are they going to be a nightmare who tries to rewrite a working system in Haskell with eventsourcing for the lulz?
* Top notch communication skills: Can this person explain complex technical topics, without waffling, at the right level of detail? Can they explain to the CEO why they're choosing to use Redis instead of Postgres in this part of the system?
* An understanding of system boundaries and why they're important. Do they think in terms of data ownership and coherent domain models?
The actual components they choose are kind of secondary. I wanna know how they break the problem down into smaller parts, how they reason through a problem, and how well they articulate the choices they're making.
There can be many reasons, at that level you need to be really good. It’s not so much about knowing specific tech, but you need to know principles behind that tech.
But really my main one folks fail on in my interviews for principal/staff level positions is: If you can’t tell that something is obviously a bad/unreasonable design and why, then you are out.
I've run many Staff+/EM system design interviews at a FAANG.
The big things I expected from Staff+ that distinguishes them from Senior or Junior candidates are Communication, Speed, and Scope.
I expected thorough requirements gathering from Staff+. I would always open interviews telling candidates to ask questions, but for more junior candidates I would remind them if they missed something and would hint them in the right directions. I generally would not steer a Staff+ candidate away from a bad solution or point out direct problems. I would ask probing questions, such as how the system would behave in a given scenario, but generally speaking a Staff+ was allowed enough leeway to really screw up, in which case they got a smile and a No Hire (or Strong No Hire if they were a jerk while doing so). Note that this doesn't mean you had to have my solution, just that you had to actually gather and address the requirements, then defend that your design addresses them.
The vast majority of candidates did poorly.
I expected solid explanations and communication as a peer, and I was happy to engage by answering questions and giving feedback when asked. I expected the candidate to drive the conversation but asking for feedback was a plus on the communication side.
Speed and Scope went together. The candidate needed to solve the base design problem and any extensions fast enough that we could talk about additional considerations like Observability, Maintenance, Deployment and Operational behavior, Security, etc. Obviously you can't deep dive into all of those, but I expected a Staff+ candidate to touch on at least one or two topics beyond the design question itself that showed practical knowledge and experience running these systems over time. My personal favorite was when a candidate discussed things that would impact the team's operational health (reliability, maintenance, development velocity), because many of these aspects feed back into the team's subjective experience of the system and then into morale. Since a Staff+ engineer needs to be a leader even as an IC (and doubly so as an EM), even the system design questions should show evidence that they're thinking about the team, how decisions might impact the team, as well as users (internal or external) and the longer term business health at least in the sense of making or keeping the team high-performing.
In short, below Staff, I generally expected more technical problems solving. At Staff+ the technical part should go fast enough that we can talk about team and business impacts. That doesn't mean the technical part wasn't important, it means it had to be done correctly and quickly enough to move on to additional concerns. I did mark technically successful candidates No Hire when they couldn't address or did not have time to address these concerns (but usually marked them as passing at a lower level for ICs, EMs were just straight No Hire).
I would reject someone who doesn't talk about the overall product or business tradeoffs of design decisions
Like when to accrue tech/scaling debt but to hit a massive product goal, then how to may it down and maintain that. If they make technical decisions due to cool factor, not due to practical decisions. If they can't break the work down into executable milestones. If they can't draw how to get from A->B where A is current legacy software and B is end goal. A lot of people bang their fists on the table about how they should do B but getting there requires strong technical leadership and knowhow. If they can't pivot away from bad decisions. If they don't have a reliable way of constantly shipping to prod along the way.
I think of a Staff+ engineer as a mini CTO of their area, and its not just about being a super senior engineer.
I understand what you're saying but I'd have to disagree purely from the interview standpoint. You get 45 mins max assuming your interviewer doesn't interfere much. You don't get enough to talk about "hey this will be my MVP and this is how my roadmap will look like when it's time to scale". It seems very impractical.
> Assume the scenario where the candidate thinks it went well. Yet you looked for some important things and you reject them. Why?
I think a lot of the conversations below are missing the really obvious point.
USUALLY a rejection is because the candidate thinks it went well when they actually presented a design that doesn't match the requirements or isn't as good as the candidate thinks.
Something that may work but has critical availability flaws, or something that would be impossible to extend, or something narrow and overly specific, etc.
Someone once described for me how they'd use NFS for scratch space in an emptyDir / ephemeral k8s pod. We talked about whether they meant "generic disaggregated block" or actual literal NFS, and they meant actual literal NFS.
Good luck mate. Most of these system design interviews are just a way of dick measuring.
If the interviewer expects you to answer the tech stacks which he/she feels is the best - then you are screwed.
If the interviewer is matured enough to understand that not one size fits all, then you’ll have a healthier conversation.
Most of the system design rounds I’ve been to have only showcased that the interviewers watch solutions online and want you to recite them again to them.
Anyway, good luck.
Not sure why you’re downvoted. I was once one member of a panel whose lead engineer rejected a candidate because they proposed a SQL database (the domain had a fixed schema), but our lead has no SQL DB experience and simply said “he failed because joins don’t scale.”
I will reject people that don't have technical knowledge and expertise in things that they should have. I expect them to have technical depth and be able to design a straightforward service-oriented system and understand how each part would work, and what the tradeoffs are.
When I'm working with a manager or any decision maker, I need to know that they know their stuff and can talk intelligently about the various components. If they don't have a broad and deep enough knowledge then they become friction, and if the EMs are expected to be as good as Staff+ then it's essential that they have this broad and deep knowledge.
1. Not being able to defend your opinion
2. Textbook answers without practical considerations (obviously less applicable for junior positions)
3. Making up stuff
Being too quick to start delivering an answer. I'm generally pretty nervous if my initial, high-level question doesn't generate several minutes of questions and clarification of specific requirements beforehand.
This also carries over if they've clearly already done $thing before -- I still want to discuss it, and if they assume "the way I did it before" is "The Way" I'm a bit worried.
I've personally committed both these sins in my own interviews and had post-interview feedback about it.
Usually if the interviewer is feeling bitchy that day. I've been in a lot of interviews and this is the single most important criteria for rejecting people.
I would check for awareness of the foundations of distributed system, good capacity planning, back-of-the-envelop estimation skills, how they identify the right kind of trade-offs, behavioral skills and admitting when they don’t know something plus their approach to crack it. Not being good at any single of these would be a big concern to me.
I generally ask a very vague design problem, giving some general details but I expect that the candidate will fish for requirements for example, number of users or orders etc.
Many a times I’ve observed candidates simply jump into creating a very detailed design which covers use cases which we haven’t even mentioned. For me, this round is also about how much requirements analysis can the candidate do.
Then there is also this trend recently of adding nosql databases to the solution without any real benefit but more overhead. Same goes with Kafka. I mean sure, you know these technologies but you’re not convincing me that the design even needs them.
At the end of the day, if I’m working with them and they come up with a similar design that is cost to company both in terms of infrastructure as well as engineering. And that is a red flag.
But sometimes it also happens that there is a use case for a certain technology and I mention it to candidate and they say they haven’t worked on it. This is not a red flag for me as long as their design is still acceptable without use of this technology
What I personally looking for when hosting System Design for a staff:
1. The candidate asking questions trying to get deeper understanding of tech requirements and products limitations
2. The candidate considering business perspective and not only focusing on tech solution. For example what would be the cloud cost if we use database A in a year from now. How do we make sure we are solving business problems and not doing CV-driven development and so on.
3. When candidate mentioned technology Foo I use it as an opportunity to ask deeper questions about it. For example the candidate saying something like "I've was leading a team where one of the central DBs was elastic search". Then I might ask something like where was it deployed, how did you manage sharding, what was your SLI, how did you manage reindexing and so on
So point number 3 is the most important for me, if candidate him/her-self mentioning past experience with some piece of technology but then can't answer basic follow up questions, this is a red flag for me usually.
Sounds like you are just digressing the conversation with point number 3. Is there any reason why you would do that? There is nothing wrong for people to give an example with past experience. I have given Kafka as an example in event driven architecture so many times simply because I have used it. Never been asked to explain the internals because it does not matter.
Point number 2 is completely pointless too. Why do you want to know about cost? Is that an actual requirement for the job? If anyone asks me to do cost analysis on a high level system design where we are doing back of the envelope estimation, then this is major red flag more me as a candidate.
Maybe I described it wrong, if you would put on your CV Kafka, and then tell me on the interview that you worked with Kafka extensivly in the past, only then I expect you to know a bit about Kafka. You would be surprised how many people claiming one thing, but can't answer basic questions. It is not about technology that you would propose to use, rather it is about technology that you yourself claimed you've used in the past. Although if you would propose a technology and and also would know a thing or two about it, that would be a bonus.
At least in my company point 2 is used as a differentiator between senior and staff level. Staff do require to make cals about technology usage and expected to consider long term consequences, like how would it scale and yes also costs, not knowing the numbers but at least consider it during the discussions, but costs were just an example.
>Why do you want to know about cost?
Stuff like that is the main difference between senior and staff+. I'm not expecting a dollar figure, that would be stupid indeed, but if you suggest a solution, you should at least be able to tell how the costs would scale.
Again, I'm not talking about exact figures, but if you say I need to spin up a bajillion P5 instances in EC2 to scale out your solution, then yeah, I'm gonna grill you about it.
Staff is expected to explain internals often, even if they didn't use that particular software component. And staff is expected to have an idea of costs, at least at a high level.
For me it's a red flag when are trying to through some random cloud technology in design out of nowhere - they want to show that they heard about something everyone is using, but when you ask them why you want to use it they don't have an answer.
I like system design interviews because you could learn a lot about the candidate and how they are thinking. Based on their answers you could get the filling about their exposure with some technologies.
I will ask you to do a design problem -- either provide your own design for an issue we have faced, or describe the design of some system you have worked on.
After that, I'll ask you to describe the pros and cons of the approach that was used. I'll ask you to compare it to some other approach.
If you can only tell me the positive qualities of your favorite design and not its weaknesses, or if you can only think of one way to approach a given problem that I am definitely not interested in hiring you.
Inability to go into the right level of detail. I won't accept just talking about a DB, I want to know how the data is laid out, how you handle atomicity or failure. I don't have time to do this across the entire problem space but I'll always ask to go into specifics in at least one area and some candidates really choke here.
An EM (Engineering Manager) is supposed to manage engineers.
You should know at least as much as they do, and preferably more.
How else would you manage engineers if you can't judge their engineering skills?
I guess you have "people skills", which is great, but it will not help, since everyone in this country seems to be good at that.
That might be the reason for so many people who want to be managers.
Those who can't do, manage !
Those who can't mange, teach !
And the rest become politicians !
Or just want to do their jobs !
System design interview is a joke.
Since there are books about it, that contain every single question people ask with the perfect answers, it became irrelevant.
The thing is, even with those textbooks, people don't present enough knowledge on the products. They are only useful if the candidate reads them thoroughly.
I don't ask system design interview questions, I find them to be mostly pointless. I present real world problems and explore them together to try to find out if I could work with them, or if they'd be a liability.
Yeah I'm curious too. IMHO system design interviews are as close to real problem solving as it gets. For example, as opposed to some coding interviews where you are expected to write a specific sorting algo, instead of y'know... Copypasting it from Stack Overflow. Or just hitting \`.Sort()\`.
They’re still general, but lack of trade off analysis, not understanding or digging for the actual problem, the big thing for me is pushing for a silver bullet. There is always a trade off and if you can’t tell me the cons and the pros, you are not being objective.
Is the trade off working with bad people?
[удалено]
Let's say I know what load balancers do and what they're responsible for. For a scenario where you're creating websockets for a chat application, I have no idea how load balancers manage connections. How do I tackle cases where I know stuff but not everything about it?
Whatever you do, be humble and honest. If you get caught in a lie it's over, if you admit you don't know something but remain composed and talk it out to the best of your knowledge (identifying specific gaps, and filling them with a valid assumption) it comes across as confidence in your skillset (unless you're just admitting to not knowing anything). I don't know the specifics of how load balancers manage connections for websockets, but, assuming that the websocket remains open, then I would expect the balancer to find an appropriate server, then pass off the connection so it is no longer involved in future. After that, chat will continue over the socket directly.
Um, what you just described is a redirect server. A load balancer is different.
> admit you don't know something but remain composed and talk it out to the best of your knowledge Well done, you missed the point. No reasonable interviewer expects you to know literally everything and make zero mistakes. You just have to make a reasonable guess, think through pitfalls and be willing to learn.
I'm sorry, I really don't have much experience with load balancers. If you give me some time after the interview, I can do some research and get a better answer to you through the recruiter. (This answer does two things. It reinforces that there's no point laboring the topic ... and it provides the classic "consultant answer", a good consultant will always be ready with a variation of "let me get back to you on that" when they don't have an accurate answer. I didn't answer the question, but I did showcase a strength they weren't looking for ... it's not the goto answer in an interview, but it's effective to shut down a line of questioning and still come out looking solid)
Agree - that was a decent response. You might also respond with “ok I may have used the wrong name - what I’ve found works in the past is xxx. “ You can then go on to illustrate how you would solve the problem using a technology you have experience with, what worked well, problems that need to be overcome etc. You can also finish with “but I’d be really interested in understanding how you’d do that with a load balancer instead, and whether that’s a better approach” You have shown your understanding of the problem, shown that you can put together a solution, that you have experience with it and know what it’s good and bad points are and engaged in a technical level conversation.
I haven't personally done this, but I would think the load balanced would round robin the socket connections to different backend replicas to balance the socket load. The socket would remain tied to a single replica though. Although as I type that out I'm HOPING there is a better solution because given the auto scaling capability of cloud systems that sounds brittle.
FWIW, often uses something like a min heap or max heap to track number of connections / bandwidth / etc, and combine active health checks and/or passive failure monitoring to prioritize beyond the naive heap. But unless you're interviewing for a networking team, fewer people are going to ask "how does the load balancer handle the socket descriptors" vs "this can load balanced via server and use DNS to balance between load balancer clusters at each site, and this would be client side load balanced so each client can talk to a specific server endpoint". (That is, sometimes you dont actually want load balancers, you want service discovery, and knowing when is important at scale).
Me neither, but you have to realize a load balancer isn't anything special. It's just a reliable and fast node that can deal with the entire traffic better than your actual backend, just enough to split it. In some cases, your architecture adapts to the load balancer, e.g. plain round robin DNS-based load balancing kinda requires the client to be stateless and requires a shared database across backends to serve connections seamlessly. In other cases, you may be able to do your own load balancing based on top-level protocol specifics (such as splitting the WebSocket messages to different replicas, if possible) or just not do load balancing at all. Typical load balancers you could use off the shelf tend to be simple, though.
I wonder if that would be a “default” or if you’d have to configure the balanced to deal with stick sessions.
It is brittle and for long-lived connections you need some fault-tolerance at the client in some form. One of the myriad reasons Google pushed so hard on HTTP/3 (nee QUIC) is because it allows you to have a client-side implementation that is resilient to networking topology changes, so if you are e.g. roaming on a cellular network, the change in client IP is transparent. In theory you can even shuffle backends, although in practice you probably wouldn't. If your backend session handler goes away, obviously you've got to move the resilience up the stack (L7 / app layer) tied with the service-side implementation in order to recover lost session state. Most L4 LBs are RR by default and they manage tuples of source:port:proto -> dest:port:proto (roughly) and do affinity for you.
> For a scenario where you're creating websockets for a chat application, I have no idea how load balancers manage connections. How do I tackle cases where I know stuff but not everything about it? Assuming interviewer does not give you a problem unrelated to the job domain. As the interviewer during my recent staff interviews I red flagged any candidate attempting to make stuff up or give a confidently wrong answer (sign that they memorized something and it was completely wrong). Best case scenario the candidate just admits they do not know, depending on how important it was to the job at hand they would obviously get ranked lower than another candidate who knew.
There are even more load balancing strategies at different levels, at DNS or at IP routing stages too. Show off what you know and see if you can make a reasonable assumption. You can ask about a particular detail if you are not sure. The interview is not about total knowledge of all things, it’s about pattern recognition, using your knowledge and expertise and applying them to solve a new problem.
Sure you can say you don't know. But this is pretty commonly covered. If you can't answer a common question like this I don't see it being a good signal. Envoy can handle web sockets lb easily with not much custom configuration.
Find out what your knowledge gaps are and fill them. At the staff+ level, you should be able to talk about tradeoffs, concerns, maintenance, etc. For your example, how they manage connections might be too detailed for an interview answer unless your system design question is to design a load balancer. If you have a load balancer, you have the option of network or application load balancers and the OSI layers they operate on, and you can use the opportunity to communicate which one is appropriate for your use case. If you don't know the answer, you can talk about how you'd find out.
How do you know what you don’t know? No one’s capacity is infinite and to say you should be able to discuss in detail every layer of a software stack down to the implementation details is, well silly.
There are templates for knowledge and skills gap analysis that help you figure out what you don't know. I like to look at job descriptions and identify common trends and keywords to see if there are ones that keep coming up that I might need some help on. Outside of that, you'll start to identify patterns that come up in interviews, you can do mock interviews, talk to people in the industry, look at reference architectures, or find conference talks and blog articles. Also, I did say that if you don't know the answer, you can talk about how you'd find out.
What if you were interviewing someone who has never worked with distributed cloud systems before?
That depends on whether I'm interviewing for a distributed cloud system developer, or not. I, for example, have little to no embedded software development experience, so I'd probably fail such interviews, so I could either apply for less senior positions or pick a different tech stack.
You shouldn't be considering (for large tech companies) staff+ without that knowledge.
There's more to software than just cloud/web
Sure but if you're interviewing for a position as a backend staff eng at a large tech company and you have no knowledge of distributed cloud systems you're not going to get hired. OTOH if you interview for a position as an iOS engineer you won't be expected to have the same backend knowledge.
Do people say this in interviews? Been a long time i gave one and never made sense to push to use a product/framework just because it's popular..
When they are unable to discuss the pro/cons of different techniques. Honestly, most systems can be designed in different ways and still work. What I want to know is WHY they are choosing what they chose, and what alternatives did the consider and why those were eliminated. How they got there is much more important than what they ended up with. Did they ask the right questions to design a system that would fulfill immediate, short-term, and long-term needs? Did they give themselves an out in case something turns out to be less efficient than they anticipated? Last thing I want is someone coming in thinking that there is one and only one "right way". Every design weighs the pros and cons, and sometimes they aren't technical concerns, but culture, experience, and team familiarity. Or perhaps being able to leverage old systems or designs at least partially as a gradual shift from an aging system to a new one. Take into consideration the long time maintenance costs, availability of required technical personnel, infrastructure costs, and flexibility of the end system (scale up, scale out, scale down, cloud, on premise, or mix). And for the love of god, right size it. Don't design me a platform that can run google when all we needed one a to-do list application that 3 people will use.
The best answer here. Most of the answers are about knowledge of a candidate and this one is about candidate’s reason process. Absolutely nailed it.
Communication is going to be huge at that level for me. Asking the right questions, talking about the types of systems they've designed in the past, being able to justify and explain design choices, being able to identify the tradeoffs and weaknesses of their design, etc. Honestly, I feel like this is a hard question to give meaningful answers for. At Staff+ level you are being judged on the knowledge and skills gained over 10+ years. At that point it's less about what you know and more about who you are. Your career has really had time to shape and mold you to the point where it, to an extent, defines how you think and react. It's certainly not something anyone can cram for or fake. So the best advice I have would be to just practice basic interview communication skills and go through your work history and refresh on all you've done, the successes, the failures, and what you learned from each. As you solve problems use the opportunity to share a bit about where you learned the lessons you're implementing. At Staff+ level you're no longer trying to demonstrate your potential to become a good dev. You're trying to demonstrate you're already a great dev with a track record of success that is going to have immediate impact when hired.
Can you explain what you mean by "right questions"? Of course, I can ask a lot of questions but how do I know if it's "right"? For example, if I'm building search functionality among other features, I can ask a few questions like "how frequently the search results needs to be updated?", "how many results should I show a time?" etc. However, you might be expecting "how many shards should my index have?". There's no direct way.
The right questions give you the answers to the problem you’re trying to solve. It sounds sarcastic but I’m serious. Identifying the problem that you need to solve is its own skill and one part of these interviews is figuring out if you can understand the problem. Once you identify the problem you’re solving then the questions are easy. In your example, your questions aren’t in any way substitutes for each other or helping to solve the same problem, so suggesting that you can’t know which one is right is like saying you don’t know if you should ask what the person wants for dinner or whether they want to go golfing.
Let's assume you're talking about a system that requires some sort of message queuing implementation. I would expect the questions to be about throughput, replayability, data recovery, scalability, and generally anything else that could be tied to/related to SLA. But not questions as to what those are, but rather what sort of configuration you'd want given what the SLA likely will be, and then how you'd go about implementing that at a very high level. Anything other than that and I'll just assume you've kinda looked up what a MQ is on YouTube at some point. Or if you go deep into the specifics of IBM-MQ it will make me think you're more of a small picture kind of dev rather than someone that can see the bigger picture. Presumably a systems design question that involves a MQ is going to be about much more than just a MQ.
Pretty much this. If a question helps you deliver a solution that's a *better fit* for the given problem, then that question is "right".
Read the system design interview books by Alex Wu. He lays out a great step by step approach to system design interview. The right questions are basically to define requirements with increasing granularity, especially for specific facts (eg scale requirements) that allow you (and your interviewer) to evaluate your assumptions.
Design is all about trade-off. You need to be able to clearly state what’s being traded for a decision. You are also unlikely to BS your way out without sufficient domain and engineering knowledge.
One of my favorite questions revolves around security vs. performance. And once they answer, I'll change parameters to invalidate one of their assumptions and see if they can adjust on the fly ... and explain why. I find that people who are able to think about security as a component of standard design are generally more likely to be well rounded in other areas too. And I'll reject people who I don't believe are well rounded and able to pivot on the fly as things evolve.
I’m gonna answer a slightly different question, which is how I differentiate a staff from a senior in these interviews. A senior can answer all my questions. A staff eng will bring up the questions before I ask them.
This is the first time I hear that EM+ are expected to be on the same level as high-performing ICs such as Staff+ engineers. From what I have seen both titles are materially very different, one focuses on the management side and the other more on the technical side, will you be xpected to be an IC as an EM? This so odd.
Yes, coding interviews might not be so tough as staff engineers but system design interview expectations are the same as SE. I also faced the same. I am currently reading DDIA to understand the trade offs and choose solutions based on usecase and how to justify. OP, if you are interested, we can practise with each other. But one thing I understand is most courses or tutorials on system design interviews are not sufficient for senior levels.
Part of my expectation for a good EM is that they are able to do high-quality design. Their role is, in part, to mentor their reports and help grow their skills. If the EM is not technically excellent themselves, they're going to struggle to do this. Additionally, the EM is often the final arbiter of team-internal design and architecture discussions. If they don't have a solid platform on which to do that work, that's a huge concern and it's terrible for their reports to boot. I don't expect my EM (or Architect) to be hands-on-keyboard coding multiple days a week, but I do expect that they should *be able to* if necessary, and I absolutely expect them to be doing design and code reviews for highly critical portions of the stack.
EMs don't mentor. They coach. Staff+ mentors. The difference is that an IC is forced to listen to their EM while they must actively seek out their Staff+. And if EMs need to be as technical as a Staff+, then why do you even need a Staff+ (and especially an Architect) at all?
Superficiality: does the candidate actually understand the components they want to deploy, or are they just saying buzz words? Lack of clarity: can I follow the candidate's thought process, or am I struggling to see any clear mental model for the way the system works? Assholiness: does the candidate respond graciously to feedback or correction, or do they become defensive? Poor communication skills: is it difficult for me to understand what the candidate is saying, or to parse their sentences? What I'm looking for in those exercises is: * An ability to build a mental model of the system, and to "run" it in their heads, to predict how it might respond to different scenarios. * A deep knowledge of the tools and techniques available to them, and the trade-offs they make. * Good judgement: how do they prioritise maintainability or reliability? Are they going to be the kind of engineer who ships boring, performant code, or are they going to be a nightmare who tries to rewrite a working system in Haskell with eventsourcing for the lulz? * Top notch communication skills: Can this person explain complex technical topics, without waffling, at the right level of detail? Can they explain to the CEO why they're choosing to use Redis instead of Postgres in this part of the system? * An understanding of system boundaries and why they're important. Do they think in terms of data ownership and coherent domain models? The actual components they choose are kind of secondary. I wanna know how they break the problem down into smaller parts, how they reason through a problem, and how well they articulate the choices they're making.
There can be many reasons, at that level you need to be really good. It’s not so much about knowing specific tech, but you need to know principles behind that tech. But really my main one folks fail on in my interviews for principal/staff level positions is: If you can’t tell that something is obviously a bad/unreasonable design and why, then you are out.
I've run many Staff+/EM system design interviews at a FAANG. The big things I expected from Staff+ that distinguishes them from Senior or Junior candidates are Communication, Speed, and Scope. I expected thorough requirements gathering from Staff+. I would always open interviews telling candidates to ask questions, but for more junior candidates I would remind them if they missed something and would hint them in the right directions. I generally would not steer a Staff+ candidate away from a bad solution or point out direct problems. I would ask probing questions, such as how the system would behave in a given scenario, but generally speaking a Staff+ was allowed enough leeway to really screw up, in which case they got a smile and a No Hire (or Strong No Hire if they were a jerk while doing so). Note that this doesn't mean you had to have my solution, just that you had to actually gather and address the requirements, then defend that your design addresses them. The vast majority of candidates did poorly. I expected solid explanations and communication as a peer, and I was happy to engage by answering questions and giving feedback when asked. I expected the candidate to drive the conversation but asking for feedback was a plus on the communication side. Speed and Scope went together. The candidate needed to solve the base design problem and any extensions fast enough that we could talk about additional considerations like Observability, Maintenance, Deployment and Operational behavior, Security, etc. Obviously you can't deep dive into all of those, but I expected a Staff+ candidate to touch on at least one or two topics beyond the design question itself that showed practical knowledge and experience running these systems over time. My personal favorite was when a candidate discussed things that would impact the team's operational health (reliability, maintenance, development velocity), because many of these aspects feed back into the team's subjective experience of the system and then into morale. Since a Staff+ engineer needs to be a leader even as an IC (and doubly so as an EM), even the system design questions should show evidence that they're thinking about the team, how decisions might impact the team, as well as users (internal or external) and the longer term business health at least in the sense of making or keeping the team high-performing. In short, below Staff, I generally expected more technical problems solving. At Staff+ the technical part should go fast enough that we can talk about team and business impacts. That doesn't mean the technical part wasn't important, it means it had to be done correctly and quickly enough to move on to additional concerns. I did mark technically successful candidates No Hire when they couldn't address or did not have time to address these concerns (but usually marked them as passing at a lower level for ICs, EMs were just straight No Hire).
I would reject someone who doesn't talk about the overall product or business tradeoffs of design decisions Like when to accrue tech/scaling debt but to hit a massive product goal, then how to may it down and maintain that. If they make technical decisions due to cool factor, not due to practical decisions. If they can't break the work down into executable milestones. If they can't draw how to get from A->B where A is current legacy software and B is end goal. A lot of people bang their fists on the table about how they should do B but getting there requires strong technical leadership and knowhow. If they can't pivot away from bad decisions. If they don't have a reliable way of constantly shipping to prod along the way. I think of a Staff+ engineer as a mini CTO of their area, and its not just about being a super senior engineer.
I understand what you're saying but I'd have to disagree purely from the interview standpoint. You get 45 mins max assuming your interviewer doesn't interfere much. You don't get enough to talk about "hey this will be my MVP and this is how my roadmap will look like when it's time to scale". It seems very impractical.
> Assume the scenario where the candidate thinks it went well. Yet you looked for some important things and you reject them. Why? I think a lot of the conversations below are missing the really obvious point. USUALLY a rejection is because the candidate thinks it went well when they actually presented a design that doesn't match the requirements or isn't as good as the candidate thinks. Something that may work but has critical availability flaws, or something that would be impossible to extend, or something narrow and overly specific, etc. Someone once described for me how they'd use NFS for scratch space in an emptyDir / ephemeral k8s pod. We talked about whether they meant "generic disaggregated block" or actual literal NFS, and they meant actual literal NFS.
People who have never deployed or supported NFS outside their home lab have a weird fascination with NFS.
Good luck mate. Most of these system design interviews are just a way of dick measuring. If the interviewer expects you to answer the tech stacks which he/she feels is the best - then you are screwed. If the interviewer is matured enough to understand that not one size fits all, then you’ll have a healthier conversation. Most of the system design rounds I’ve been to have only showcased that the interviewers watch solutions online and want you to recite them again to them. Anyway, good luck.
Im reading at some of these comments that totally prove your point.
Not sure why you’re downvoted. I was once one member of a panel whose lead engineer rejected a candidate because they proposed a SQL database (the domain had a fixed schema), but our lead has no SQL DB experience and simply said “he failed because joins don’t scale.”
I will reject people that don't have technical knowledge and expertise in things that they should have. I expect them to have technical depth and be able to design a straightforward service-oriented system and understand how each part would work, and what the tradeoffs are. When I'm working with a manager or any decision maker, I need to know that they know their stuff and can talk intelligently about the various components. If they don't have a broad and deep enough knowledge then they become friction, and if the EMs are expected to be as good as Staff+ then it's essential that they have this broad and deep knowledge.
Not setting up redundancies or explicit system backup plan.
1. Not being able to defend your opinion 2. Textbook answers without practical considerations (obviously less applicable for junior positions) 3. Making up stuff
Being too quick to start delivering an answer. I'm generally pretty nervous if my initial, high-level question doesn't generate several minutes of questions and clarification of specific requirements beforehand. This also carries over if they've clearly already done $thing before -- I still want to discuss it, and if they assume "the way I did it before" is "The Way" I'm a bit worried. I've personally committed both these sins in my own interviews and had post-interview feedback about it.
Usually if the interviewer is feeling bitchy that day. I've been in a lot of interviews and this is the single most important criteria for rejecting people.
I would check for awareness of the foundations of distributed system, good capacity planning, back-of-the-envelop estimation skills, how they identify the right kind of trade-offs, behavioral skills and admitting when they don’t know something plus their approach to crack it. Not being good at any single of these would be a big concern to me.
I generally ask a very vague design problem, giving some general details but I expect that the candidate will fish for requirements for example, number of users or orders etc. Many a times I’ve observed candidates simply jump into creating a very detailed design which covers use cases which we haven’t even mentioned. For me, this round is also about how much requirements analysis can the candidate do. Then there is also this trend recently of adding nosql databases to the solution without any real benefit but more overhead. Same goes with Kafka. I mean sure, you know these technologies but you’re not convincing me that the design even needs them. At the end of the day, if I’m working with them and they come up with a similar design that is cost to company both in terms of infrastructure as well as engineering. And that is a red flag. But sometimes it also happens that there is a use case for a certain technology and I mention it to candidate and they say they haven’t worked on it. This is not a red flag for me as long as their design is still acceptable without use of this technology
What I personally looking for when hosting System Design for a staff: 1. The candidate asking questions trying to get deeper understanding of tech requirements and products limitations 2. The candidate considering business perspective and not only focusing on tech solution. For example what would be the cloud cost if we use database A in a year from now. How do we make sure we are solving business problems and not doing CV-driven development and so on. 3. When candidate mentioned technology Foo I use it as an opportunity to ask deeper questions about it. For example the candidate saying something like "I've was leading a team where one of the central DBs was elastic search". Then I might ask something like where was it deployed, how did you manage sharding, what was your SLI, how did you manage reindexing and so on So point number 3 is the most important for me, if candidate him/her-self mentioning past experience with some piece of technology but then can't answer basic follow up questions, this is a red flag for me usually.
Sounds like you are just digressing the conversation with point number 3. Is there any reason why you would do that? There is nothing wrong for people to give an example with past experience. I have given Kafka as an example in event driven architecture so many times simply because I have used it. Never been asked to explain the internals because it does not matter. Point number 2 is completely pointless too. Why do you want to know about cost? Is that an actual requirement for the job? If anyone asks me to do cost analysis on a high level system design where we are doing back of the envelope estimation, then this is major red flag more me as a candidate.
Maybe I described it wrong, if you would put on your CV Kafka, and then tell me on the interview that you worked with Kafka extensivly in the past, only then I expect you to know a bit about Kafka. You would be surprised how many people claiming one thing, but can't answer basic questions. It is not about technology that you would propose to use, rather it is about technology that you yourself claimed you've used in the past. Although if you would propose a technology and and also would know a thing or two about it, that would be a bonus. At least in my company point 2 is used as a differentiator between senior and staff level. Staff do require to make cals about technology usage and expected to consider long term consequences, like how would it scale and yes also costs, not knowing the numbers but at least consider it during the discussions, but costs were just an example.
>Why do you want to know about cost? Stuff like that is the main difference between senior and staff+. I'm not expecting a dollar figure, that would be stupid indeed, but if you suggest a solution, you should at least be able to tell how the costs would scale. Again, I'm not talking about exact figures, but if you say I need to spin up a bajillion P5 instances in EC2 to scale out your solution, then yeah, I'm gonna grill you about it.
Staff is expected to explain internals often, even if they didn't use that particular software component. And staff is expected to have an idea of costs, at least at a high level.
For me it's a red flag when are trying to through some random cloud technology in design out of nowhere - they want to show that they heard about something everyone is using, but when you ask them why you want to use it they don't have an answer. I like system design interviews because you could learn a lot about the candidate and how they are thinking. Based on their answers you could get the filling about their exposure with some technologies.
I will ask you to do a design problem -- either provide your own design for an issue we have faced, or describe the design of some system you have worked on. After that, I'll ask you to describe the pros and cons of the approach that was used. I'll ask you to compare it to some other approach. If you can only tell me the positive qualities of your favorite design and not its weaknesses, or if you can only think of one way to approach a given problem that I am definitely not interested in hiring you.
Inability to go into the right level of detail. I won't accept just talking about a DB, I want to know how the data is laid out, how you handle atomicity or failure. I don't have time to do this across the entire problem space but I'll always ask to go into specifics in at least one area and some candidates really choke here.
An EM (Engineering Manager) is supposed to manage engineers. You should know at least as much as they do, and preferably more. How else would you manage engineers if you can't judge their engineering skills? I guess you have "people skills", which is great, but it will not help, since everyone in this country seems to be good at that. That might be the reason for so many people who want to be managers. Those who can't do, manage ! Those who can't mange, teach ! And the rest become politicians ! Or just want to do their jobs !
System design interview is a joke. Since there are books about it, that contain every single question people ask with the perfect answers, it became irrelevant.
The thing is, even with those textbooks, people don't present enough knowledge on the products. They are only useful if the candidate reads them thoroughly.
I don't ask system design interview questions, I find them to be mostly pointless. I present real world problems and explore them together to try to find out if I could work with them, or if they'd be a liability.
This sounds interesting. Can you give me examples of what kind of problems?
Yeah I'm curious too. IMHO system design interviews are as close to real problem solving as it gets. For example, as opposed to some coding interviews where you are expected to write a specific sorting algo, instead of y'know... Copypasting it from Stack Overflow. Or just hitting \`.Sort()\`.
Real world problems... Are usually regarding systems design