A Few of a Senior Software Engineer's Top Priorities

Howdy folks! It’s been a long while since I’ve put pen to proverbially paper and written new content for the site. With winter on the horizon and the long, dark nights approaching, what better a time to get back into writing.

This post encapsulates my own thoughts on some of the topics I consider most important for Senior Software Engineers always be thinking about when developing systems professionally.

Who am I?

These kind of posts always need some form of author qualification - My name is Elliot, and I’ve been a software engineer for over 8 years now.

I started my journey in JPMorgan in the graduate program and worked my way up to Senior Associate (Shortened to Senior Ass by my mortgage provider, but that’s a different story).

During my time there, I worked on their internal cloud portal before shifting to delivering the largest CloudFoundry estate in the world on nsx-t and VMware.

I then shifted over to Curve and helped delivered their 2nd generation payment platform, and now I’m pushing for a staff-level role at CircleCI.

Shifting Priorities Over time

As I’ve progressed through my own career, I’ve noted that my priorities have shifted as I’ve leveled up.

At the very start of my career, the focus was purely on delivery. Getting through the work and focusing on learning the best design patterns, the tools, and the tricks of the trade.

The primary aim was doing everything I can to deliver on the quarterly goals set by the seniors/staff-level engineers and the product owners.

Over the last few years, my priorities have shifted - It’s become less a case of delivery, and more a case of helping to make decisions that became increasingly impactful.

As I’ve gained more and more experience, and gotten more and more wins under my belt, I’ve started to be pulled into more meetings that are more focused on the larger strokes of software engineering.

The Broad Strokes

Over the past few years, the broad strokes that I’ve become increasingly concerned with are, in no particular order:

• Observability
• Maintainability
• Testing
• Resiliency and Scalability
• Security

Observability

Delivering something to production is a fantastic start, but what I am really concerned with is - “how well is it working?”.

When I first started in software, I would deliver a new feature with enough tests to pass the fairly light pull-request review processes I had in my initial teams.

Now, delivery is just the start of a feature’s journey. I need to be able to see how many times it’s being called, the various latencies the consumers of said features can expect, or how many times it is erroring out.

This observability is vital when it comes to incident remediation. When you hit a performance cliff, or a piece of infrastructure fails, you will need to be able to remediate things as soon as possible to avoid monetary, or reputational loss for your company.

Datadog dashboards have become my first port of call whenever trouble is brewing.

Maintainability

Whilst observability is important, I also need to consider the maintainability of systems going forward.

In the past, I’ve defaulted to creating a new widget whenever a widget-sized hole rears itself as a requirement.

Ideally though, each new widget comes with an operational overhead of keeping said widget healthy - all of the points in this list must be replicated, to a degree, over these new widgets.

Architecturally, keeping concepts well-bounded in their own domains makes a lot of sense.

Pragmatically, it can cause teams to be spread thin over 10-20 smaller services (sometimes in multiple languages).

This can be incredibly challenging and, at times, concessions should be made to reduce the overall maintenance budget so that you are not directly impacting the team’s ability to deliver.

Tests, Tests and More Tests

There are multiple different levels of testing in software. From your unit tests, through acceptance tests, regression tests, enhancement tests, and e2e tests.

Regardless of your favourite flavour, the further on in my career I get, the more I value a strong test suite over all the apps I work with.

Note - I’d like to highlight that quite frankly I couldn’t care less about test coverage, but more about the quality of tests in said suite.

The stronger the test suite, the more confident I am making changes to unfamiliar code-bases. Therefore, the faster I can deliver new features without running the risk of causing regressions or incidents.

Resiliency and Scalability

Observability, maintainability and tests are a fantastic grounding, but what happens if our services start to take off, or a high-load upstream service starts to rely on us, we need to be ready.

We need to know the operational limits of our service prior to us reaching an inevitable performance cliff and our service nose-diving into the murky depths below.

Ultimately, before a services goes live, having a matrix of the number of instances of your services and how many requests per second it can handle is incredibly valuable.

Load-testing, with tools such as vegeta, prior to production traffic hitting my services is invaluable and provides us with that warm-fuzzy feeling inside that our services aren’t going to topple over at a light breeze.

Not only that, I also want to ensure that, as my traffic scales, that my auto-scaling solution works as intended.

Note - a well-directed DDOS will put any service into a bit of a pickle. What I want to know is how many requests per second my service currently gets through observability, and how many it can practically handle.

Security

The final, and arguably the most important point of this unordered list - Security.

Successful services will always tend to draw the unwanted ire of some script-kiddie, or wannabe L33t H4x0r.

The more successful the service, the more likely this will inevitably occur sadly.

When developing out new services, security must be thought out at every level. Your goal as an developer should be to make the life of one of these reprobates targeting your service as miserable as fucking possible.

Hackers/scammers/scumbags only continue to exist because of the fact that they can sustain themselves infiltrating services, selling data, or holding people/companies to ransom by demanding bug-bounty type rewards for their findings.

Conclusion

I hope you enjoyed my ramblings - please note that this article is purely based on my own personal experiences and your own mileage may vary.

If you think I’ve missed anything, then please let me know in the comments section below!