This is incredibly true. I once turned 60kLoC of classic ASP in VBScript into about 20kLoC of python/django including templates. And added a bunch of features that would have been impossible on the old code-base.

It turned the job from hellish (features were impossible to add) to very nearly boring (there wasn't much to do anymore). So with this newfound freedom I built some machines to automate the data entry and once that got rolling the job got even more boring. Because it was a small company with a very long learning curve the owner didn't let people go, but instead kept them on so that he didn't have to hire and train new people as growth accelerated.

But with all the automation some slack found its way into the system and problems that had normally required me to stop working on my normal job and help put out fires now got handled by people who weren't stretched a little too thin.

Sadly there's (seemingly) no way to interview people for this ability so we're stuck with the standard "write some algorithm on a whiteboard" type problems that are in no way indicative of real world capabilities.

Mirror since the site is currently down:

The most important skill in software development

Posted on 18 June 2015 by John Here’s an insightful paragraph from James Hague’s blog post Organization skills beat algorithmic wizardry:

When it comes to writing code, the number one most important skill is how to keep a tangle of features from collapsing under the weight of its own complexity. I’ve worked on large telecommunications systems, console games, blogging software, a bunch of personal tools, and very rarely is there some tricky data structure or algorithm that casts a looming shadow over everything else. But there’s always lots of state to keep track of, rearranging of values, handling special cases, and carefully working out how all the pieces of a system interact. To a great extent the act of coding is one of organization. Refactoring. Simplifying. Figuring out how to remove extraneous manipulations here and there.

Algorithmic wizardry is easier to teach and easier to blog about than organizational skill, so we teach and blog about it instead. A one-hour class, or a blog post, can showcase a clever algorithm. But how do you present a clever bit of organization? If you jump to the solution, it’s unimpressive. “Here’s something simple I came up with. It may not look like much, but trust me, it was really hard to realize this was all I needed to do.” Or worse, “Here’s a moderately complicated pile of code, but you should have seen how much more complicated it was before. At least now someone stands a shot of understanding it.” Ho hum. I guess you had to be there.

You can’t appreciate a feat of organization until you experience the disorganization. But it’s hard to have the patience to wrap your head around a disorganized mess that you don’t care about. Only if the disorganized mess is your responsibility, something that means more to you than a case study, can you wrap your head around it and appreciate improvements. This means that while you can learn algorithmic wizardry through homework assignments, you’re unlikely to learn organization skills unless you work on a large project you care about, most likely because you’re paid to care about it.

The site seems to be down.

Cached version: http://webcache.googleusercontent.com/search?q=cache:Mf9074z...

the number one most important skill is how to keep a tangle of features from collapsing under the weight of its own complexity

Agreed.

very rarely is there some tricky data structure or algorithm that casts a looming shadow over everything else

Agreed, BUT...

In order to organize, sooner or later, you will have to get clever (with tricky data structures or algorithms).

How I have always built something big and/or complex:

  1. Add lines of code. 
  2. Add lines of code.
  3. Add lines of code.
  4. Holy shit! What have I done?
  5. Refactor.
  6. Combine similar sections.
  7. Genericize with parameter-driven modules.
  8. Still too many lines of code! Optimize Step #7 with something clever.
  9. Go to Step 1.

For me, organizing complex code has always been a delicate balance between readibility and cleverness.

Don't remove enough lines of code and it's too much to navigate. Remove too many and it's too much to comprehend.

Organization + Cleverness + Balance = Long Term Maintainability

The most important skill in operations: systematic debugging. If the developers did well at organizing, this is much easier.

The most important skill in low-level technical support: diplomacy.

The most important skill in high-level technical support: figuring out what people are actually complaining about.

Note that many low-level technical support problems look like high-level problems, and vice versa.

I'd say it's not really about "organizing complexity" - because that tends to just push it around somewhere else - but reducing complexity which is most important.

In my experience it has been that designs which look "locally simple" because they have such a high level of abstraction are actually the most complex overall. Such simplicity is deceptive. I think it's this deceptive simplicity which causes people to write a dozen classes with 2-3 methods of 2-3 lines each to do something that should only take a dozen lines of code (this is not that extreme - I've seen and rewritten such things before.)

Perhaps we should be focusing on teaching the techniques for reducing complexity more than hiding it, and that abstraction is more of a necessary evil than something to be applied liberally. From the beginning, programmers should be exposed to simple solutions so they can develop a good estimate of how much code it really takes to solve a problem, as seeing massively overcomplex solutions tends to distort their perspective on this; at the least, if more of them would be asking things like "why do I need to write all this code just to print 'Hello world', and the binary require over a million bytes of memory to run? Isn't that a bit too much?", that would be a good start.

“Here’s something simple I came up with. It may not look like much, but trust me, it was really hard to realize this was all I needed to do.”

This reminds me of something Scott Shenker, my computer networking professor at Berkeley, drilled into us every chance he got: Don't manage complexity. Extract simplicity.

Finding complex solutions to complex problems is comparatively easy. Finding simple solutions to complex problems is hard.

This times 1000! The really sad thing is that 9.9 out of 10 technical interviews are all about how many Algo's you know. Even if your job will never require actually implementing a single one.

These interviewers do not seem to care at all about the actual process of writing and refactoring code, or what your finished products actually look like.

I know plenty of programmers who can write highly optimized code, but do so in a way where it is completely impossible to maintain.

It's especially heightened when you are given a problem such as "validate if this uses valid brackets, and you have 10 minutes." When under time constraints to solve what is basically an algorithm 99% of programmers are not going write their best code or write code in the way they would normally write code.

If you are using lots of algo's on your programming interviews, I suggest you take a step back and determine if those skills are actually what you want to be testing for in this job. Odds are that it is NOT algo's. Give your interviewer some really sloppy code to refactor into something beautiful. Give them a few hours to work on it, watch how they put the pieces together.

If your position isn't going to require someone to write advanced algorithms on daily basis, testing for them only cuts out a huge swath of potential talent. I also think it probably leads to less diversity in your work place, which is a bad thing.

A Web Developer will never need to solve the towers of Hanoi problem, but they will need to write clean code that can be maintained by others.

</rant>

I agree that it is an important skill in engineering to recognize when the complexity got too high - At that point you need to take a step back (or several steps back) and find a different path to the solution - Sometimes that means throwing away large amounts of code. It's about acknowledging (and correcting) small mistakes to make sure that they don't pile up into a giant, disastrous one.

Another thing I learned is that dumb, explicit code is highly desirable - It's better to have 10 dumb components to handle 10 different use cases than 1 clever component that can handle all 10 cases.

I think the most important skill is being able to break down problems into their essential parts and then addressing each part individually but without losing track of the big picture.

Organisational skill is actually an entrepreneurial skill. I'm learning the hard way that there are two diametrically opposing skills you need to master for both running a business and programming effectively:

1. Skills in Creating 2. Skills in Organising those creations

The thing is, the Creating part is always exciting but it's disruptive in nature. The Organising part is boring because it's about taming or tempering the creation in such a way that it can be referenced later on (just like filing your invoices, or timesheets - yuck - but necessary).

Unless you've got a systematic method to organise your creations, you will always be alone with your ideas, find it hard to resume creative chains of efforts and ultimately flounder without profit.

Both in business and in programming.

Damn right it's the most important software development skill.

I disagree with the premise that organizing code is not a recognized or appreciated skill among developers.

At least not since this was published:

http://www.amazon.com/Refactoring-Improving-Design-Existing-...

Martin Fowler really struck a chord all the developers trying to do the right thing by cleaning up badly structured code, by giving the practice a name and explaining why it's important. Refactoring is definitely a widely acknowledged and accepted practice today, although probably more so in some communities than others.

This has been my experience too. I've been involved cleaning up a half dozen or so projects that got out of control. In each case, technical complexity was blamed as the reason. After digging in, I found that following commonalities:

- Incomplete, conflicting and misunderstood requirements.

- Lots of "We never thought we would need to do X".

- Poor team communication.

- Mistrust, frequently well earned.

- Harmony valued over truth.

Once these were winnowed away, the problems rarely overwhelmed the technical teams. This isn't to diminish the importance of technical skills. Rather - when everything else is f*cked up, you can't blame the technology or expect a 10xer to pull you out of it.

I like this little article a lot. Personally, I try to write code in a way that reads like a book. Lots of comments, explicit function names, explicit variable names, object names, class names, ect. Talking about languages higher level than C/assembly here obviously.

I am amazed at all the code I see that has terrible/too generic names for functions and variables and objects. Some people get so obsessed over short function names, one character variable names, and complicated looking one liner evaluations.

Google cache, for those experiencing connectivity issues.

http://webcache.googleusercontent.com/search?q=cache:Mf9074z...

There is just a nice magical moment though when you grok what an application does and how it is organized, especially when you didn't write it, which is likely the most often case.

I think a large issue is when an application has new people working on it or entirely new teams that maintain it. That is when the original authors methodology and organization of the application falls to the immediate need of the day.

The functionality of the software should speak for itself. Commenting your code with why you are doing something is important to help other maintainers later on, including yourself, understand what you were thinking when it was written.

I would add to the conversation the fact that most projects fail on the set of the initial requirements. In my experience so far I have seen that constantly changing what you want the app to do creates a huge mess in the codebase even if you use latest tool and methodologies.

Looks like there is a great value to organise your app in way to be able to throw away large chunks of code and start over in case there is a big design change.

"The art of programming is the art of organizing complexity, of mastering multitude and avoiding its bastard chaos as effectively as possible." --Dijkstra

:)

Organization is the hardest part for me personally in getting better as a developer. How to build a structure that is easy to change and extend. Any tips where to find good books or online sources?

I disagree. Communication is the most important. It's the number one cause of failed software project. Miscommunicated features, capabilities, scope, failure to name a few. My favourite is the last one. Not standing up and recognizing that an approach is not working due to fear has to stand out as a big one.

Which is why jupyter/ipython is so great -> you can do some fantastic documentation of code with working examples and visualisation, you can do it while you write it !

Relevant: http://www.safetyresearch.net/blog/articles/toyota-unintende...

Summary: Toyota settled an unintended acceleration lawsuit connected with analysis of the source code for a 2005 Toyota Camry showed it was defective "spaghetti code."

There'a a lot of poorly-organized code in the world, and a typical excuse for not cleaning it up is that "it works" so there would be no return on fixing it. In the Toyota case, the code may have contributed to unintended acceleration, and did result in a legal exposure for which Toyota felt it was necessary to settle a lawsuit.

This can be reduced to:

    Do Not Program Yourself into a Corner

Nice post, organization definitely is one of the most overlooked aspects of programming. It takes a lot of experience and thinking to be able to organize properly. It's really what separates the beginner programmers and the experienced ones.

I agree that complexity is far up there. But also risk. Also long term thinking. And net cost or net profit. The more years I have under my belt, I think more and more not only about complexity, but also risk, cost, profit. Code and hardware is just a means to an end. Not the end itself.

But yes, seek the minimum amount of complexity to materialize the inherent, necessary complexity. But don't allow a drop of complexity more than that. Architecture astronauts, pattern fashionistas, I'm looking at you. KISS. Spend your complexity dollars where it gives you something you truly need or want. Don't do things Just Because.

Design patterns by the Gang of Four was great for this.

Interesting. Most comments seem focused on code complexity. In real life situations the complexity is blend of human interactions (attitude, consistency,team,leads, peers, family, emotions) business, market, competition, budget, time, attrition, unexpected events, and more.

Life is complex. Business and workplace dynamics can be complex. People are complex with their own strengths, quirks and situations. Having a broad outlook, developing patience and skills to deal with life and work is part of becoming mature.

It is a reasonable point that organization is important but I have to disagree about "most important".

The most important skill in software development, by far, is managing your own psychology.

This is always a good idea. Here are some of the things I do

- Establish conventions early; Conventions in managing projects, conventions in code. And stick to those conventions. Be predictable, in the code. Use simple names.

- Protect your interfaces. By this I mean, use a central place like a wiki to document your interfaces, so all involved parties may agree upon. Write unit-tests for interfaces. Use libraries like mockito and hamcrest that make it a breeze.(You would lock your home every time you go out, don't you?)

- I mentioned this in the previous bullet, but write tests. Write lots of them, write tests that document any tricky, magical behavior. Cover all the cases(A boat with one hole is bad as one with two holes). Cover all the invariants, any thing that you notice but didn't mention in the source code. Write tests for the bugs you just fixed.

- If you are developing in Java, please use an IDE. I use Intellij, but Eclipse is good too. It makes refactoring code much easier. Rename fields, pull classes up the hierarchy, create abstract classes, create getters and setters automatically with refactoring tools. I am not against emacs or vi, but it is hard to manage Java's sprawl with them.

One of the best programmers I know writes code like it has been generated with a program. It is boring, dull and looks alike in every direction. Every field and method is documented, it says what its purpose is, and why it is needed. He is very fast(fast for a startup, not your average enterprise), accurate and gets a lot of stuff done without magic.

No part of a system is the most important part, from a car to a huge organization, all parts are equally required to interact and hence make such system 'work'.

Having a clear functional organization at the start (and respect it throughout development) is very important, but after that is equally important to code clean and efficient code, to test, to debug, etc. Then, going up and dow on the solution stack is important to make the right decision on hardware, OS, server, services, etc.

Not only is it the most important skill (unless for small home projects maybe), it's also the one which takes the longest to learn and hence the one you really improve on during the years and which distinguishes the experienced ones from the lesser experienced ones. Coincidently, it's also the skill for which you won't find a ready-made answer on stackoverflow or any other site/book.

thinking of it, I've also seen this as a typical difference between fresh CS graduates and those who have been programming for 10+ years. The latter would sometime take way longer to come up with clever math-oriented algorythms than the first, because the graduate has been trained for it and still has it fresh in memory, but experienced programmer would make up for that by being able to use the algorithms in all proper 'best practice' ways one can think of. Whereas the graduate would just slam it in somewhere and call it a day even though there are now x more dependencies and whatnot, you get the picture.

I can't load the page, but I would definitely agree with the title.

From my own experience programming here are some the most common and best ways to better organize complexity:

1) Create DSLs. The Sapir-Whorf hypothesis is the theory that an individual's thoughts and actions are determined by the language or languages that individual speaks. By creating DSLs we are able to reason about a problem domain with increased efficiency.

2) Reduce cognitive load by reducing state. By reducing the number of variables in a given section of code we can more easily reason about it. values.map (x) -> x * x is a lot more understandable than newArr = [] ; for (i=0; i<values.length; i++) { newArr.push( values[i] * values[i] ); }

3) Build tools to build tools. The history of computing is one of building tools on top of tools. From assembly language to C to high level languages. What is is the next step? I suspect it is some kind of polyglot environment that is a hodgepodge of languages all working together combined with automated code creation from AI.

For those not able the view it. [Cacheview](http://webcache.googleusercontent.com/search?q=cache:http://...)

Since so many people prioritize getting then task done than writing organized/beautiful code more often then not we get code that isn't organized properly.

Thus, as a result: Interpreting complexity is by far the most important skill in software development. More-so then organizing complexity.

Aside from personal discipline and experience, I’ve found that using strongly typed and compiled languages combined with good tools are the best way to accomplish this.

Being able to search for and manipulate symbols at the AST level goes a long way towards eliminating any resistance to refactoring.

This is why I love Haskell. Haskell seems to increase my ability to manage complexity by one level.

Disclaimer: Haskell is not a silver bullet, not a panacea and I'm only claiming a modest increase, not miracles, but it helps me deal with complexity better than any other language I know.

In a recent job I ended up rewriting some of the codebase with this kind of stuff in mind... the results:

- 10% of the LOC as previously

- Code significantly more understandable

- Jr. developer on the team suddenly became a rockstar b/c he could understand what was going on.

Or in other words: code should be as simple as possible, but no simpler. I guess you could call it organization or readability or just good design. It requires deeply understanding what you're trying to accomplish and structuring your code to reflect that. I don't think there's any rote, step-by-step procedure that will get you there. Often it is a flash of creative insight that breaks the log-jam and reveals the hidden inner structure of the problem. Once that is revealed the code writes itself. In other words, good code should always make the problem that was solved look easy.

I take it from his site not loading that the most important skill is learning how to scale your site and ensuring it remains accessible during high load times, or using CloudFlare to at least ensure it gets cached.

"""But there’s always lots of state to keep track of, rearranging of values, handling special cases, and carefully working out how all the pieces of a system interact."""

I'm not a functional programming evangelist but that reads like a very good reason to go for FP. I think a similar point was made in "Functional JavaScript". I don't remember it exactly and it's on my shelf at home but there was some passage about the biggest downside of typical OOP codebases being the mental effort of keeping track of values and value changes.

> Do the difficult things while they are easy and do the great things while they are small. A journey of a thousand miles must begin with a single step. (source: http://www.brainyquote.com/quotes/quotes/l/laotzu398196.html...)

This applies to systems as much as any thing else it possibly could.

This pretty much aligns with my take on that from four years ago: http://berislav.lopac.net/post/13061099545/the-most-importan...

As wise men said: All problems in software can be solved with more layers of abstraction, except of the problem of too many layers of abstraction.

And Metalevel complexity. I may have been able to craft beautiful code, but I sensed chaos in the way I operate and manages my own resources / tools. I've seen people being organized at many, if not all, layers, solving problem and solving how to help solving problems. Witnessing that makes me feel calm and envious at the same time.

ps: it's also reminiscent of recursion, dogfooding etc.

It's a very important skill for a programmer to have, especially in the modern environment where distributed systems built from many integrated components are the norm. That said, it's awfully difficult to disentangle the various skills needed for programming and assign an importance to each one, mush less to determine which of them is actually most important of all.

The website is down but here is a link from the archive.org site. http://web.archive.org/web/20150622134205/http://www.johndco...

Here's the Google cache if someone is looking for a mirror: http://webcache.googleusercontent.com/search?q=cache:www.joh...

Text only mirror:

http://webcache.googleusercontent.com/search?q=cache:http://...

undefined

Yeah this is so true.

I worked on a system where we had to support imperial and metric units. It was done in a pretty bolted on fashion with if statements all over the place. And sometimes it isn't even clear if it could be done in any other way.

Any HN'ers have suggestions on how to do it elegantly.

> Organizing complexity is the most important skill in software development

I agree with this profoundly. Unfortunately, complexity is in the eye of the beholder. When comparing solutions to a problem, different developers will not always agree on which is the least complex.

Our humans are "comparing machines". For this reason we tend to valuate people that solve problems more than the ones who never create them. This is really bad.

Also, in business, if someone is really good administrator, it seems he never does nothing.

On the topic of organization and related to another post about good Software Development books. What are some books that teach code organization as discussed in this post. One I can think of is "Refactoring" by Martin Fowler.

What are some others?

undefined

I was thinking that recently. The way I try to stay organized is to comment(with timestamp) every time something is changed so I can refer to it later. Does anyone have more tips on how to stay more organized?

there is an excellent book by jon lakos called 'large scale c++ design' which treats organizational or physical design of large scale c++ projects (> 1giga loc). highly recommended.

Arguably, this is what higher level languages like Java and C++ provide. Tight organizational language metaphors that help implement design patterns in a thoughtful, consistently structured manner.

Important according to what metric? Making the software developer feel good, or making the company money? The former is almost certainly true, the latter is almost certainly not.

google cache record: http://webcache.googleusercontent.com/search?q=cache:Mf9074z...

The server seems to be getting hammered.

Does anyone know of any book or website or other resource that has before and after examples of good refactoring, or something similar?

There is a term for it. It's called managing scope. I'm surprised the article didn't mention this.

Out of all the things I know how to do in programming, reducing complexity is probably the one I'm best at. So how do I get a job doing this? Or a series of lucrative consulting gigs? :-)

I'm pretty sure I'm not as smart as I used to be, and I'm definitely not as smart or productive as some of the younger programmers I've worked with. (Sorry for the ageist remark!)

This may be my secret advantage: I have to keep my code simple enough that even I can understand it.

Here's a fun example that I've seen more than a few times in various forms: four-way navigation, either involving up/down/left/right or north/south/east/west, or both.

In one (somewhat disguised) project it worked like this: the code had several different modules to provide a keyboard interface for geographic navigation, while keeping the geo code separated from the low level details of key codes and events and such.

There was a keyboard manager that mapped keycodes to readable names that were defined in an enum:

  switch( keyCode ) {
      case 37:
          return KEY_LEFT;
      case 38:
          return KEY_UP;
      case 39:
          return KEY_RIGHT;
      case 40:
          return KEY_DOWN;
  }

  switch( keyEnum ) {
      case KEY_LEFT:
          BroadcastMessage( 'keyLeft' );
      case KEY_RIGHT:
          BroadcastMessage( 'keyRight' );
      case KEY_UP:
          BroadcastMessage( 'keyUp' );
      case KEY_DOWN:
          BroadcastMessage( 'keyDown' );
  }

  // Don't forget to reverse the directions here
  events.on( 'keyLeft', function() {
      moveRight();
  });
  events.on( 'keyRight', function() {
      moveLeft();
  });
  events.on( 'keyUp', function() {
      moveDown();
  });
  events.on( 'keyDown', function() {
      moveUp();
  });

  function moveUp() {
      map.pan( maps.DIRECTION_NORTH );
  }
  function moveDown() {
      map.pan( maps.DIRECTION_SOUTH );
  }
  function moveLeft() {
      map.pan( maps.DIRECTION_WEST );
  }
  function moveRight() {
      map.pan( maps.DIRECTION_EAST );
  }

Imagine if math worked like this: You'd have to have two of every function, one for the positive numbers and another one for negative numbers. And probably four different functions if you are dealing with a complex number!

That of course suggests a solution: use numbers instead of names, +1 for up and -1 for down, ditto for right and left. And pass these numbers on through any of these layers of code so you only need half the functions. If you need to flip directions along the way (like the left arrow key navigating right), just multiply by -1 to reverse it instead of having to make special cases for each direction name.

You might even decide to combine the two axes, so instead of vertical and horizontal, you've got +1 and -1 there too (or 1 and 0, or something that lets you handle both axes with one piece of code). Now you could be down to a quarter of the original code.

Unfortunately, I was brought in on this project near the end to help wrap up a few other tricky problems, and all this navigation code was already set in stone. (And to be fair, working and tested, and who wants to go back and rewrite proven code, even if it is four times the code you need?)

But this would make a pretty good "how would you clean this code up" interview question!

undefined

Yeah, the best developers I've ever seen are about simplifying things, avoid overcomplication for too much "flexibility".

The zen of Python https://www.python.org/dev/peps/pep-0020/ say:

    Simple is better than complex.

I agree that most complexity in software systems comes from managing state. So here is a simple solution - stop doing it. Stop managing state.

Use the right tools for the job. Most mainstream programming languages are ridiculously inadequate for building any production software of any significant complexity without introducing more problems than you are trying to solve.

Use a mature functional programming language that works with immutable data and is designed for building complex industrial systems.

Use a language that was designed from the beginning with the understanding that your software WILL be full of bugs and errors, yet systems must always continue to run.

Use Erlang.

i see "scaling a web server" is still at the bottom of the list, while you idiots whine about whatever-the-fuck.

you lose. good day, sir.

page not found

Great read. 100% agree.

Oh, I thought the most important skill in software development would be SCRUM.