Before the Project

This chapter will focus on project management from an agile perspective (framework-free).

The Requirements Pit

Requirements gathering is more like requirements digging—nobody knows exactly what the want. Programmers help people understand what they want. The requirement (once expressed) should be seen an invitation to explore.

For example, the statement “Shipping should be free on all orders over $50.” should beget a bevy of questions about including tax, which shipping methods are included, internationalization, frequency of changing that $50 number, etc. The client (who has great domain knowledge, hopefully) has probably thought of some of these cases, but discussing them helps flesh out the actual requirements. A developer’s role is to interpret what the client says and feed back to them the implications.

Requirements are a process. They are learned in a feedback loop. Help your client understand the consequences of their stated requirements. Sometimes you won’t know the domain of knowledge well enough to provide salient feedback, so use “is this what you meant?” through mockups and feedback.

Walk in your client’s shoes. Work with a user to think like a user.

There’s a difference between requirements and policy. Policy should be metadata. Policy can be something like “Only an employee’s supervisor and the personnel department may view that employee’s record.” Taken literally as a requirement, it might be hard-coded. Taken as policy, you’d build / use an access control system that allowed authorized users to view a record—defining what is an authorized user would then be metadata (or configuration).

There’s also a difference between doing what the requirement asks, but not how the customer wants. Early prototypes and tracer code are key to nailing both.

Requirements are best recorded as user stories—describing a small portion of what the application does from the perspective of the user. Keep the stories small enough to fit on an index card (real or virtual), which encourages develpoers to ask clarifying questions (record the answers, for documentation).

Developers should maintain a project glossary that contains all domain-specific vocabulary for that project. It’s hard to succeed on a project if users and developers cal the same thing by different names.

Section Challenges

• [ ] Exercise 33 Which of the following are probably genuine requirements? Restate those that are not to make them more useful (if possible).

1. The response time must be less than ~500ms.
2. Modal windows will have a gray background.
3. The application will be organized as a number of front-end processes and a back-end server.
4. If a user enters non-numeric characters in a numeric field, the system will flash the field background and not accept them.
5. The code and data for this embedded application must fit within 32Mb.

• [ ] Can you use the software you are writing? Is it possible to have a good feel for requirements without being able to use the software yourself?
• [ ] Pick a non-computer-related problem you currently need to solve. Generate requirements for a non-computer solution.

Solving Impossible Puzzles

The key to solving puzzles is both to recognize the constraints placed on you and to recognize the degrees of freedom you do have, for in those you’ll find your solution. Challenge any preconceived notions and evaluate whether or not they are real, hard-and-fast constraints. Categorize and prioritize your constraints.

You can help yourself solve puzzles by getting out of your own way. If you get stuck, do something else so that you let your subconscious (an amazing associative neural net) can process it. If you can’t do that, try to explain the problem to someone. And, like Hitchiker’s Guide to the Galaxy says, “DON’T PANIC”.

Section Challenges

• [ ] Take a hard look at whatever difficult problem you are embroiled in today. Can you cut the Gordian knot? Do you have to do it this way? Do you have to do it at all?
• [ ] Were you handed a set of constraints when you signed on to your current project? Are they all still applicable, and is the interpretation of them still valid?

Working Together

I’m a lover of pair programming, so this section really resonated. I love the resultant product that comes from the tensions of two people (or more) working out a solution together. I think two is almost always better than one.

• [ ] Have you tried pair programming? What was your experience?

The Essence of Agility

From the manifesto:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

Any framework or method that pushes the right side over the left side should be abandoned. Agility is:

1. Work out where you are.
2. Make the smallest meaningful step towards where you want to be.
3. Evaluate where you end up, and fix anything you broke.

Section Challenges

• [ ] The simple feedback loop isn’t just for software. Think of other decisions you’ve made recently. Could any of them have been improved by thinking about how you might be able to undo them if things didn’t take you in the direction you were going? Can you think of ways you can improve what you do by gathering and acting on feedback?

While You Are Coding

Coding is an intensely creative act that happens through instincts / nonconscious thoughts and the active thoughtful application of principles.

Listen to Your Lizard Brain

The first trick is to notice that it’s happening; then work out why. You may not be able to, but try to crystallize it into something solid that you can address. Let your instincts contribute to your performance.

• fear of the blank page is usually caused by
• some kind of important doubt lurking below perception
• you might be afraid you’ll make a mistake
• how to figure out what’s lurking below perception?
• let your subconscious brain process the problem by doing something else away from the keyboard
• try externalizing the issue: rubber duck, doodle
• maybe even prototype around the problem space giving you disease; this context switch can help you work through it
• in other people’s code, notice what give you pause so that you can lean from the person who wrote it

Concurrency

• concurrency is when the execution of two or more pieces of code act as if they run at the same time: it is a software mechanism
• parallelism is when they do run at the same time: it is a hardware concern
• Concurrency requires that you run code in an environment that can switch execution between different parts of your code. This is often implemented using things like fibers, threads, and processes.
• Almost any decent-sized codebase will need to handle concurrency: it’s just a real-world requirement.
• temporal coupling happens when your code imposes a sequence of things that is not actually required to solve the problem at hand.
• Shared state is the biggest liability once two things can happen at the same time.

Breaking Temporal Coupling

There are two aspects of time that are important to us: concurrency (things happening at the same time), and ordering (the relative positions of things in time). We should think about concurrency in our project by analyzing our workflow through an activity diagram.

• drawn with rounded boxes for actions
• arrow leaving an action leads to another action (A -> B means B must wait on A) or to a thick line called a synchronization bar
• once all actions leading to a synchronization bar are complete, you can proceed along any arrows leaving it (to another action)
• actions with no arrows leading in can be completed at any time.
• this activity helps to identify activities that could be performed in parallel

Section Challenges

• [ ] How many tasks do you perform in parallel when you get ready for work in the morning? Could you express this in a UML activity diagram? Can you find some way to get ready more quickly by increasing concurrency?

Shared State is Incorrect State

Scenarioizing servers in an restaurant selling pie (in a pie case) to their tables. Both see the last piece of pie and promise it to their table. One is disappointed.

• The problem is not that two processes can write to the same memory; the problem is that neither process can guarantee that its view of that memory is consistent.
• This is because fetching then updating the pie count is not an atomic operation: the underlying value can change in the middle.
• Semaphores (a thing that only once process can own at a time) can solve this: only the server holding a plastic Leprechaun from the pie case can sell a pie.
• trade-offs: this only works if all follow the convention
• mitigation: move resource locking / semaphore handling into resource
• multiple-resource transactions (pie and ice cream) should generally be a separate resource (so that you don’t have a server holding pie without ice cream)
• non-transactional updates: concurrency problems aren’t restricted to writing to memory, but can pop up in files, databases, external services, etc.
• whenever two or more instances of your code can access some resource at the same time, you have a potential for a problem
• random failures are often concurrency issues
• many languages have library support for mutexes (mutual exclusion), monitors, or semaphores
• one could argue that functional languages (tendency to make all data immutable) make concurrency simpler, though they also run in the real world, subject to temporal restrictions, so you need to be aware of concurrency

Actors and Processes

Actors and processes offer a means to implement concurrency without the burden of synchronizing access to shared memory. Use actors for concurrency without shared state

• an actor is an independent virtual processor with its own local (and private) state
• each actor has a mailbox, whose messages get processed as soon as the actor is idle
• there’s no single thing in control: nothing schedules what happens next, no orchestration of raw data to final output
• the only state in the system is held in messages and the local, private state of the actor
• all messages are one-way: there’s no built-in concept of replying
  • replies can be built by including a mailbox address in the message; make replying part of the message processing
• processes each message to completion, and only processes one message at a time
• a process is typically a more general-purpose virtual processor, often implemented by the operating system to facilitate concurrency

The diner / server scenario with actors:

• the customer becomes hungry
• they respond by asking the server for pie
• the server asks the pie case for pie
• if available, pie case sends pie to customer and notifies waiter to add to bill
• if not available, pie case informs server

In all of the code for this, there’s no explicit concurrency handling, as there is no shared state. Erlang language and runtime are a great example of an actor implementation. Erlang calls actors processes, but they’re not regular operating system processes as described in this chapter’s notes. Also has a supervision system that manages process lifetimes.

Section Challenges

• [ ] Do you currently have code that uses mutual exclusion to protect shared data? Why not try a prototype of the same code written using actors?
• [ ] The actor code for the diner only supports ordering slices of pie. Extend it to let customers order pie a la mode, with separate agents managing the pie slices and the scoops of ice cream. Arrange things so that it handles the situation where one or the other runs out.

Blackboards

Blackboards provide a form of laissez faire concurrency, where the blackboard is the storage repository for independent processes, agents, actors, etc. They may be a good fit when writing an application with many sometimes independent and sometimes inter-dependent steps. Use them to coordinate workflows. Imagine writing an application to process loan applications.

• responses (to credit inquiries, bank account balances, etc.) can arrive in any order.
• data aggregation may be done by many different people, distributed across different time zones
• some data gathering may be done automatically by other systems; data may arrive asynchronously
• certain data may be dependent on other data (e.g., cannot start a car’s title search until the system has proof of ownership)
• the arrival of new data raises new questions and policies; e.g., bad result from a credit check means stricture requirements for down payment

Messaging systems (think NATS, SQS, Kafka) can be like blackboards, since they offer message persistence and the ability to retrieve messages through pattern matching. Using blackboards comes with trade-offs:

• harder to reason about business logic because everything is disconnected
• observability can suffer unless you implement a system
• agreeing on a communication format (or at least having a repository of different formats) requires work
• more troublesome to deploy as there are more moving parts

Section Challenges

• [ ] Exercise 24 Would a blackboard-style system be appropriate for the following applications? Why or why not?
• Image processing: you’d like to have a number of parallel processes grab chunks of an image, process them, and put the completed chunk back.
• Group calendaring: you’ve got people scattered across the globe, in different time zones, speaking different languages, trying to schedule a meeting.
• Network monitoring tool: the system gathers performance statistics and collects trouble reports, which agenst use to look for trouble in the system.
• [ ] Do you use blackboard systems in the real world—the message board by the refrigerator, or the big whiteboard at work? What makes them effective? Are messages ever posted with a consistent format? Does it matter?